Added two new shard sessions for commit ordering:
pre and post.
Added API to set the commit order and changed
tx conn to honor it.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
To enable conveying out of band warnings, add a new proto definition
for a QueryWarning and add a repeated array to the vtgate session.
Signed-off-by: Michael Demmer <mdemmer@slack-corp.com>
Also, simplify the protoc call by removing some intermediate tools and steps involving the python grpcio-tools wrapper.
The motivation for changing the build was that I had trouble getting the old build to work. It might just have been me.
This approach is arguably simpler because it involves fewer tools, no temp files, and no calling "sed" on the output.
Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Please refer to #2694 and #2670 for motivation and reasoning for
this change.
I've tried to follow best practice in inserting the copyright
headers. In other open source projects, not all files carry
the notice. For example documentation doesn't. I've followed
similar ground rules.
I did not change the php because there is a separate LICENSE
file there by Pixel Federation. We'll first need to notify
them our intent before changing anything there.
As for the presubmit check, it's going to be non-trivial
because of the number of exceptions, like file types,
directories and generated code. So, it will have to be
a separate project.
SingleDb is now deprecated. Legacy behavior is supported through
VTGate's Begin, etc.
The new transaction functionality is implemented in TxConn and
it uses TransactionMode instead. These will be called by the
Executor.
This addresses #2723.
This change is essentially a new protocol for V3. Although
backward compatible, it changes the connection model.
Basically, the newer V3 features will work only if you used
the new protocol.
The new model deprecates keyspace_shard, tablet_type and options
from ExecuteRequest and moves them into the Session. This means
that the Session is generally not empty, and may be updated
by any call to Execute or ExecuteBatch, even if the statements are
not transactional. Consequently, transactional methods like Begin,
etc. are deprecated in favor of Execute("begin").
Transaction modes will now be supported by new `SET` syntax, which
will correspondingly update the Session variable.
This also makes a connection that contains a session non-multiplexable.
We'll need to resolve whether it's still worth exposing this flexibility
to the clients, and if so, how.
For now, I've updated the Go driver to use the "modern" protocol.
However, the low level rpc (Impl) continues to support the older
functions like Begin, etc. This allows us to test the legacy
functionality.
All other clients: Python, PHP and Java are currently unchanged.
Given that this is a major protocol change, it hints at a 3.0,
but the changes are 2.1 compatible with the following exceptions:
* Go driver uses the new protocol.
* vtclient binary requires `-target` instead of `-tablet-type`, etc.
All tests are passing including PHP & Java clients. In terms
of upgrade:
* PHP and Java can be upgraded by just updating the code.
* Python will probably require a brand new library. The existing
vtdb contains way too much baggage, and it may not be worth
retrofitting this new incompatible protocol onto what's
currently there. I'm looking for a name. `vtdb2`, `vitessdb`?
I've updated the code with a simple search and replace.
I also had to run "make proto" and regenerate the Python protobuf files because they include the Java class name in the generated code.
A new SingleDb flag has been added to Begin, which would survive
through the Session. If set, any transaction that exceeds a single
shard will be failed.
VTGate acquires a command line flag: transaction_mode, which can be
single, multi or 2pc. In single mode, it will fail Begins that request
multi-shard. In multi mode, it will fail commits that request 2pc.
In 2pc mode, everything will be allowed.
The per-request flags specify what the app wants, and the vtgate flags
specify what it allows.
* 2pc: Go and python clients
For Go, the transaction mode settings are supported through
the context. This means that it will only work with go1.8.
For Python, it's a cursor constructor parameter.
* 2pc: end-to-end test
They are added to both to queryservice and vtgateservice.
Right now, they only contain the exclude_field_names option.
exclude_field_names is used by vttablet to strip the name from the
fields record. Note the clients don't use that yet, I want the ability
to do it first, then I'll add support for it in the clients.
Allowing ExecuteOptions in vtctl commands. That's how they're tested
now.
Side-changes:
- Removing wantFields from dbclient interface. We never want them.
- Small optimization on query fields for sequences.
The vtgate API takes a starting timestamp, or a starting EventToken. It
will only use the starting EventToken if it's relevant. This is mostly
for tests, but could be used by real clients too to avoid the timestamp
search on the servers.
The only restriction in the vtgate routing implementation is that a
query can only end up on one shard. The stream aggregation code inside
vtgate will be added later.
This change includes:
- proto changes.
- implementing the server side interface.
- implementing the client side interface.
- adding a vtctl VtTabletUpdateStream command to stream from a given
tablet. This is used in end-to-end tests.
- using the python vtgate_client update_stream API in all end-to-end
tests.
- removing the python vttablet direct stream_update API.
- vtgate now better preserves remote errors through its API now, as
withSuffix and withPrefix will preserve the error codes of all
VtError, not just *VitessError.
- Also adding callerid vtgateclienttest tests for all API calls.
1. define rules in the Maven build files to compile the data protos at build time.
2. define a new vtgate service interface that uses the proto3 data structures and
defines an abstract service.
Batch requests in vttablet currently allow "begin" and "commit"
statements. They were treated as special cases where vttablet would
start a transaction if such statements were seen.
Initially, this looked flexible, but I realize that this has drawbacks.
For example, someone could issue a begin without a commit, or one could
issue begin inside begin, or a begin-commit-begin, or a "bEgIn", etc.
Checking for all these corner cases complicates the logic, and does not
add much value to the app.
This change adds an AsTransaction flag to the VTGate Batch
requests. If it's true, then the entire batch is executed in a
transaction. Otherwise, the statements are executed loosely.
Consequently, I'll be outlawing begin and commit as valid statements
inside batch requests.
The next change will be for vttablet.
cannot have the same <package name>.<data type> as these, or we cannot
load them at the same time. So to fix this:
- renaming the conflicting ones from xxx.proto to xxxdata.proto.
- renaming vtgateservice.VTGate to vtgateservice.Vitess
Note we can still change the names I chose here, just not back to
conflicting ones. If anyone has better ideas, we can implement
in subsequent changes. This is to get the import to google3 unstuck.
Andres Taylor
yuxiaobo
zwcn
Sugu Sougoumarane
xichengliudui
Michael Demmer
David Weitzman
Sugu Sougoumarane
Michael Berlin
harshit-gangal
Erez Louidor
Alain Jobart
Ammar Aijazi
Shengzhe Yao