* When auto-commit is on, passDML is on and ExecuteBatch is in a transaction,
there is no need to explicitly create a transaction. We can forward the DML
directly to the database.
* This optimization yielded significant more throughput in vttablets. We got
around 25-30% improvement. Most of our queries are single point
inserts/updates that already use auto commit when coming from vtgates, so this
improvement is something that we've been wanting to do for a long time.
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Although it is spelling mistakes, it might make an affects while reading.
Co-Authored-By: Nguyen Phuong An <AnNP@vn.fujitsu.com>
Signed-off-by: Kim Bao Long <longkb@vn.fujitsu.com>
The new lag tracking introduces the following changes:
* VStreamer sends its current time along with every event. This
allows for VPlayer to correct for any clock skew that may
exist between the two machines. This results in a more accurate
calculation of lag.
* If there are no events to send for a period of time, VStreamer
sends a heartbeat event. This allows us VPlayer to essentially
know for sure that it's still caught up.
* If VPlayer receives no event for its wait period, then it updates
the SecondsBehindMaster stat to indicate that it's actually falling
behind.
The VStreamer timeout for heartbeat is set slightly lower than the
VPlayer idle timeout. This ensures that Vplayer won't timeout
exactly when it's about to receive the heartbeat event.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
To make this possible, some things are added:
- The capability to lock all tables on a tablet, to momenterily stop updates
- Once the database is locked, we can create multiple consistent snapshot
transactions that all share the same view of the data
- Adds the capability to have replication move forward to a specific point
in the transaction log
This commit also refactors tabletserver and tx_engine, moving logic of
state transitions into the tx engine.
Signed-off-by: Andres Taylor <antaylor@squareup.com>
Also had to added transmission of field info, which may come
in handy for encoding the values on the player end.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
If this flag is set for a table, we'll treat the column list
as authoritative and expand select *.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
To enable conveying out of band warnings, add a new proto definition
for a QueryWarning and add a repeated array to the vtgate session.
Signed-off-by: Michael Demmer <mdemmer@slack-corp.com>
MigrateServedTypes has been made idempotent: if it fails in
the middle, you can safely retry the operation. If the operation
has previously succeeded, retrying it will be a no-op (except
for master migration).
For master migration. A new Frozen field has been added to the
tablet control record. This field signifies the point of no
return. If a migrate fails before reaching this state, then
we undo everything and re-enable the source shards. Once we
go past the 'frozen' state, you can only go forward. If there
are failures after the frozen state, the migrate can be safely
retried until successful. Once successful, a retry will return
an error saying that there's no resharding in progress.
The resharding end to end test has been updated to demonstrate
these behaviors.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
Pinning was an internal feature used for pinning the
dual table to shard 0. Exposing this ability in vschema
allows someone to pin an unsharded table to a specific
shard.
This is still not a full feature because we ned to
think about the various use cases like resharding etc.
But it can be used for testing purposes.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
The number of vreplication control functions is too high. I counted
seven at the minimum. So, I'm now trying this new approach: a single
VReplicationExec function that can execute queries. The goal is to
accept SQL statements that access a 'vreplication' table which will
be interpreted by the engine, which will in turn update the
_vt.vreplication table and also accordingly act upon the change,
like start, stop etc.
For now, these queries will only be understood by VReplicationExec.
In the future, we can look at allowing these as part of the
QueryService API.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
Added the following fields to vreplication:
workflow: who created and manages the replication
source: source info for the replication (proto)
state: Running, Stopped, etc
message: any message relevant to current state
BinlogSource contains the source info. It currently supports only
keyrange or table list for now, but can be extended to support
other rules. The information is stored in proto 'String' format
because its keyrange rendering is more readable than JSON.
The current change only populates the fields. Next change will
use the values.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
This change deprecates _vt.blp_checkpoint in favor
of vreplication, which stands for Vitess Replication.
The goal is to make vreplication a standalone sub-module
of vitess that other services can use, including the
resharding worflow.
This first step just changes to using vreplication instead of
blp_checkpoint. The table and APIs to perform more autonomous
functions.
Also, split strategy flags were unsafe and untested. So, they
have been deleted. More wholistic operations will be added
to better manage resharding and vreplication.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
Also, simplify the protoc call by removing some intermediate tools and steps involving the python grpcio-tools wrapper.
The motivation for changing the build was that I had trouble getting the old build to work. It might just have been me.
This approach is arguably simpler because it involves fewer tools, no temp files, and no calling "sed" on the output.
Signed-off-by: David Weitzman <dweitzman@pinterest.com>
This makes RefreshConfig match the behavior of ReinitConfig.
Since the initial InitConfig happens in mysqlctld, it's important that
vttablet requests mysqlctld to regenerate config remotely, or else it
may not use the same settings.
This would allow re-using the current StreamHealthCheck to convey the
status of a group of tablets, between vtgate and l2vtgate (which will
be merged into vtgate).
This is the first step towards making VTGate more schema aware.
The VSchema has been extended to include a list of columns and
their types.
For now, this info has to be supplied as part of the VSchema.
In the future, this can be dynamically updated by vtgates
through a new protocol that will allow the vtgates to
subscribe to schema changes from the vttablets.
It's still useful to manually provide this information because
it can be used as fallback in case the auto-update mechanism
fails.
With this added awareness, two new features can be built:
1. If a column contains text, VTGate can use this information
for collating values correctly.
2. Add a strict mode where VTGate will reject a query if
unrecognized column names are used. Also, we can auto-resolve
column names to the correct table if they are unique in a query.
As part of this change, I added some additional wrappers in json2
for protos. This is because the default behavior does not encode
or decode enums in a user-friendly manner.
Sugu Sougoumarane
SataQiu
Rafael Chacon
Michael Demmer
deepthi
Kim Bao Long
Nguyen Hai Truong
xichengliudui
Andres Taylor
Michael Demmer
David Weitzman
Anthony Yeh
Alain Jobart
huiqing zhou