* Added a MasterStatus() method and carried it through the entire gRPC
chain, similar to what we do for SlaveStatus(). Other WIP changes to be
updated after getting approval on the MasterStatus() parts.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Small fixes per review suggestions.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Descriptive comments added per review suggestions.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added unit tests and small fixes.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Refactored per review suggestion. Under new design, it's not possible to unit test anymore, since we don't have mock connections. Removed unit test because of this.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Fix very odd rebase issue I've never seen before, where rebase didn't show me conflicts in various files that it allowed to continue without fixing known conflicts.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added new MasterStatus return for DemoteMaster and carried it through the entire call chain.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Remove deprecated field everywhere we possibly can, and ensure compatibility for newer client interacting with older server.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Remove copying of XXX fields, per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added the option to StopReplicationAndGetStatus() to stop only the IO Thread, and passed it all the way down the call chain to MysqlDaemon, where we can now call a new method (implemented in all flavors) which stops only the io thread, if that's what was requested.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Oops
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Remove hook per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Adjusted stop slave io thread to pass in a ctx because it's a new function. Adjusted StopReplicationAndGetStatus so that it stops the slave before getting slave status. This will ensure that the relay log information is correct.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Add back in logic to bail if slave is already stopped.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* We have to patch these in because after calling stop slave there won't be a master host and master port anymore when we grab slave status. Instead we need to patch in positions so we retain master host and master port, otherwise set master will assume the tablet is the master because it has no master host and master port. In retrospect its probably a bad idea that we assume no master host and no master port means we've found the master (in set master).
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Refactored per offline discussions. We now return before and after slave status, so we are more explicit, and don't nest business logic into subfields of a hybrid struct.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Fix issues that cropped up after merge conflict
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Change way we get this state to also pull in the Connecting state. We are either running, or attempting to run. Either way we are not not running.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Changed references from slave to replica per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Embed StopReplicationStatus into StopReplicationAndGetStatusResponse and rename fields to make it clear.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Various fixes per review suggestions.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Lets try out issuing a stop no matter what and see what happens.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Adding back in bailouts. They are necessary.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Get rid of more slave references.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Changed stopIOThreadOnly to an enum so we can change the way in which we stop replication in the future.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Add a test to ensure that we can stop the io thread only.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Fix incorrect test methodology.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Used a generated enum from proto per convention for the stop replication mode.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Scrub more references of slave without obfuscating MySQL statements that are being called under the hood.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
The fakesqldb was waiting .25 seconds on every call to Close.
This speeds up the tabletserver test considerably.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
- added ability to fetch the GTID from the restore time
- added ability to apply the binlogs till the above GTID
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
This fixes two bugs:
1. AddGTID was mutating the original set. GTIDSet implementations are
supposed to be immutable.
2. AddGTID and parseMariadbGTIDSet did not enforce any order among
domains, yet Equal and other functions expected order to be enforced.
This switches from a list to a map to better represent the fact that the
order of domains doesn't matter, which also simplifies some code.
We always enforce order when serializing back to a string.
Signed-off-by: Anthony Yeh <enisoc@planetscale.com>
* Corrected comment type.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added Union to the GTIDSet interface
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Wrote Union implementations for all current implementors of GTIDSet interface.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* We need to check if stacks are empty before making comparisons.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added unit tests for mysql56 and mariadb implementations.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Multiple fixes per review suggestions.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added helper method for popping intervals so this logic doesn't need to live inside the while loop itself.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Multiple fixes.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added test to ensure that return from Union is sorted by domain correctly. Also ran formatting.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* We don't need to explicitely non-allocate the second assignment.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Let this panic because supplying nil inputs is an invalid function call.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Comment update per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Add check for receiver being nil but other being non-nil, in case of union always return non-nil set.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added wrapper struct, modified usage
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Add go.mod
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Refactor dbconfigs and its calls
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Fix failing vstreamer test
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Fix copy and paste mistakes
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Made withCredentials a private function and refactored the calls
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Fix Tests
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Clean up code
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Extend DbConfigs and change method name
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Add Connect Method
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Pass Context as parameter
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Modify definition of DBConfigs
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Make Connect function part of the Connector definition
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
* Remove obselete functions
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
The mysql server uses buffered I/O. This works fine if the results
are streamed all at once. But if the streaming is intermitent, like
in the case of messages, the client may not get timely responses.
This new change flushes any unsent data if no other data is sent
within mysql_server_flush_delay (100ms).
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
While testing vttablet-mysqld over TCP, we noticed mysql.Connect
sometimes blocks forever if the backend mysqld hit max conns. This
patch adds a timeout to this codepath. It defaults to 0 (no timeout)
for back compatibility.
Signed-off-by: Adam Saponara <as@php.net>
When generating masterGTIDSet in file:pos most likely you will have a topology
like the following:
Source A -> Target B (B has a vreplication stream from A)
From the target perspective, the source A is the master and you want to generate
a gtid that is based on binlog file position of that server.
As an example, let's see this topology:
Master A -> Source B -> Target C (C has vreplication stream from B)
Prior to this change, masterGTIDSet was returning the binlogfile:pos of A. But
in reality, the Target C wants the position of B.
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
* Prior to this commit, flavorpos was using lexicographical comparison of the gtids.
Thas was a bug in this context.
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Azure ignores the mysql_native_password in
the MySQL Login Request packet, and then issues
a Auth Switch Request specifying the mysql_native_password
again, and passing a new salt.
Signed-off-by: Jacques Grove <aquarapid@gmail.com>
Fixes#5532Fixes#5569Fixes#5571
With this fix, unit tests pass for all flavors.
Also fix test.go to cover the newer flavors.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
* StripChecksum was changing the type of the event. This was a bug.
* Adds test to vstreamer to reflect new support for statement based replication
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
* Fixed nil bind variables error by resetting bindvars to an empty map. Also added a recover to catch panics that occur inside Logf due to FormatBindVariables call potentially panicking.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Changed import order back. Order was auto-changed by IDE on save
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Using stored ParamsCount for capacity per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Added explanation comment per review suggestion.
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
* Removed explicit panic after offline talk with sougou and deepthi
Signed-off-by: Peter Farr <Peter@PrismaPhonic.com>
The vstreamer sent GTIDs "as they came". With the new change,
GTIDs are sent only when they matter: on COMMIT, DDL or OTHER.
This new approach makes the protocol easier to understand. Also,
it makes it easier for filePos to continuously send file and position.
The correct values will get used when significant events like
COMMIT are encountered.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
In this scheme, the filePos reader detects whether we are in a
transaction or not, and emits appropriate GTID events.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
* Fix typo in some comments.
* Make VReplicator private again. This change is no longer needed. Originally we
wanted "vtshovel" to be an external process. Given that this now hooks into the
existent engine, there is no need to make this public.
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Fixes#5206
The conn schemaname var can get set through a new connection
or a ComInitDB packet. However, a use statement bypasses this
mechanism and gets handled at the vtgate level. This causes
problems because schemaName in the connection disagrees with
TargetString in vtgate.
To fix this, the new scheme calls a new handler function: ComInitDB
every time schemaName is initialized or changed in the connection.
This way, the TargetString remains the authoritative source
for the current db name.
I couldn't write a test for this specific behavior change because
the mysql go client does not implement support for ComInitDB.
But I've verified the things generally work as expected.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
* Adds binary to run vtshovel.
* At the moment only working in ephemeral mode (i.e no data is persisted back to
vrsettings).
* vtshovel only works for statement based replication right now. This is due to
now having a good way to have a schema loader. We will itereate on this.
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
From mysql docs:
```
It starts with the client connect()ing to the server which may send a
ERR packet and finish the handshake or send a Initial Handshake
Packet...
```
`sql_state` fields are skipped because they are not set according to docs.
```
Note
In case the server sent a ERR packet as first packet it will happen
before the client and server negotiated any capabilities. Therefore
the ERR packet will not contain the SQL-state.
```
Signed-off-by: Adam Saponara <as@php.net>
Merged prepared statement code changes did not take into account
staticcheck warnings. This PR addresses those warnings.
Signed-off-by: Saif Alharthi <saif@saifalharthi.me>
This adds in the ability to re-resolve DNS names used in a connection
of an external database. The default is to do nothing new. A new
command line options `pool_hostname_update_rate` has been added that
will resolve the hostnames used in each pool at the given rate and
will force a reconnect if they have changed
Signed-off-by: Dan Kozlowski <koz@planetscale.com>
If ISM is retried, it's possible it gets stuck. The specific use
case where this happens is when the first attempt went past the
point where semi-sync gets enabled. If we retry ISM after that,
the part where we create the reparent journal table will hang
because the master will have semi-sync enabled, but no replica
will be pointing to it.
To overcome this, we always disable semi-sync when resetting
replication. All the code that enables replication turns semi
sync on as needed already.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
`copy` permits using to mix `[]byte` and `string` arguments without
explicit conversion. I removed explicit conversion to make code simpler.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
To make this possible, some things are added:
- The capability to lock all tables on a tablet, to momenterily stop updates
- Once the database is locked, we can create multiple consistent snapshot
transactions that all share the same view of the data
- Adds the capability to have replication move forward to a specific point
in the transaction log
This commit also refactors tabletserver and tx_engine, moving logic of
state transitions into the tx engine.
Signed-off-by: Andres Taylor <antaylor@squareup.com>
There were a few regressions that got introduced due to
recent changes:
* MYSQL_FLAVOR got dropped from Dockerfile.mariadb.
* New build.env did not get included in the docker builds.
* CapabilityClientDeprecateEOF behaves differently for MySQL 5.7
vs older versions: MySQL 5.6 and MariaDB. For 5.7, the capability
always comes back true, but it comes back as true or false
for other versions. So, I just removed the check.
Also removed column length check in one of the tests because
they are not consistent across flavors.
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
This completes yuananf's work from https://github.com/vitessio/vitess/pull/3683
which was split into several diffs and never fully landed.
One large benefit of multi-statement support is that the "rewriteBatchedStatements"
option in Java's connector-j driver doesn't work without it. There's a bug where
connector-j fails to check the server capabilities and assumes multi-statement support.
Signed-off-by: David Weitzman <dweitzman@pinterest.com>
By setting CapabilityClientConnAttr in the ServerGreeting the client
starts to send key/value pairs in the login packet.
The keys vary per client but often include client pid, client version
client program, driver version, user on the client
Signed-off-by: Daniël van Eeden <git@myname.nl>
Andres Taylor
deepthi
Jacques Grove
Peter Farr
Arindam Nayak
Sugu Sougoumarane
Rohit Nayak
Harshit Gangal
akilan
Morgan Tocker
Luis Calado
huiqing
Anthony Yeh
Keyur Govande
Saif Alharthi
Rafael Chacon
Rohit Nayak
Serry Park
Andy Bursavich
Adam Saponara
Toliver Jue
Ajeet jain
yuxiaobo
tanjunchen
hwdef
Dan Rogart
Guangming Wang
Alex Charis
Dan Kozlowski
kuba--
xichengliudui
tpetr
dcadevil
Michael Hebelka
Deepthi Sigireddi
zhoulin xie
Andres Taylor
Iskander (Alex) Sharipov
Simon J Mudd
David Weitzman
Michael Demmer
Daniël van Eeden