* Don't block on vschema channel in case of heavy vschema changes and vstream load
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Prevent VStreamer engine deadlocks during state transitions
The VStreamer engine is somewhat unusual in two ways:
1. It is open and running on replica tablets rather than only
running on primary tablets.
2. It has no controllers so the main engine mutex is widely shared.
Because of this, when a tablet has open vstreams (direct binary log
streams) performing work and a state transition starts, it can
deadlock with the tabletmanager's state lock when checking if
the engine is open or not.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* feat: remove cli code which isn't used in vtorc
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: port over the /api/problems endpoint to the http server of VTOrc
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: added comments to api/problems/ endpoint explaining the future scope
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove api endpoints registered in old VTOrc ui
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove gracefulTakeover tests since the api is removed and add tests for the api/problems API
Signed-off-by: Manan Gupta <manan@planetscale.com>
* refactor: fix license year and imports
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add apis for disabling and enabling global recoveries
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add api for reading replication-analysis
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add health api as well
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add tests for the api's ported over
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add liveness endpoint to tests too
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: return an internal server error when VTOrc isn't healthy
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: reduce flakiness in api tests
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Rotate binary log before for snapshot connections
Rotate the binary logs to limit the GTID auto positioning overhead.
This is needed as the currently open binary log (which can be up to 1G in size
by default) will need to be scanned and empty GTID events will be streamed for
those GTIDs in the log that we are skipping. In total, this can add a lot of
overhead on both the mysqld instance and the tablet.
Rotating the log ensures that we are processing a fresh binary log that will
be minimal in size and GTID events.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Don't cause VStream failure if we can't rotate the binary log
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Limit binlog flushes and add e2e test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add flag and Get/Set methods for the rotation threshold
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use dashes for new flags
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Setup 2 healthy replicas, confirm semi-sync and repl state
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Cleanup after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Reduce backup job time so that transform at least gets a chance to run
Signed-off-by: Matt Lord <mattalord@gmail.com>
* More reliably clear out tablet data
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Initial Implementation of partial movetables. Implement cross-shard queries that obey shard routing rules.
Refactor rss updation with routed shard and add to multi col resolution. Run vtgate and reservedconn tests for both regular and with a partially moved shard.
Add ability to run tests with partial keyspaces by adding to a list in the CI generator. Implement that in test.go and cluster setup. Revert previous iteration of running partial keyspace tests in parallel. One test is failing: TestSetSystemVarAutocommitWithConnError.
Add partial vtgate misc test
Add partial keyspace tests to vtgate transaction
Add vtgate queries partial keyspace tests
Review and fix skipped tests where applicable. Refactor to allow conditional skipping of tests for partial keyspaces only so that other tests work exactly as earlier without specifying the DbName, for example
Add move shards example
Cache routing tables and minor refactor. Move move_shards example
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Add lookup index to move shards example. Comment error logs while trying to delete from missing _vt.vdiff table
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Remove move shards demo example
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Add vtgate flag to enable shard routing. Update tests to get vtparams based on partial keyspace setting, so that global routing continues to be tested as earlier without DbName being specified
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
To create subsequent partial movetables workflows we should ignore previous partial movetables (now frozen)
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Remove vdiff2 deletes since they were failing unit tests etc
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Use vtgate flag to enable shard routing rules
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix incorrect test setup
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Expect modified frozen query
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
set vtgate param --enable_shard_routing in partial movetables test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix failing tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix flags test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Use VTParams helper for partial keyspace tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Specify vtParams.DBName for partial keyspace queries
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Self-review. Remove bypass caching.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Report partial SwitchTraffic results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Try to address most of the upstream PR review comments
From: https://github.com/vitessio/vitess/pull/9987
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Enforce proper routing for partial migrations
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Account for local timezone in check for vdiff progress
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Update go version in workflows. Fix gofmt issue
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix failing CI tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Comment failing test for partial keyspaces
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix TestVtGateVtExplain by specifying dbname
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix leftover debug code
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Move partial keyspace tests to a single shard. Refactor test.go to use multiple config.json files
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve test skip comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* gofmt
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix failing tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Try TestFunctionInDefault for partial keyspaces. Update skip comment.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Delete temporary log lines
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* feat: add flag to turn off replication manager
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add test for replication manager failing when the primary tablet's port changes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: path repair replication to also handle the of case of IO error
Signed-off-by: Manan Gupta <manan@planetscale.com>
* ci: add new package to run in cluter 18
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: fix flags test expectation
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: new flags should use hyphens
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix comments in replication manager
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: rewrite replication manager check conditions to be more readable
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: added query logging to safe session
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: added [explain format=vtexplain] parsing and AST
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add planning of vtexplain queries
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: keep the logging even in the precense of autocommits
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: use table result instead of warnings, and move the logging to scattercon
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: log commit to vtexplain logger and added e2e test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: vindex query to mark vindexExec in session
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: run only for gen4 planner
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: vtexplain logs with bindvars replaces, and begin being counted with the real query
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add ACTUALLY_RUN_QUERIES as a comment directive to vtexplain and block dml queries if not specified
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: make sure we add the actually_runs_queries directive to inserts we want to inspect
Signed-off-by: Andres Taylor <andres@planetscale.com>
* docs: added info to release notes
Signed-off-by: Andres Taylor <andres@planetscale.com>
* typo: fix error message
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: ignore cases on comment directives
Signed-off-by: Andres Taylor <andres@planetscale.com>
* fix: CommentDirective now hides internals so that the lower casing of keys can be preserved everywhere
Signed-off-by: Andres Taylor <andres@planetscale.com>
* test: improve assertion
Signed-off-by: Andres Taylor <andres@planetscale.com>
* chore: clean up accidental change
Signed-off-by: Andres Taylor <andres@planetscale.com>
* refactor: clean up so more responsibility for vtexplain ends up in the engine primitive
Signed-off-by: Andres Taylor <andres@planetscale.com>
* fix: no need to Unquote multiple times
Signed-off-by: Andres Taylor <andres@planetscale.com>
* fix: rename comment directive
Signed-off-by: Andres Taylor <andres@planetscale.com>
Co-authored-by: Harshit Gangal <harshit@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* Add support for MySQL 8.0 in backup tests
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add 8.0 workflow
Signed-off-by: Matt Lord <mattalord@gmail.com>
* whitespace
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync
This needed to be done after the shard was setup in order to satisfy
the semantic assumptions related to semi-sync in the tests.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove extraneous changes
Signed-off-by: Matt Lord <mattalord@gmail.com>
* We need lz4 for TestXtrabackupStreamWithlz4Compression
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Specify stream type everywhere
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove repeated server install
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Moar...
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Move vtctlbackup test to 8.0
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename vtbackup test and move to MySQL 8
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Split the xbstream tests so the workflow doesn't time out
Otherwise it was going over the 10min limit and getting
killed.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use MySQL 8 compat method for setting passwords
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Test increasing timeout at another level
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Don't use the init passwords file with 8.0
mysqlctl doesn't start...
This is likely due to the change in initialization behavior with
MySQL 8.0 as it goes though two phases and you can't simply
start up mysqld and pass it data, it has to initialize and restart
first.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Fix incorrect password update statement for vt_repl user
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Bump timeouts for 8.0 backup tests
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Bump it more :(
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Increase backup/restore timeout in backup_utils
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Apply new 8.0 template everywhere
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Fix bugs around how the compression flags were getting passed
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use 45m timeout for the workflow, 30m for the run.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* These changes were no longer needed so limiting diff.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Explicitly skip new linter check
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
* feat: add parsing for bitnums
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add invalid cases to tests
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: normalize bitnums to bit vals too
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add normalization for bit literals
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: parameterize binary value to hex
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: added e2e test in vtgate and vttablet
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: fix the type conversion from hexnum and hexval to binary
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: refactor test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: return varbinary and flaghex for hexval and hexnum in eval engine typeOf
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: separate expectation for mysql and vitess result
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: fix test expectation
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: added bitnum bind variable and test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* proto: vtadmin side update
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: fixed test and added more bitnum bind var test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* added license header
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* Increase timeouts for upgrade_downgrade workflows
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove errant addition
Signed-off-by: Matt Lord <mattalord@gmail.com>
* test: fix flakiness in TestFloatValueDefault by adding ordering to the query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove tabletgateway_cellalias config since it points to a directory which doesn't exist
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove redundant test which tries to run test that doesn't exist
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Fix bug where update statements were not converting the timezones if source_timezone was specified
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Try moving the conversion test to the migrate/vdiff2 workflow.
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* Delete all legacy sharding related code
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Move used until functions from initialsharding to cluster
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtctl commands
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Kill vtworker and SetKeyspaceServedFrom cmd
Signed-off-by: Matt Lord <mattalord@gmail.com>
* WaitForDrain related stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Legacy local straggler workflow
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtworker stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Update throttlerservice protobuf
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove last mentiones of legacy sharding in vtctl
Signed-off-by: Matt Lord <mattalord@gmail.com>
* remove binlog_use_v3_resharding_mode
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct vtgate help output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* feat: use the new RPC for resetting replication
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add code to get the full status and use it
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove uptime from instance propoerties in VTOrc
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add end to end test for checking the output of ReadInstanceBufferable
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove a mysql query for server id, etc and use full status information instead
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use primary status output instead of mysql query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove semi-sync available, since it is unused
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use semi sync enable from full status instead of mysql query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use semi sync statuses from full status instead of mysql query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: check output of more fields in the test
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove ReplicationCredentialsAvailable since it is unused
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use executed, purged gtid set, server uuid, gtid mode and binlog row image from full status instead of mysql query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add testing for replication status fields
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: check for nil values to prevent panics when some fields are empty
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use replication status from full status instead of mysql query
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add license and shorten package name
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: added failing e2e test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: proto changes for vindex_only shard session
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* session to ignore multi-db transaction error if shard session is used only for vindex query
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: added additional e2e tests that fails
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: claim a vindex only session if other query uses it
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* refactor: addressed all review comments
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* VDiff2: rearchitect VDiff to run on tablets
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve show summary output and add 'show last'
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve e2e test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* go mod tidied
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* regenerate vtadmin protos
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks/improvements to the e2e test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use UUID types directly and remove string literals
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* You guessed it!! More minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improve handling of last and all with no results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove double printing for show all output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Return empty json array for empty show all results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add vdiff2 test to CI
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Trigger rebuild
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Another round of tweaks during review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Final round of code tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct has_mismatch handling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Adjust reporting for extra/mismatched row samples
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use proto.Clone to copy VStreamRowsResponse
And use small vstream_packet_size in e2e test.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor tests. Call VDiff2 side by side with all VDiff1s. Currently failing m2m3 test workflow.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add doc.go to package and use a const for trunctated notation
And rename readme.md to README.md so that it is listed first
in lexicographical order.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct year in header for new doc.go file 🤦
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Fix bug where we were not using in_keyrange() for filtering from source. Change logic to update table level mismatch flag to post-reconciliation of extra rows. Enhance vdiff2 test to include split/merge and full merge (working, but needs code refactoring).
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use unbuffered channels in VDiff2 goroutine comms
TL;DR — I think it was because we were using buffered channels in the table
differ. So the order was not guaranteed and could be off by 1. I suspect the
deep copy synchronized things as a side effect.
I think it may also explain why even after the previous test correction
to return in the wait after sending true to the results channel, vtctlclient
would sometimes get another result after things were torn down. Because the
channels were buffered, where the comment says that we block, we didn’t
until there was 1+1 messages sent to the channel and the order was not
enforced between the goroutines.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Increase vdiff timeout
Signed-off-by: Matt Lord <mattalord@gmail.com>
* VDiff2 does not support inter cluster diffs today
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Go back to using buffered chans for the vstreams
Added a comment explaining why we *want* to do the Clone here.
If we need other synchronization points in the code paths then we
can handle that independently.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor vdiff2 tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks after final review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct test config after merging main branch
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add VDiff2 to the v14 release notes summary
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Help output should not show --v2 as defaulting to true
And add note about how to use VDiff2 with example usage to the
release notes summary.
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* Improved schema tracking E2E test with PRS
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Keep track of current tablet information everytime we receive a healthcheck response
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Use the target tablet type in the update_controller of schema tracking
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Add new E2E test to config.json
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Remove fix in the healthcheck
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* reduced the size of cluster in schema tracking PRS sharded E2E test
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Do not test TestAddColumn is vtgate < v14
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Retry waitForVTGateAndVTTablet when WaitForTabletsToHealthyInVtgate returns an error
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
* Wait for vtgate and tablets to be healthy in prefixfanout tests
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Setup already waits for vtgate proceess to be healthy
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Let's give tablets more time to become healthy
Sometimes GitHub Actions is *super* slow and our tests should
still be able to pass.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Wait longer and check more frequently
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Mark vtgate_vindex test as heavy
Signed-off-by: Matt Lord <mattalord@gmail.com>
* 60s is a more than reasonable upper limit in tablet+mysqld startup
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Also rename 17->vtgate_general and mark as heavy
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Update test config to match workflow renames
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename 20 to xb_backup
And get related files aligned
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Actually wait for all tablets in all shards to be healthy
We were waiting for 1 replica tablet when the clsuter defined
for the test did not have any replica tablets.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* We need to reset the replica and rdonly table counts for each shard
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add log msg and get rid of extra flags added
Signed-off-by: Matt Lord <mattalord@gmail.com>
* run CI tests again
Signed-off-by: Matt Lord <mattalord@gmail.com>
* run CI tests one last time for goodness sake
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor correction to new Info log msg
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Support throttling vstream work on source tablets
This provides mechanisms to limit the impact of large vreplication
workflows on production source tablets.
It supports throttling based on the InnoDB history list length and
the replication lag seen.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add unit and e2e test for copy table throttling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use shorter internal name for max replica lag seconds
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use consistent case for json vars in config test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add MySQL specific context to flags
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add copy table row streamer stats
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rearrange things to wait for MySQL in initial row stream and copy phase cycles
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct & improve on e2e and unit tests
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Update test config
I spent way too much time trying to get both tests to work within
the same go test run... giving up for now.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Adding additional stats work
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add ability to modify the RowStreamerConfig at runtime via /debug/env
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add current RowStreamerConfig values to /debug/vars
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add waitingForMySQL stats to vreplication phase timings
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Track waitForMySQL by table on vstreamer side
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Move row streamer config export to vstreamer engine
And make it a gauge so that it's always showing the current value
as we allow this to be changed in the running process via /debug/env.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Shorten/generalize InnoDB row history test func names
Signed-off-by: Matt Lord <mattalord@gmail.com>
* fix: when mysql is down and vttablet is up
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: in reserved conn with no transaction if the queryservice i.e vttablet is down then it should go through gateway
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* refactor: execute and streamExecute should be able to reset the shard if needed and not executeLock
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: e2e test for bug fix
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: when mysql is down, query should go through gateway to elect new healthy tablet for the given tablet_type
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* ci: make vtorc tests run against mysql 8.0
Signed-off-by: Manan Gupta <manan@planetscale.com>
* CI: run vtorc tests on both mysql versions
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Rename cluster test 23 and set LimitResourceUsage for it
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename shards 11,14,16 and set LimitResourceUsage for them
Signed-off-by: Matt Lord <mattalord@gmail.com>
Rohit Nayak
Harshit Gangal
Manan Gupta
Matt Lord
Andres Taylor
Rameez Sajwani
FlorentP
Shlomi Noach
Rohit Nayak
Florent Poinsard
Deepthi Sigireddi
Manan Gupta