* Initial Implementation of partial movetables. Implement cross-shard queries that obey shard routing rules.
Refactor rss updation with routed shard and add to multi col resolution. Run vtgate and reservedconn tests for both regular and with a partially moved shard.
Add ability to run tests with partial keyspaces by adding to a list in the CI generator. Implement that in test.go and cluster setup. Revert previous iteration of running partial keyspace tests in parallel. One test is failing: TestSetSystemVarAutocommitWithConnError.
Add partial vtgate misc test
Add partial keyspace tests to vtgate transaction
Add vtgate queries partial keyspace tests
Review and fix skipped tests where applicable. Refactor to allow conditional skipping of tests for partial keyspaces only so that other tests work exactly as earlier without specifying the DbName, for example
Add move shards example
Cache routing tables and minor refactor. Move move_shards example
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Add lookup index to move shards example. Comment error logs while trying to delete from missing _vt.vdiff table
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Remove move shards demo example
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Add vtgate flag to enable shard routing. Update tests to get vtparams based on partial keyspace setting, so that global routing continues to be tested as earlier without DbName being specified
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
To create subsequent partial movetables workflows we should ignore previous partial movetables (now frozen)
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Remove vdiff2 deletes since they were failing unit tests etc
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Use vtgate flag to enable shard routing rules
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix incorrect test setup
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Expect modified frozen query
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
set vtgate param --enable_shard_routing in partial movetables test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix failing tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Fix flags test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Use VTParams helper for partial keyspace tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Specify vtParams.DBName for partial keyspace queries
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Self-review. Remove bypass caching.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Report partial SwitchTraffic results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Try to address most of the upstream PR review comments
From: https://github.com/vitessio/vitess/pull/9987
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Enforce proper routing for partial migrations
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Account for local timezone in check for vdiff progress
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Update go version in workflows. Fix gofmt issue
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix failing CI tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Comment failing test for partial keyspaces
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix TestVtGateVtExplain by specifying dbname
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix leftover debug code
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Move partial keyspace tests to a single shard. Refactor test.go to use multiple config.json files
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve test skip comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* gofmt
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix failing tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Try TestFunctionInDefault for partial keyspaces. Update skip comment.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Delete temporary log lines
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* Add tracking session state changes for transaction start
This logic depends on the custom option that PlanetScale as in it's
MySQL fork to get the current committed GTID at the start of a
transaction snapshot.
With this GTID, we can optimize vreplication to avoid having to lock
tables around vreplication operations.
See
9fa67ec3eb
for the change in MySQL that is published in our fork.
The change here allows getting back this value through any query
operation that might start a transaction.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Use 'session_track_gtids = START_GTID' where possible, and avoid LOCK tables while still getting a transaction consistent GTID
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
* feat: simplify replication status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: reserve the two fields on proto
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: fix typing error to use correct variable
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: remove test which is no longer required
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Minor refactoring and stop action handler scaffolding
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Initial implementation of Stop action
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add e2e test for Stop action
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove unnecessary conditional
uuid.Parse() with an empty string returns an error.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Don't mark a completed VDiff as stopped
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Need to use withDDL.ExecIgnore in the vdiff retry loop
to avoid potentially doing DDL during tablet state transitions
which can cause the transition to be slow or even hang.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Don't allow resume if VDiff not completed.
An uncompleted vdiff should only be retried, not resumed.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Record and display VDiff errors per shard
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Better align errors in text based output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Work with underlying database errors
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Retry when appropriate on vdiff engine startup
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address bugs in resume/retry logic after adding support for both
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Make auto-retry the default
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Ensure report is valid json before unmarshaling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Auto retry error'd VDiffs
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Don't retry error'd VDiffs on engine start anymore
We now have a goroutine that will do this periodically
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Limit withDDL usage to entry points
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add e2e test and fix bugs it exposed
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Ensure we always signal the retry goroutine to stop on engine.Close()
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use vdiff engine mutex during retries
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Close retry goroutine on vde ctx cancel w/o done channel
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Open & Close of VDiff engine controls retry goroutine
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rely on sync.Once to apply VDiff schema
And avoid withDDL usage anywhere else.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor change after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Tidy up vdiff retry goroutine mgmt
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improving gorouting mgmt -- trying to eliminate flakes
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Moar safety
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Making more tweaks to exec and term more quickly
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Aye dios mio
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Go back to 30s ticker
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Make engine open more efficient to improve PRS times
Otherwise PRS was timing out too often waiting for the replicas to
catch up with the new promoted primary.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* isOpen=true before initControllers to ensure proper Close
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Do lazy init in 2 entry points when doing actual work.
Check for cancelled context in the controller.run code path.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Bug fixes and improvements
TODO: figure out how we should be tracking progress for merges where
there's a single target shard -- and thus a single
_vt.vdiff_table record -- but > 1 source shard.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Final (🤞) set of bug fixes
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add progress reporting
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improved template for error handling in text format
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Bug fixes around progress calculation
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Comment improvement
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Ensure we get correct DB name from vreplication workflow.
And quickly timeout the receive attempted on the controller's
done channel when checking if it's active.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Bug fixes and improvements to progress reporting
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes after self review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove unnecessary case in select
We'll never hit that as the first one will receive immediately as
the channel was closed or we'll hit the default.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add unit test for progress reporting
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Be more realistic in half way unit test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use more descriptive var names in unit test to make logic clear
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Protect all access to vde.controllers
Add comment about safe usage of vde.addController()
Signed-off-by: Matt Lord <mattalord@gmail.com>
* De-flake ers_prs tests using state waits with timeouts
This is a test of the fix in:
https://github.com/vitessio/vitess/pull/10720
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor improvement -- also to exec the test suite again
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use simpler defer stmt
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct VDiff final state handling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improve WorkflowDiffer logging & error handling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use withDDL.ExecIgnore during engine startup
The VReplication engine does the same.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* feat: add parsing for bitnums
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add invalid cases to tests
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: normalize bitnums to bit vals too
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add normalization for bit literals
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: parameterize binary value to hex
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: added e2e test in vtgate and vttablet
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: fix the type conversion from hexnum and hexval to binary
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: refactor test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: return varbinary and flaghex for hexval and hexnum in eval engine typeOf
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: separate expectation for mysql and vitess result
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: fix test expectation
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: added bitnum bind variable and test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* proto: vtadmin side update
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: fixed test and added more bitnum bind var test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* added license header
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* Delete all legacy sharding related code
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Move used until functions from initialsharding to cluster
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtctl commands
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Kill vtworker and SetKeyspaceServedFrom cmd
Signed-off-by: Matt Lord <mattalord@gmail.com>
* WaitForDrain related stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Legacy local straggler workflow
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtworker stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Update throttlerservice protobuf
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove last mentiones of legacy sharding in vtctl
Signed-off-by: Matt Lord <mattalord@gmail.com>
* remove binlog_use_v3_resharding_mode
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct vtgate help output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* lint: add a new line to proto file and generate generated file
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix workflow to also setup go when there are proto changes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add vttablet rpc to reset replication parameters
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: added end to end testing for the rpc and fixed bug
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix typing error
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add basic full status rpc functionality and add test for it
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add all the fields needed in full status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: moved the test to reparent tests and improved it
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: bug fix for no replication status and no primary status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add version to the full status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add binlog information to full status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix the comment explaining the binlog information
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add semi-sync statuses to full status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: call the correct command
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add server uuid and id to full status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: make server_id a uint32 to accept the correct range of values
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add few more fields to the full status like version comment, semi-sync settings, binlog_row_image
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: generate vtadmin proto files
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add assertion to check binlog row format is read correctly
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: split GTID mode in its own function because mariadb doesn't support it
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix parsing of empty mariadb gtid set
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: add doucmentation for existing fields in ReplicationStatus
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add relay log file position to the replication status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: augmented full status test to check all the different positions stored
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add additional fields to replication status and read source user
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: read sql delay from show replica status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: read ssl allowed from show replica status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: read has replication filters from show replica status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: read auto position and using gtid from show replica status output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add replication lag unknown too to replication status
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: return nils from replication and primary postiion if it is not present
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: rename FileRelayLogPosition in replication status output to RelayLogSourceBinLogEquivalentPosition and augment test to make sure rpc changes are backward compatible
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: update vtadmin proto files
Signed-off-by: Manan Gupta <manan@planetscale.com>
* refactor: rename BinLog to binlog in renamed proto field
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: added failing e2e test
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: proto changes for vindex_only shard session
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* session to ignore multi-db transaction error if shard session is used only for vindex query
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: added additional e2e tests that fails
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: claim a vindex only session if other query uses it
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* refactor: addressed all review comments
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* VDiff2: rearchitect VDiff to run on tablets
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve show summary output and add 'show last'
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve e2e test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* go mod tidied
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* regenerate vtadmin protos
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks/improvements to the e2e test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use UUID types directly and remove string literals
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* You guessed it!! More minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improve handling of last and all with no results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove double printing for show all output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Return empty json array for empty show all results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add vdiff2 test to CI
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Trigger rebuild
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Another round of tweaks during review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Final round of code tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct has_mismatch handling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Adjust reporting for extra/mismatched row samples
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use proto.Clone to copy VStreamRowsResponse
And use small vstream_packet_size in e2e test.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor tests. Call VDiff2 side by side with all VDiff1s. Currently failing m2m3 test workflow.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add doc.go to package and use a const for trunctated notation
And rename readme.md to README.md so that it is listed first
in lexicographical order.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct year in header for new doc.go file 🤦
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Fix bug where we were not using in_keyrange() for filtering from source. Change logic to update table level mismatch flag to post-reconciliation of extra rows. Enhance vdiff2 test to include split/merge and full merge (working, but needs code refactoring).
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use unbuffered channels in VDiff2 goroutine comms
TL;DR — I think it was because we were using buffered channels in the table
differ. So the order was not guaranteed and could be off by 1. I suspect the
deep copy synchronized things as a side effect.
I think it may also explain why even after the previous test correction
to return in the wait after sending true to the results channel, vtctlclient
would sometimes get another result after things were torn down. Because the
channels were buffered, where the comment says that we block, we didn’t
until there was 1+1 messages sent to the channel and the order was not
enforced between the goroutines.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Increase vdiff timeout
Signed-off-by: Matt Lord <mattalord@gmail.com>
* VDiff2 does not support inter cluster diffs today
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Go back to using buffered chans for the vstreams
Added a comment explaining why we *want* to do the Clone here.
If we need other synchronization points in the code paths then we
can handle that independently.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor vdiff2 tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks after final review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct test config after merging main branch
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add VDiff2 to the v14 release notes summary
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Help output should not show --v2 as defaulting to true
And add note about how to use VDiff2 with example usage to the
release notes summary.
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* feat: parse advisory lock functions
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: added new tokens to keywords file
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* refactor: planner to use new locking func struct
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: add locking functions provided in the select expressions to the lock engine primitive
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: block mixing of locking function with other expression in dual query
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: proto changes to incluse advisory lock info
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: changed unsafe pooling logic based on new locking function parsing
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: advisory lock implementation to track and remove the lock and close lock session when no lock held
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: support only liternal string in lock name
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: allow lock functions to be executed outside of reserved connections except for get_lock
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: use evalengine to evaluate the name from the ast expression
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: allow impossible query to execute out of reserved connection for pool unsafe queries
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* fix: getfields for lock function and name to be extracted as evalengine.Expr in plan time
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* test: added e2e test from #10130
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* feat: change endAction to endLockAction
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* refactor: for backward compatibility
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* addressed review comments
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
* [vtadmin] Rename reparent-related RPCs, endpoints, methods, RBAC actions
- `EmergencyReparentShard` => `EmergencyFailoverShard`
- `PlannedReparentShard` => `PlannedFailoverShard`
- `ReparentTablet` => `RefreshTabletReplicationSource`
- `TabletExternallyReparented` => `TabletExternallyPromoted`
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Update vtadmin-web code that referenced old type
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* fixup! Update vtadmin-web code that referenced old type
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Remove confusing comments
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* figured out how to run the lint-fixer
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add parameter Cells in vstream api flags, for tablet picker to look for candidates in
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add e2e test for the new Cells vstream flag
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Self-review
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* feat: add durability_policy to the keyspace information in the topo server
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add durability_policy parameter to the CreateKeyspace command in vtctl and vtctld
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add tests for durability_policy field in the keyspace information
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add a new rpc to update the durability policy in the keyspace information
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add testing for the newly added rpc SetKeyspaceDurabilityPolicy
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: changed the e2e test to also test the vtctldclient interface
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: add a comment explaining the test objective
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: fix vtctld api test to expect durability_policy as a new returned value in the keyspace information
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix the help messages for the new command
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix casing and grammar in the flag description
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: change flag to be hyphenated instead of underscored
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix the year of license of new file
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove the vtctl command
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: remove the test for the deleted command
Signed-off-by: Manan Gupta <manan@planetscale.com>
* refactor: fix the alphabetic order of functions and rpc registrations
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix ordering in vtctld file too
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: improve the long help message
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix typing error
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: added documentation for keyspace type
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Add protos for remaining reparent actions and run generators
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Initial reparent-actions implementations
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add pools and configs for reparent operations
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Update impls to use new pools
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* jk, TER only updates the topo
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add http wrappers
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add unit tests
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Update local rbac to allow reparents
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Start adding proto messages
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix ordering
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* fixup! Start adding proto messages
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Implement cluster-level schema reloading
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Implement api method handler
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add http handler wrappers
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* default to "all keyspaces" when nothing else set
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add tests for reloadKeyspaceSchemas
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add tests for reloadShardSchemas
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add tests for reloadSchemaTablets
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* always forgetting about rbac
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add docs for proto sub-messages
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add 'allow_primary' to vtadmin.proto's DeleteTabletRequest
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Update vtadmin-api to pass through allow_primary
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Update UI to set allow_primary=true
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Update wording for notification on a successful DELETE request
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Add Advanced.test.tsx + deletion tests
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Update UI wording: allow_master -> allow_primary
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Fix lint warnings
Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>
* Add Vars type
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* [vtadmin] Tablet APIs string alias -> typed alias
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix assertion to handle proto messages
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Cleanup redundant filtering
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* This effectively reverts 65226ad1d77629fe29e852393a19250a9a910024
It was NOT backwards compatible.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* This adds the new state fields back so that we can switch to them in v15
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Allow for safe/smooth upgrades within 14.0-SNAPSHOT
We are only appending the last io_thread_connecting field
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Avoid intermediate io_thread_connecting protobuf field
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Continue using replication states in v14 tests
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Translate SQL running status to state as well
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use backward compat ReplicaWasRunning check
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add backward compat SQLThreadWasRunning function
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add comment about when backward compat can be removed
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Support older clients with new tablets
Signed-off-by: Matt Lord <mattalord@gmail.com>
* feat: remove SQLThreadWasRunning unused function
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add an upgrade test to verify the replicationstatus is backward compatible
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix ReplicaWasRunning so that it doesn't have code dependent on the upgrade
Signed-off-by: Manan Gupta <manan@planetscale.com>
* refactor: rename function to reflect the output type
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* Add e2e test case for source and target in different time zones
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Added source_time_zone option to MoveTables v2, pass it through to table plan builder and convert datetime columns if option is provided. e2e test demonstrates this works. VDiff needs to be modified
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add time zone support in VDiff. Improve tests.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add vdiff unit test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add SourceTimeZone to Workflow Show
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Update workflow test for added attribute SourceTimeZone
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Support SourceTimeZone in reverse workflows, update e2e test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Update unit tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Refactor vdiff timezone-related changes
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comment nits
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Fix: lower col name also for time zone cols
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use field type for type comparison instead of the string representation
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Check if current primaries have time zone loaded
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Address review comments
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use same date for conversion and as date format
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use lower tablet refresh timeout in vreplication traffic switching
And refresh all tablets in dry runs to alert the user to potential
issues.
* Check primary tablet health in the traffic switcher pre-check
* Add msgs about specific tablets that failed refresh to cmd output
* Keep function semantics the same but provide details if wanted
This keeps the previous behavior of ignoring partial refresh related
errors, while providing the defails of WHY we had a partial refresh
for anyone that's interested.
* Add partial refresh details to response message
This way the caller has that info and can decide what to do with
it (if anything).
* Update tests and vtadmin for updated refresh response message
* Cancel switch writes on partial tablet refresh when safe
Signed-off-by: Matt Lord <mattalord@gmail.com>
* [vtctldserver] Migrate GetPermissions PR#9903
This PR migrates GetPermissions rpc to VtctldServer
Run it locally: vtctldclient --server ":15999" GetPermissions zone1-100
unit test: go test vitess.io/vitess/go/vt/vtctl/grpcvtctldserver -run TestGetPermissions
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
* [vtctldserver] Migrate GetPermissions PR#9903
This PR migrates GetPermissions rpc to VtctldServer
Run it locally: vtctldclient --server ":15999" GetPermissions zone1-100
unit test: go test vitess.io/vitess/go/vt/vtctl/grpcvtctldserver -run TestGetPermissions
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
* [vtctldserver] Migrate GetPermissions PR#9903
This PR migrates GetPermissions rpc to VtctldServer
Run it locally: vtctldclient --server ":15999" GetPermissions zone1-100
unit test: go test vitess.io/vitess/go/vt/vtctl/grpcvtctldserver -run TestGetPermissions
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
* [vtctldserver] Migrate GetPermissions PR#9903
This PR migrates GetPermissions rpc to VtctldServer
Run it locally: vtctldclient --server ":15999" GetPermissions zone1-100
unit test: go test vitess.io/vitess/go/vt/vtctl/grpcvtctldserver -run TestGetPermissiions
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
* Add ShardReplication{Add,Remove} proto definitions and generate
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Port implementations to VtctldServer
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add unit tests
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoints
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add ShardReplicationFix proto definitions and run generators
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add `ShardReplicationFix` implementation
No unit tests, because `topo.FixShardReplication` is tested in isolation and we're just a passthrough
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoint
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add SourceShard{Add,Delete} rpcs and generate
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add SourceShard{Add,Delete} implementations
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Port implementation back to wrangler, preserving old error behavior
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add unit tests
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoints
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add RestoreFromBackup proto defs and regenerate
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Port implementation from legacy vtctl command
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoint
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add test cases
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Consider repl NOT running with IO_Thread connecting and last attempt failed
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use replication IO/SQL thread status and last IO/SQL errors in ReplicationStatus
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Minor changes & improvements after review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct GroupReplication flavor unit test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correcting minor mistakes in func comment
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove unintentional behavior changes
And make function names nicer (removing double context)
And add some helper functions
Signed-off-by: Matt Lord <mattalord@gmail.com>
* One more place where we would have slightly changed behavior
Signed-off-by: Matt Lord <mattalord@gmail.com>
* One more spot that keep existing behavior
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Modify the replicationdata proto w/o breaking backwards compat
Update vtadmin web protobufs
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add proto definitions, run generators
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Port implementation to vtctldserver
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Update fakebackupstorage and add test cases
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoint
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Move ShardReplicationStatuses out of wrangler for import
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Move some more functions out of wrangler that we need in grpcvtcldserver
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add BackupShard protos
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Implement BackupShard
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Refactor for shared event stream loop
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add test cases
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoint
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add missing SetReplicationSource code from wrangler version; adjust tests
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Backport new implementation to legacy vtctl commands
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix bad port of `SetReplicationSource`
The ignored error was deliberate here; if there is no primary in the shard,
there is no need to re-point replication on the backup tablet.
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add CLI entrypoint for ApplySchema
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix typo in --ddl_strategy help text
Signed-off-by: Andrew Mason <andrew@planetscale.com>
Frances Thai
vitess-bot[bot]
Phani Raj
Rohit Nayak
Dirkjan Bussink
Manan Gupta
Manan Gupta
Matt Lord
Harshit Gangal
Deepthi Sigireddi
Shlomi Noach
Arvind Murty
Andrew Mason
doeg
rsajwani