* Move vtorc from go-sqlite3 to modernc.org/sqlite (#12214)
* Move vtorc from go-sqlite3 to modernc.org/sqlite
This moves vtorc from the go-sqlite3 library that uses CGO, to use
modernc.org/sqlite which is a pure Go implementation.
vtorc is the only component we have to build with CGO but it's causing
pain for releases since we need to build it against an old Linux for
linking against glibc.
Using modernc.org/sqlite allows for using Go only again and makes all
Vitess components buildable without CGO.
In
https://datastation.multiprocess.io/blog/2022-05-12-sqlite-in-go-with-and-without-cgo.html
someone ran some basic benchmarks. It shows that the pure Go version can
be twice as slow, but the usage of vtorc is very limited and we operate
on small datasets, so I think the performance impact purely of a
somewhat slower sqlite implementation is negligable.
None of this is in a hot query serving path or anything like that, so I
have little concern performance wise.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Fix error handling in RowToArray
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
---------
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* empty-commit
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Run go mod tidy
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
---------
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
This is to pick up the fix for https://github.com/golang/go/issues/56152.
Signed-off-by: Andrew Mason <andrew@planetscale.com>
Signed-off-by: Andrew Mason <andrew@planetscale.com>
Co-authored-by: Andrew Mason <andrew@planetscale.com>
* feat: remove configurations that aren't needed like raft and initialization of database
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use waitreplicastimeout for all timeouts related to waiting for execution on a replica
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove agents in vtorc since we don't use them
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more unused configurations
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: cleanup post and pre processes that VTOrc doesn't intend to support
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove unused code and config for shell process
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove unused configurations
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: convert flags to pflag architecture
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove cli flags which aren't required
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more unused configs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove the old UI of orchestrator and cleanup app and http packages
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove web/vtorc files and fix Dockerfiles to not try to copy them
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix all the examples to not pass in the ListenAddress and not export port for the old UI
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove raft code and command applier
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove unused code and configs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more unused code
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove http configs that are no longer required
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more unused configs and dead code
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more unused configs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove access tokens since they are unused
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove RecoveryPeriodBlockMinutes which was only for backward compatibility
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove unused package
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove replicas in instance since they weren't used anywhere
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: copy over vtorc things that vtgr was using to remove vtgr's dependency on vtorc package
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: change restart function to use tmc rpc calls
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove detection queries for vtorc since we read these from topo-server
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: improve read topology test to also verify the errantGTID set
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use internal function to find difference in gtid sets instead of using the MySQL connection
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove population of group replication information
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove configs related to connecting to MySQL instances since it is no longer required
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove code to connect to MySQL backend since VTOrc now only uses SQLite3
Signed-off-by: Manan Gupta <manan@planetscale.com>
* cleanup: go mod tidy
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove unused/redundant flags, we are down to 0 :)
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove filtering configs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove write buffering and associated config parameters
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove more miscellaneous configs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove one more rejection parameter
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: move discovery queue parameters to be constants
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: move some more configs to be constants
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add flags for all the configurations we have kept
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix imports in main.go
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: fix vtorc test output after so many changes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add release-notes docs for config changes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove index definitions for tables that are already deleted
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add API removal documentation as well to release notes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove duplication of debug pages in release notes
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use some flags in e2e test to verify they work
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use sliceVar for the flag definition of cluster_to_watch
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Cleanup usage of go.rice in favor of go:embed
The usage of go.rice predates the availability of go:embed, but we
should switch to using go:embed instead to ship specific assets like
config files that we need.
go.rice is also incompatible with Go 1.19 and while it might see a fix
in the future, it seems better to go with the recommended Go approach
that is available these days.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Move vtctld to also use `go embed` instead of go.rice
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Remove last rice-box related comments
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Remove config moving
This right now breaks building the actual tests since the tests might
also end up loading the regular code which has a `go embed` and refers
to the package with the config embeds.
This doesn't mean that the config isn't properly included in the
binaries. Also with using `go embed` we have a build time dependency on
the files and we always know the latest is included, so we don't have
the issue of potentially outdated files either.
All in all, it seems simplest to remove this logic and trust that Go
itself works as advertised.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Newer versions don't have the compile issue anymore (see the upstream
issue), so we can update and clean this up.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Update docs for keyspace commands
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* fixup! Update docs for keyspace commands
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Initial autodoc generation
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix linking
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Extract common docgen logic for later reuse
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* run `go mod tidy` again
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* refactor: move out plan into its own file
Signed-off-by: Andres Taylor <andres@planetscale.com>
* refactor: wrap primitive in a PlanResult struct
Signed-off-by: Andres Taylor <andres@planetscale.com>
* test: cleanup tests to move to using assert package
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add a tablesUsed field to logstats and use it in the output
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: wireup the tables used from the planning phase to log stats
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: make sure to keep escapes in json strings
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add tables used field to the plan output
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add tables used to more plan types
Signed-off-by: Andres Taylor <andres@planetscale.com>
* refactor: move logstats to it's own package
Signed-off-by: Andres Taylor <andres@planetscale.com>
* chore: updated to newest version of cmp
Signed-off-by: Andres Taylor <andres@planetscale.com>
* test: refactor
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: simplify logstats printing
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add tables used for INSERT queries
Signed-off-by: Andres Taylor <andres@planetscale.com>
* feat: add ks for dual queries
Signed-off-by: Andres Taylor <andres@planetscale.com>
* comment: add deprecation warning to logstats fields
Signed-off-by: Andres Taylor <andres@planetscale.com>
* docs: add deprecation warning
Signed-off-by: Andres Taylor <andres@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* VDiff2: rearchitect VDiff to run on tablets
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve show summary output and add 'show last'
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Improve e2e test
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* go mod tidied
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* regenerate vtadmin protos
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks/improvements to the e2e test
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use UUID types directly and remove string literals
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Another round of minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* You guessed it!! More minor tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Improve handling of last and all with no results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove double printing for show all output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Return empty json array for empty show all results
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add vdiff2 test to CI
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Trigger rebuild
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Another round of tweaks during review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Final round of code tweaks
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct has_mismatch handling
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Adjust reporting for extra/mismatched row samples
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Use proto.Clone to copy VStreamRowsResponse
And use small vstream_packet_size in e2e test.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor tests. Call VDiff2 side by side with all VDiff1s. Currently failing m2m3 test workflow.
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Add doc.go to package and use a const for trunctated notation
And rename readme.md to README.md so that it is listed first
in lexicographical order.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct year in header for new doc.go file 🤦
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Fix bug where we were not using in_keyrange() for filtering from source. Change logic to update table level mismatch flag to post-reconciliation of extra rows. Enhance vdiff2 test to include split/merge and full merge (working, but needs code refactoring).
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Use unbuffered channels in VDiff2 goroutine comms
TL;DR — I think it was because we were using buffered channels in the table
differ. So the order was not guaranteed and could be off by 1. I suspect the
deep copy synchronized things as a side effect.
I think it may also explain why even after the previous test correction
to return in the wait after sending true to the results channel, vtctlclient
would sometimes get another result after things were torn down. Because the
channels were buffered, where the comment says that we block, we didn’t
until there was 1+1 messages sent to the channel and the order was not
enforced between the goroutines.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Increase vdiff timeout
Signed-off-by: Matt Lord <mattalord@gmail.com>
* VDiff2 does not support inter cluster diffs today
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Go back to using buffered chans for the vstreams
Added a comment explaining why we *want* to do the Clone here.
If we need other synchronization points in the code paths then we
can handle that independently.
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Refactor vdiff2 tests
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
* Minor tweaks after final review
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct test config after merging main branch
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Add VDiff2 to the v14 release notes summary
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Help output should not show --v2 as defaulting to true
And add note about how to use VDiff2 with example usage to the
release notes summary.
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
* Move error definitions to separate file
This starts the refactoring for more structured errors by moving it into
a separate file first.
Also removes a few unused error types.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Update single usage of github.com/pkg/errors to errors
This moves the single usage of the github.com/pkg/errors package to use
regular errors.
The rest is all in schemadiff and will be cleaned up with structured
errors. It removes the package as a direct dependency (but still other
Vitess dependencies pull it in).
The usage here does not need the package features at all.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Move column level errors to use structured errors
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Move all apply style errors to structured errors
This moves over all the Apply style errors to be using a structured
version of the error instead.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* More structured errors for parsing failures
This adds a few more structured errors, for example for when we fail to
parse things or handle unknown table options.
Also adds a few more table options the parser knows about but that we
can ignore otherwise but don't want to error on.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Always check complete parsing before other operations
It was still possible to use `NewSchemaFromStatements` to trigger a
panic since it would try to normalize without checking if the statement
was fully parsed.
This moves the check into the helper function used for create new
statements for schemadiff and always ensures we check parse status
before we try to normalize.
Also parse everywhere including in tests with strict mode to show errors
as soon as possible.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Move all to regular errors package
Now that all specific usage for github.com/pkg/errors is removed, let's
remove the package usage and only use regular errors. We don't need it
anymore.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
This removes using an additional UUID package. This was the only place
this package was used, everywhere else uses github.com/google/uuid. Even
this library uses that package as a dependency.
Fewer dependencies is better and also updating github.com/google/uuid
while I was at it.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
This updates to Tengo which is based on latest upstream which includes a
fix for MySQL8 & handling of utf8 (mb3) collations.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
This includes specifically an update to x/sys which is needed because
otherwise it can cause crashes on MacOS with Go 1.17.
See https://github.com/golang/go/issues/47776 for more details.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Florent Poinsard
Dirkjan Bussink
vitess-bot[bot]
Manan Gupta
Phani Raj
Andrew Mason
Andres Taylor
Rameez Sajwani
Harshit Gangal
Tanjin Xu
Rohit Nayak
Shlomi Noach
Vicent Marti
Florent Poinsard
Rohit Nayak
Deepthi Sigireddi
Vicent Martí
AdamKorcz
Andrew Mason
John Watson