* Update test runners to run all tests including outside package
Sometimes it's necessary to have a separate _test package inside another
package. Our current test runner doesn't run such tests though, leading
to tests being broken without us knowing.
Worst case it hides real bugs. So we should always ensure to also run
these tests as well.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Fix tests
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
* Pre-work tidying: tidy imports, remove commented out code
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* [rowlog] Migrate to pflag, add flag for testing resumability
Closes#11408.
Signed-off-by: Andrew Mason <andrew@planetscale.com>
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* ci: add upgrade test for new vtctl
Signed-off-by: Manan Gupta <manan@planetscale.com>
* ci: add upgrade test for new vttablet
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix vttablet upgrade-downgrade tests to not require the get_upgrade_downgrade_label step
Signed-off-by: Manan Gupta <manan@planetscale.com>
* ci: add tests for query serving for next release
Signed-off-by: Manan Gupta <manan@planetscale.com>
* ci: add tests for backups for next release
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix comment in next release script
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Run latest gofmt on everything & address linter warnings
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rebuild proto impls with new protoc
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Field name is needed for zk
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Make protos with 1.18.4
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Cleanup usage of go.rice in favor of go:embed
The usage of go.rice predates the availability of go:embed, but we
should switch to using go:embed instead to ship specific assets like
config files that we need.
go.rice is also incompatible with Go 1.19 and while it might see a fix
in the future, it seems better to go with the recommended Go approach
that is available these days.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Move vtctld to also use `go embed` instead of go.rice
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Remove last rice-box related comments
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Remove config moving
This right now breaks building the actual tests since the tests might
also end up loading the regular code which has a `go embed` and refers
to the package with the config embeds.
This doesn't mean that the config isn't properly included in the
binaries. Also with using `go embed` we have a build time dependency on
the files and we always know the latest is included, so we don't have
the issue of potentially outdated files either.
All in all, it seems simplest to remove this logic and trust that Go
itself works as advertised.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* feat: use the latest operator yaml file
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add initial script for vtop example
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add script to install kubectl and kind and add a github workflow
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add code to verify vtadmin and vtorc are running correctly
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add move tables and reshard part to the example
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove the inserted row in vtorc check so that move tables isn't affected
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: run the ci test outside docker and skip build since it is already done in a previous step
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: source build.env before running the test in ci
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use custom image names in the test so as to not add image pull policies to the configuration
Signed-off-by: Manan Gupta <manan@planetscale.com>
* ci: remove setup steps that aren't used
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: use the common files instead of creating new ones
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: increase timeout of the ci
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: turned off semi-sync for the test
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: insertion should be retried since VTOrc might take a few seconds to fix it
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: remove the workflow file since it is too heavy
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add double dashed flags and remove configuration for ci
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix readme for cluster example
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: copy init_db.sql file to 101 yaml
Signed-off-by: Manan Gupta <manan@planetscale.com>
* docs: fix comment in helper file
Signed-off-by: Manan Gupta <manan@planetscale.com>
Some tests use t.TempDir to create temp dirs for storing per-test
files, such as MySQL configuration files and logs.
On Mac OS, this can generate long directory names, such as
/var/folders/96/k7gzd7q10zdb749vr02q7sjh0000gn/T/tmp.yqDFqOY3. If during
a test a mysqld socket is put in a long directory like that, mysql
client won't be able to connect to it, and the test will fail.
This commit modifies tools/unit_test_runner.sh so that on Mac OS, as
long as TMPDIR isn't already set, it will export
TMPDIR=/tmp/vttest_XXXXX (where XXXXXX is dynamically replaced with
random characters). Golang uses that as a base for further temporary
files and directories.
Signed-off-by: Max Englander <max@planetscale.com>
* Delete all legacy sharding related code
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Move used until functions from initialsharding to cluster
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtctl commands
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Kill vtworker and SetKeyspaceServedFrom cmd
Signed-off-by: Matt Lord <mattalord@gmail.com>
* WaitForDrain related stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Legacy local straggler workflow
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove vtworker stragglers
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Update throttlerservice protobuf
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Remove last mentiones of legacy sharding in vtctl
Signed-off-by: Matt Lord <mattalord@gmail.com>
* remove binlog_use_v3_resharding_mode
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Address review comments
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Correct vtgate help output
Signed-off-by: Matt Lord <mattalord@gmail.com>
* Parser INTERVAL() function
This adds support for parsing the INTERVAL() function. It adds a single
shift / reduce conflict but the behavior there is what we want to match
the MySQL behavior as wel too.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Add basic test
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* feat: reduce shift-reduce conflicts by using the precedence symbol FUNCTION_CALL_NON_KEYWORD
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: fix static check workflow to setup go in cases of parser changes too
Signed-off-by: Manan Gupta <manan@planetscale.com>
* Add infrastructure for generating authz tests for vtadmin
The lack of verifying authz checks are where they should be is one of the
most glaring issues in vtadmin (in my opinion; it's also my "fault" things
are this way). At the same time, writing all the code by hand to verify
every single endpoint would be a giant pain (which is the main reason
things are this way). So, let's codegen all the bits we don't care about!
The bonus here is that the config.json now can serve as authoritative on
what permissions are required for what endpoints.
The goal here is to have the config primarily specify the rules needed for
each endpoint, with as minimal "overhead" (currently specifying test cases
and mock data) as possible.
I want to separate the introduction of this setup from its complete
adoption, so I will submit a follow-up change that adds the rest of the
endpoint tests.
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* add missing license headers
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Add make target and CI check
Signed-off-by: Andrew Mason <andrew@planetscale.com>
* Fix parsing of encryption attribute for create database
The create database statement also allows for an encryption value to be
set, so this fixes parsing support for that.
It also renames the `CharsetAndCollation` to `DatabaseOption` since
`CharsetAndCollationAndEncryption` seems overdoing things and
`DatabaseOption` more matches how we call this in other places, like a
`TableOption`.
Updates the test to parse unconditional with no magic comments so we
don't depend on the version and the comment logic is already tested
separately somewhere else, so I think it's better to not mix those here.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
* Add comment for shift / reduce conflict and allow extra conflict
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
For more complex integer operations in most modern shells you should
use double parens (shell uses let builtin) or double brackets (shell
uses test builtin) to ensure proper variable expansion and expression
execution (we explicitly use bash today in the build scripts).
In this case the lack of determinism and potential errors comes from
expansion of variables w/o a declared type (`declare -i number` or
`typeset -in`) or implicit type via e.g. `let foo=${bar}+0`. So if
we did not get a valid number from any of the regexp capture groups
(in my case I had go 1.18 installed so no patch version) then the
variable expansion can result in strings getting compared to numbers.
With the single brackets the go check does not work in this scenario
and we see this output from `make build`:
$ make build
Thu Apr 21 11:42:02 EDT 2022: Building source tree
./tools/shell_functions.inc: line 29: [: -lt: unary operator expected
./tools/shell_functions.inc: line 30: [: -gt: unary operator expected
./tools/shell_functions.inc: line 31: [: -lt: unary operator expected
./tools/shell_functions.inc: line 32: [: -gt: unary operator expected
./tools/shell_functions.inc: line 33: [: -lt: unary operator expected
The build will still complete but we do not enforce the go version as
intended. With this change we do (for example, in my case):
$ make build
Thu Apr 21 14:18:40 EDT 2022: Building source tree
ERROR: Go version reported: go version go1.18 darwin/arm64. Version 1.18.1+ required. See https://vitess.io/contributing/build-from-source for install instructions.
make: *** [build] Error 1
Signed-off-by: Matt Lord <mattalord@gmail.com>
* feat: add precedence rules to the grammar to reduce the number of conflicts
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: update the tool checker for make parser
Signed-off-by: Manan Gupta <manan@planetscale.com>
* feat: add parsing support for JSON Schema Validation Functions
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: fix number of expected conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: update number of expected shift/reduce conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* tests: use ignoreNormalizer in the failing tests
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* test: add planner tests for search and validation functions
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add end to end test for json search and validation functions
Signed-off-by: Manan Gupta <manan@planetscale.com>
* test: add tests for shorthands as well
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* feat: add parsing support for search functions
Add the structs, parsing, formatting and tests
JSON_SEARCH and value MEMBER OF not supported yet
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* feat: add parsing support for JSON_SEARCH()
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* feat: change JSONParamPath type from struct to Expr
update parser
update formatting
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* tests: add test for updated parser
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* tests: update invalid test case
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor: remove commented code
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: update number of expected shift/reduce conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor: reorder words in ascending order
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: fix number of expected shift/reduce conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* feat: add parsing support for MEMBER OF
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor: use text_literal_or_arg in place of text_literal
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* Retain full git sha1 instead of the short version
Signed-off-by: Ankit Malpani <malpani@stripe.com>
* Update test to indicate full buildGitRev
Signed-off-by: Ankit Malpani <malpani@stripe.com>
Co-authored-by: Ankit Malpani <malpani@stripe.com>
* feat: add parsing support for JSON value creators functions
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor: use backticks in tests
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* feat: changed array to use pointers instead of structs
Signed-off-by: Manan Gupta <manan@planetscale.com>
* add the functions name in non reserved keywords
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: fix number of expected conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: fix number of expected shift/reduce conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* test: add planner and end to end tests for value creator functions
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
* feat: add parsing support for json utility functions
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor:use backticks in test_cases and remove unused field in JSONUtility struct
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* tests: use backticks for strings
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* refactor: use separate structs for each utility function
Create separate structs and format function for each struct
Update parser to use corresponding structs
Remove previously used JSONUtilityExpr struct
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* ci: fix number of expected conflicts
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
* test: add planner tests for the json utility functions
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
Manan Gupta
vitess-bot[bot]
Rameez Sajwani
FlorentP
Andrew Mason
Florent Poinsard
Matt Lord
Dirkjan Bussink
Max Englander
Rohit Nayak
Kushal Kumar
malpani
Vicent Marti
Manan Gupta