This will fix the recent test failures and undo the change from https://github.com/youtube/vitess/issues/3203 where we had to disable the tests "unit" and "mysql_server" because they were constantly failing. With this change, the two tests are enabled again.
More importantly, with this change we no longer have to maintain Vitess' dependencies in the Travis CI environment.
Instead, we'll depend on our "bootstrap" Docker images to run the test.
Note that I had to come up with a caching mechanism for "make build". By default, we provide the test within the Docker container with a clean copy of the Git repository and run "make build" before *each* test i.e. a Travis build with 10 tests in it would have run "make build" 10 times. To avoid this, I added a new option "-use_docker_cache" to test.go which will run "make build" before all tests and use "docker commit" to create a new temporary cache image.
Overall, this change has several consequences:
- Travis CI builds will be dependent on an up to date Docker image. For example, if somebody adds a dependency to vendor/vendor.json and forgets to rebuild and push the used Docker image, "make build" will fail. I'll add a test for this in a separate commit.
- We'll use MySQL 5.7 (instead of MariaDB) for the Travis test runs.
- We can no longer use Travis container infrastructure and have to use the VM infrastructure instead. That means an additional 20 seconds wait time until Travis has started a VM and runs the build.
- More overhead per Travis build because we need to pull the Docker image first (takes up to two minutes). Before, we used Travis' caching mechanism to cache dependencies between builds.
- Slightly slower tests because the VM infrastructure has less resources than the container one (~2 CPUs (bursted) instead of 2 CPUs, see https://docs.travis-ci.com/user/reference/overview/).
- The ".git" directory will be copied into the Docker image as well now because travis/check_make_proto.sh needs it. I'll create a separate commit to integrate this.
Please refer to #2694 and #2670 for motivation and reasoning for
this change.
I've tried to follow best practice in inserting the copyright
headers. In other open source projects, not all files carry
the notice. For example documentation doesn't. I've followed
similar ground rules.
I did not change the php because there is a separate LICENSE
file there by Pixel Federation. We'll first need to notify
them our intent before changing anything there.
As for the presubmit check, it's going to be non-trivial
because of the number of exceptions, like file types,
directories and generated code. So, it will have to be
a separate project.
The test requires many dependencies which prolong the current test duration by up to 10 minutes on Travis.
Once we figured out how to cache the dependencies both on Travis and our local machines, we can re-enable the test.
Instead of commenting the code for installing the dependencies, I've moved it to bootstrap_web.sh for now.
The Makefile previously listed tests explicitly for groups like
site_test and worker_test. These lists got out of date when tests were
removed from test/config.json, and the make rules broke. Now the groups
are defined in config.json itself, so there is one place to update
everything.
I can't get signals to work on the child processes launched by bash,
even if I use a helper script that explicitly signals its children.
That meant that test timeouts didn't actually terminate the test in
non-Docker mode.
We were only using bash to split the command line into args anyway.
Instead, let's just have the config give us the args pre-split.
Sending signals without bash in the way seems to work better.
This is not actually a shebang. Go doesn't support those.
This hack only works when run from within a shell.
The file gets treated as a shell script, and the first line
replaces the process with an invocation of "go run".
This can be used, for example, to check how often a test fails:
```
$ go run test.go -retry 1 -runs 100 some_flaky_test
...
42 PASSED, 0 FLAKY, 58 FAILED, 0 SKIPPED
```
Previously, each test would freshly re-copy the working repo before
running. Changes to the files could affect the next test in the series.
Now the entire invocation of test.go uses the same snapshot of the
working dir.
This required a global cleanup step, to delete the temp dir, so I
reworked signal handling to be global. This also opens the door to
running tests in parallel within a given test.go invocation.
This is an alternative to 'make integration_test',
with the following advantages:
* Tests run in Docker, so no bootstrap is necessary.
* Tests are hermetic and can run in parallel.
* Test against different flavors just by setting a flag.
* Failing tests are retried to see if they are flaky.
* A failed test will be recorded for later inspection, while the script
continues to run other tests.
* A test that takes too long will be considered stuck and retried.
There's plenty of room for improvement, but now that we have something
in a more readable language than Makefile, we can iterate.
David Weitzman
Anthony Yeh
Alain Jobart
Michael Berlin
Sugu Sougoumarane
Robert Navarro
Joshua Thompson
Shengzhe Yao