This will fix the recent test failures and undo the change from https://github.com/youtube/vitess/issues/3203 where we had to disable the tests "unit" and "mysql_server" because they were constantly failing. With this change, the two tests are enabled again.
More importantly, with this change we no longer have to maintain Vitess' dependencies in the Travis CI environment.
Instead, we'll depend on our "bootstrap" Docker images to run the test.
Note that I had to come up with a caching mechanism for "make build". By default, we provide the test within the Docker container with a clean copy of the Git repository and run "make build" before *each* test i.e. a Travis build with 10 tests in it would have run "make build" 10 times. To avoid this, I added a new option "-use_docker_cache" to test.go which will run "make build" before all tests and use "docker commit" to create a new temporary cache image.
Overall, this change has several consequences:
- Travis CI builds will be dependent on an up to date Docker image. For example, if somebody adds a dependency to vendor/vendor.json and forgets to rebuild and push the used Docker image, "make build" will fail. I'll add a test for this in a separate commit.
- We'll use MySQL 5.7 (instead of MariaDB) for the Travis test runs.
- We can no longer use Travis container infrastructure and have to use the VM infrastructure instead. That means an additional 20 seconds wait time until Travis has started a VM and runs the build.
- More overhead per Travis build because we need to pull the Docker image first (takes up to two minutes). Before, we used Travis' caching mechanism to cache dependencies between builds.
- Slightly slower tests because the VM infrastructure has less resources than the container one (~2 CPUs (bursted) instead of 2 CPUs, see https://docs.travis-ci.com/user/reference/overview/).
- The ".git" directory will be copied into the Docker image as well now because travis/check_make_proto.sh needs it. I'll create a separate commit to integrate this.
They always fail/timeout. We'll undo this change when we switch to running the tests in Docker on Travis.
Note that I'm undoing the change from https://github.com/youtube/vitess/pull/3205 where I disabled "mysql_server_test" for all executions through test.go.
* Set go_import_path so forks of the project can benefit from Travis CI. Without
this change there are some assumptions in the bootstrap script and also in the
import of packages that won't be hold in a fork of the project and the tests
will break.
Travis CI is in the process of changing all images from Precise to Trusty.
See their announcement: https://blog.travis-ci.com/2017-07-11-trusty-as-default-linux-is-coming
This would break how we install our dependencies.
Note that we're using the Container based infrastructure (sudo: false). They plan to get rid of this combination (Precise + sudo: false) by Sep 5. Before that, we should try "sudo: required" or do the planned longterm change to run the tests within Docker images.
These files unnecessarily trigger an update of the cache which prolongs the test build by ~30 seconds.
I had to generalize the path for the SNAPSHOT jar files because these two files were not matched:
- $HOME/.m2/repository/io/vitess/vitess-client/1.1.0-SNAPSHOT/vitess-client-1.1.0-SNAPSHOT-tests.jar
- $HOME/.m2/repository/io/vitess/vitess-jdbc/1.1.0-SNAPSHOT/vitess-jdbc-1.1.0-SNAPSHOT-fatjar.jar
It has not been working for a while and I was not able to fix it.
Additionally, it recently started to flake e.g. we saw this error:
> Bad response status from coveralls: 422 - {"message":"Couldn't find a
repository matching this job.","error":true}
https://travis-ci.org/youtube/vitess/jobs/206464990
Let's remove it and replace it with something else at a later point.
The new etcd2 topology service is the new more consistent version of the
etcd topology. It has the following improvements:
- Uses the same file structure as zk2topo. It then also uses the generic
vtctld.Explorer as well.
- Uses the new etcd clientv3 library, with new key-value API and leases.
It is not backwards compatible with the old etcd topology.
- Uses a configurable toplevel directory for topology data in global and
local cells.
Vitess 2.1 will still include the old etcd topology client, but it will
be deprecated, and removed from Vitess 2.2. Use topo2topo binary to
upgrade, see documentation.
Implementation details:
- bootstrap.sh now downloads etcd.
- etcd is used in unit tests of the library, instead of using a mock.
- Fixing wrangler.InitTablet error return.
And adding -allow_update to test calls, so they work anyway.
- Running tabletmanager.py for zookeeper, zk2, etcd, etcd2 flavors.
- Ignoring the 'test' tag for vendored libraries. See:
https://github.com/kardianos/govendor#ignoring-build-tags-and-excluding-packages
- in end-to-end tests, restarting etcd to wipe it.
- publish web site.
Before commit 6f10800, the target was always run although its config is set to "Manual: false". In said commit I fixed test.go to respect the "Manual" field for the -shard execution mode as well.
Therefore, Travis needs to explicitly run this test now.
(In general: It's set to manual because it only makes sense to run it on Travis and not e.g. when running the tests locally in a Docker image.)
bootstrap.sh is currently failing in our Travis CI tests because "golint" dropped support for Go 1.5.
While we're at it, let's upgrade to 1.7 instead of 1.6. (After all, 1.7 also comes with a significant speed improvement.)
Export VT_GO_PARALLEL from the Makefile to subprocesses. Fixes the problem that the variable was not used for 'make unit_test_race'.
VT_GO_PARALLEL is used to explicitly set the degree of concurrent packages tested in Travis. We do this because a value of 4 is faster than the default of 2 (for the two available CPUs).
Morgan Tocker
Michael Pawliszyn
Harald Nordgren
Derek Perkins
Anthony Yeh
Michael Berlin
Sugu Sougoumarane
Rafael Chacon
Alain Jobart
Pavel Ivanov
thompsonja
Joshua Thompson