(after adding a rule to build before tests).
Global build takes about 2s to build all binaries, and it saves
about 4s in one run of mysqlctl.py, so it's gonna be good!
Currently, the worker tests are hanging from time to time.
Travis kills these tests automatically after 10 minutes (of not
receiving any output). We can be more aggressive about it and kill them
ourselves after 5 minutes.
1. move common dependencies into parent pom.
2. upgrade com.google.guava to 18.0.
3. rename java build target to java_client_test.
4. fix makefile so java client test will run tests from each sub package.
We sporadically see that the worker.py test times out on Travis CI after 10 minutes. Enabling logging will give us better visibility to debug the problem.
1. Get rid of SetupCommand.java and move its logic to TestEnv.
2. Add getRpcClientFactory method in TestEnv.
3. Use setters & getters in TestEnv.
4. define vtgate.test.env and vtgate.rpcclient.factory properties in vtgate-client/pom.xml.
I removed functions from client.py because they need
to be rethought in light of the new API, but missed
removing them from client_test.py.
I've now added client_test to our integration tests
to make sure it doesn't get missed.
I'll submit this right away because it breaks import.
Moved tests out of the "ci_skip_integration_test" target because they don't seem to be so flaky.
Included "ci_skip_integration_test" target in Travis because the tests in there don't seem to be so flaky and we want maximum coverage.
Between 1, 2 and 4, 4 resulted in the shortest total duration. 4 was also better than 2 which is the capped number of availble cores in a Travis CI container.
I suspect that the default is 32 (the number of cores) and by setting it explicitly we effectively reduce the value - but reduce the stress on the system and therefore everything goes faster.
With this change, the following dependencies are no longer installed as well:
- New Relic monitoring (is no longer necessary)
- Java dependencies (no more necessary since we killed most of the old Java code)
Use single line shell command to generate proto file instead of
hard coding command for each proto.
Steps:
1. list all proto files.
2. remove 'proto/' prefix and '.proto' suffix.
3. run protoc for each proto and put in go/vt/proto/${proto_file_name}
cannot have the same <package name>.<data type> as these, or we cannot
load them at the same time. So to fix this:
- renaming the conflicting ones from xxx.proto to xxxdata.proto.
- renaming vtgateservice.VTGate to vtgateservice.Vitess
Note we can still change the names I chose here, just not back to
conflicting ones. If anyone has better ideas, we can implement
in subsequent changes. This is to get the import to google3 unstuck.
The automation framework allows to automate cluster operations which
require a series of manual steps e.g. resharding.
A Cluster Operation has a list of task containers which are processed
sequentially. Each task container can contain one or more tasks which
will be executed in parallel.
Here's an example of a cluster operation with two task containers. The
second task container has two tasks:
- step 1
- step 2a | step 2b
If the task container contains one task, the task can emit new task
containers which will be inserted after the current task container. This
mechanism is used to fully expand Cluster Operations by special tasks
which emit new task containers e.g. "ReshardingTask".
This patchset implements the minimal steps to automate "resharding"
whereas task implementations for "vtctl" and "vtworker" are missing.
These will be added in later, separate commits.
With this change, developers must have "mvn" installed or "make test"
will fail.
Travis CI did already run the test. Now "make test" and Travis CI config are both in sync again.
This is an alternative to 'make integration_test',
with the following advantages:
* Tests run in Docker, so no bootstrap is necessary.
* Tests are hermetic and can run in parallel.
* Test against different flavors just by setting a flag.
* Failing tests are retried to see if they are flaky.
* A failed test will be recorded for later inspection, while the script
continues to run other tests.
* A test that takes too long will be considered stuck and retried.
There's plenty of room for improvement, but now that we have something
in a more readable language than Makefile, we can iterate.
This will let contributors test changes without having to do bootstrap
at all.
It also makes it easy to test against multiple flavors without having to
swap out installed packages. You can even run tests against multiple
flavors simultaneously without interfering, if you have enough RAM.
This makes it faster to use Docker images to test new changes, as
suggested in #492.
Since we don't need to re-bootstrap for every push to master, this also
makes our automated builds faster and more reliable (they often fail due
to network lag when installing dependencies with apt-get).
Lastly, this will make it easier to swap out bootstrap images built for
different flavors, such as MariaDB 10.0 vs MySQL 5.6.
Moving the protos to proto/
Adding a Makefile rule to re-generate the go files from them
(assuming protoc is in the path).
Moving the generated files to directories where the package name
matches the directory name, always easier that way.
Now the bsongen command can live next to the type it operates on.
Note that you need Go 1.4 to run "go generate", but that only needs
to be done when you've modified a type that is the subject of a
generate command.
Make min rdonly instances configurable for vtworker tasks
Remove unnecessary tablets wherever possible
Use lower buffer pool size in MySQL to save memory
Track performance using New Relic
Add travis configuration and dependencies script
Split integration test make targets to allow parallel execution in Travis
Add go packages required for coverage upload to Coveralls
Add coveralls and travis badges to README
Travis: Retry make targets to address flaky targets
Anthony Yeh
Alain Jobart
Dean Yasuda
Michael Berlin
Shengzhe Yao
sougou
Ammar Aijazi
Anand Henry
Anand Henry
Peter Lobsinger
Peter Lobsinger
shrutip