The test requires many dependencies which prolong the current test duration by up to 10 minutes on Travis.
Once we figured out how to cache the dependencies both on Travis and our local machines, we can re-enable the test.
Instead of commenting the code for installing the dependencies, I've moved it to bootstrap_web.sh for now.
The Makefile previously listed tests explicitly for groups like
site_test and worker_test. These lists got out of date when tests were
removed from test/config.json, and the make rules broke. Now the groups
are defined in config.json itself, so there is one place to update
everything.
I can't get signals to work on the child processes launched by bash,
even if I use a helper script that explicitly signals its children.
That meant that test timeouts didn't actually terminate the test in
non-Docker mode.
We were only using bash to split the command line into args anyway.
Instead, let's just have the config give us the args pre-split.
Sending signals without bash in the way seems to work better.
This is not actually a shebang. Go doesn't support those.
This hack only works when run from within a shell.
The file gets treated as a shell script, and the first line
replaces the process with an invocation of "go run".
This can be used, for example, to check how often a test fails:
```
$ go run test.go -retry 1 -runs 100 some_flaky_test
...
42 PASSED, 0 FLAKY, 58 FAILED, 0 SKIPPED
```
Previously, each test would freshly re-copy the working repo before
running. Changes to the files could affect the next test in the series.
Now the entire invocation of test.go uses the same snapshot of the
working dir.
This required a global cleanup step, to delete the temp dir, so I
reworked signal handling to be global. This also opens the door to
running tests in parallel within a given test.go invocation.
This is an alternative to 'make integration_test',
with the following advantages:
* Tests run in Docker, so no bootstrap is necessary.
* Tests are hermetic and can run in parallel.
* Test against different flavors just by setting a flag.
* Failing tests are retried to see if they are flaky.
* A failed test will be recorded for later inspection, while the script
continues to run other tests.
* A test that takes too long will be considered stuck and retried.
There's plenty of room for improvement, but now that we have something
in a more readable language than Makefile, we can iterate.
Michael Berlin
Robert Navarro
Joshua Thompson
Anthony Yeh
Shengzhe Yao