* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
- Adds a new "infrastructure" section to the docs, which describes
architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
docs, to make the documentation easier to navigate.
Since they are out of sync with `Procfile` and long term are best replaced
with one of the solutions in bug 1318020.
The development setup instructions have been updated to stop recommending
gunicorn, since Django's runserver is more appropriate for beginners
(and I'm pretty sure the memory leaks seen previously no longer occur).
To ensure the CI and local environments match stage/production (where
Elasticsearch is not configured) and don't run differing code-paths due
to the various `ELASTICSEARCH_URL` conditionals used in auto-classify.
At such time when work on the fuzzy auto-classify matcher continues,
this can be reverted.
Also remove all Karma support and update the docs to only mention ``Jest``.
One of the test files was testing some AngularJS filters. I converted these
tests to test the equivalent helper functions.
So that their purposes are clearer and it's easier to differentiate
between generated content and files committed to the repository.
* Neutrino build: `build/` -> `.build/`
* Sphinx build: `_build/` -> `.build-docs/`
* Django collectstatic: `treeherder/static/` -> `.django-static/`
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.
Highlights of the upgrade are:
* Major version updates to the individual tools within (such as webpack,
Babel and ESLint), significantly improving performance, fixing
transpilation/minification correctness bugs, adding support for newer
ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
automatic shared/vendor code chunk splitting (no need for the manually
maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
frontend needs, reducing maintenance burden and allowing for easier
future Neutrino upgrades.
In addition this PR:
* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
compilation errors in the browser, saving having to switch back
to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
which saves having to manually navigate to `localhost:5000` after
running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
correctly on Windows, even when setting environment variables.
Performance comparison:
* Local `yarn build`:
- Cached: 2m34s -> 23s
- Uncached: 2m34s -> 58s
* Local `yarn start`:
- Cached: 34.5s -> 13.6s
- Uncached: 34.5s -> 31.3s
* Local `yarn test`
- Cached: 61.5s -> 19.8s
- Uncached: 61.5s -> 22.0s
* Local `yarn lint`
- Cached: 3.8s -> 1.8s
- Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
14 minutes -> 9 minutes
To work around the intermittent MySQL deadlocks encountered during
a small percentage of deployments. This will ensure deployments don't
cause email spam to app collaborators and Heroku admins, as well as
saving us from having to manually retry the release each time it
occurs.
The `IGNORE_PREDEPLOY_ERRORS` option has also been removed, since
it's mostly redundant with `SKIP_PREDEPLOY`.
Runnable jobs for buildbot were calculated via a celerybeat task
(that was disabled in #4007) and the results stored in the
`runnable_jobs` table. This can all be removed now that buildbot is
EOL, since the remaining support for Taskcluster runnable jobs does
not use that celery task/Django model.
Since as of #3980 (bug 1470622) the frontend no longer calls the
`/retrigger/` `/cancel/` or `/cancel_all/` Treeherder APIs.
Whilst looking at the pulse related fixtures, I spotted that the
`mock_message_broker` fixture was already unused.
CommonMark has made a breaking package change in a minor version
release (0.8.0), so we need to pin to the last working release. See:
https://github.com/rtfd/CommonMark-py/issues/134
Fixes:
```
...
File ".../recommonmark/recommonmark/parser.py", line 9, in <module>
from CommonMark import Parser
ImportError: No module named CommonMark
```
The Travis docs setup step has also been modified to always upgrade
dependencies, which will mean we will catch cases like this on Travis
next time.
Now that consumers of OrangeFactor have been switched to the new
intermittent failures view UI/API, we can stop submitting failure
classifications to OrangeFactor's Elasticsearch instance.
* Adds a `rabbitmq-server` install step, since it's no longer
installed in the image by default.
* Removes the MySQL 5.7 install step, since Ubuntu 16.04's default
MySQL is now 5.7 rather than 5.6.
* Adjusts which services are manually started to account for changes
in which are enabled by default in the new image.
This enables `DeprecationWarnings` for things that Python 2 knows
are not compatible with Python 3. The `error` entry in the pytest
`filterwarnings` setting ensures these will be surfaced as Exceptions
and so result in test failures.
See:
https://docs.python.org/2/using/cmdline.html#cmdoption-3
The removal of `sorted()` from `test_bug_job_map_api.py` is to fix:
`DeprecationWarning: dict inequality comparisons not supported in 3.x`
Since:
* Redis has additional features we need (eg for bug 1409679)
* the Redis server, python client and Django backend are more
actively maintained (so have persistent connections/pooling that
actually works, which gives a big perf win)
* we can use Heroku's own Redis addon rather than relying on a
third-party's (to hopefully prevent a repeat of the certificate
expiration downtime)
This commit:
* Switches pylibmc/django-pylibmc to redis-py/django-redis, and
configures the new backend according to:
http://niwinz.github.io/django-redis/latest/https://github.com/andymccurdy/redis-py
* Uses redis-py's native support for TLS to connect to the Heroku
Redis server's stunnel port directly, avoiding the complexity of
using a buildpack to create an stunnel between the dyno and server:
https://devcenter.heroku.com/articles/securing-heroku-redis#connecting-directly-to-stunnel
* Uses explicit `REDIS_URL` values on Travis/Vagrant rather than
relying on the django-environ `default` value (for parity with
how we configure `DATABASE_URL` and others).
* Removes the pylibmc connection-closing workaround from `wsgi.py`.
Note: Whilst the Heroku docs suggest using `django-redis-cache`, it's
not as actively maintained, so we're using `django-redis` instead:
https://devcenter.heroku.com/articles/heroku-redishttps://github.com/niwinz/django-redishttps://github.com/sebleier/django-redis-cache
Before this is merged, Heroku Redis instances will need to be created
for stage/production (prototype done) - likely on plan `premium-3`:
https://elements.heroku.com/addons/heroku-redis
We'll also need to pick an eviction policy:
https://devcenter.heroku.com/articles/heroku-redis#maxmemory-policy
Once deployed, the `memcachier-tls-buildpack` buildpack will no longer
be required and can be removed from prototype/stage/prod, along with
the `TREEHERDER_MEMCACHED` environment variable and Memcachier addon.
The UI has already been removed. This cleans up the data ingestion
and removes the `JobDuration` model, however leaves the `running_eta`
field on the `Job` model for the next time that table is touched (since
the table is large, so altering the schema would likely require
downtime).
Since after my update to the Travis images, the pre-installed version
of Elasticsearch matches the one we're looking for, meaning the
download/install can be skipped, saving time.
Since I've adjusted the Travis images to always use the latest yarn
at the time the images are created, so it's not worth installing the
latest version at every runtime:
https://github.com/travis-ci/travis-cookbooks/pull/935
Since it reduces the intermittent failure rate in bug 1401048 and is
also likely more representative of what people are using to access
Treeherder in production (whilst still being more stable than Nightly).
This:
* reduces duplication
* opens the door to sharing functionality with `vagrant/setup.sh`
* will make it easier to visualise the Travis bootstrap process
when moving both Travis and Vagrant to a unified Docker environment.
set-env was renamed in the buildpack to set_env, breaking this script.
Rather than just renaming our usage, it makes more sense to stop using
what is really an internal buildpack API and instead use `.profile`:
https://devcenter.heroku.com/articles/dynos#the-profile-file
On Heroku, the environment variable `DYNO` is always set in the
environment. As such, we can just use that to determine whether
code is being run on Heroku, rather than having to add our own
environment variable.
To ensure that:
* the Heroku router's 30 second timeout doesn't beat gunicorn to it,
given that routing time is included in Heroku's timing
* the app is more likely to remain responsive, when receiving many
badly filtered API calls
Change of new environment variable `PULSE_PUSH_SOURCES`.
Keep old `publish-resultset-runnable-job-action` task name by creating a
method that points to `publish_push_runnable_job_action`.
All environments are now using the native MySQL 5.7 libmysqlclient
library, so the custom vendoring script is unused. The tests checking
that the library isn't vulnerable aren't needed any more, since
heroku-16 doesn't even have that package installed (it's not available
on Xenial).
Armen Zambrano
Ed Morley
Cameron Dawson
Sarah Clements
Sarah Clements
George Hickman
Ed Morley
Alisha Aneja
Max Chehab