- Adds a new "infrastructure" section to the docs, which describes
architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
docs, to make the documentation easier to navigate.
`manage.py check --deploy` is now run during Travis testing and as part
of stage/prod/Heroku deployment. It checks for a number of common
configuration mistakes & ensures security best practices are being
followed:
https://docs.djangoproject.com/en/1.8/ref/checks/
There are some backwards incompatible changes:
http://whitenoise.evans.io/en/latest/changelog.htmlhttps://github.com/evansd/whitenoise/compare/v2.0.6...v3.0
Specifically:
* The CLI compression utility must now be called via
`python -m whitenoise.compress` rather than `python -m whitenoise.gzip`.
* The `whitenoise.django.GzipManifestStaticFilesStorage` storage backend
has moved to `whitenoise.storage.CompressedManifestStaticFilesStorage`.
* The internal `add_files()` method has been split into two and the part
which we need to subclass is now named `update_files_dictionary()`. See:
07f9c0bece
As of pip 8, peep has now been integrated into pip.
Migrating from peep to this native feature has several advantages:
* It avoids the complexity/learning curve of using a wrapper around pip.
* It means we do not need to fork the official Heroku Python buildpack
(which handles pip installation of requirements files) in order to use
hash verification on Heroku. (Once the buildpack updates to pip 8.)
* Omitted sub-dependencies result in install-time errors rather than
the user discovering omissions at run-time.
* pip's native caching is used, and all packages are installed in one
pip invocation, so it's significantly faster.
* It has better handling of errors and corner cases.
Key facts about the native feature:
* hash-checking mode is enabled if at least one hash is found in the
requirements files passed to pip, or can be force enabled by passing
`--requires-hashes` when running `pip install`.
* Once enabled, hash-checking mode enforces that all packages:
- are pinned to a specific version
- have hashes listed
- have all sub-dependencies specified
* Older versions of pip will error out if either `--require-hashes` or
the requirements file `--hash` syntax is used, meaning it's not
possible to accidentally lose hash-checking protection if the pip used
is older than expected.
For more details, see:
https://pip.pypa.io/en/stable/user_guide/#hash-checking-modehttps://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
The pip version on Travis and in the Vagrant virtualenv has been updated
to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519.
The Heroku Python buildpack pip was updated in bug 1241909.
The requirements files hashes were ported using `peep port`, and then
comments/URLs re-added by hand.
Since they are deprecated and all submitters have switched over to using
Hawk credentials instead.
The automatically created migrations file was edited to remove the
unused `models` import, since otherwise flake8 complains. We could
alternatively exclude the migrations directory from flake8, however we
would then miss linter errors in any hand-written migrations files.
In addition, Django have fixed the issue in 1.9:
a7bc00e17b
Since it should be unnecessary, and causes additional load until the
cache is re-populated, since:
* the builds-{4hr,running,pending} tasks have to ingest all jobs,
including those previously seen
* the pushlog task defaults back to "the last 10 pushes", which in the
case of inactive repos, means extra busy work, and for active repos
means we might actually miss commits (particularly if the prod push
was to fix an issue that has say broken ingestion for the last X mins)
In update.py, the line outputting revision.txt has to be moved later,
since the `dist/` directory won't exist until grunt build has run. In
addition, since `grunt build` removes the entire `dist/` directory, we
no longer need to manually remove *.gz.
We use the `--production` options for both `npm install` and
`grunt build`, so that the `devDependencies` in package.json are
ignored, and we only install/load the ones listed under `dependencies`
in package.json - since that's all that is required for the build.
We have to use `./node_modules/.bin/grunt` rather than `grunt`, since
grunt-cli is not installed globally on the treeherder admin machine for
greater isolation between stage and production.
Also remove unnecessary check for the API key and app id, and make any
errors raise (since it won't affect the deploy itself, but will make
issues more obvious in the IRC bot comments).
On Heroku, there is no load balancer or Varnish-like cache in front of
gunicorn, so we must handle gzipping responses in the app.
In order for WhiteNoise to serve gzipped static content, assets must be
gzipped on disk in advance (doing so on-demand in Python would not be
as performant). WhiteNoise will then serve the `.gz` version of files in
preference to the original, if the client indicated it supported gzip.
For assets covered by Django's collectstatic, gzipping the assets only
requires using WhiteNoise's GzipManifestStaticFilesStorage backend,
which wraps Django's ManifestStaticFilesStorage to create hashed+gzipped
versions of static assets:
http://whitenoise.evans.io/en/latest/django.html#add-gzip-and-caching-support
The collectstatic generated files will then contain the file hash in
their filename, so WhiteNoise can also serve them with a large max-age
to avoid further requests if the file contents have not changed.
For the UI files under `dist/`, we cannot rely on the Django storage
backend, since the directory isn't covered by STATICFILES_DIRS (it is
instead made known to WhiteNoise via `WHITENOISE_ROOT`). As such, files
under `dist/` are gzipped via an additional step during deployment. See:
http://whitenoise.evans.io/en/latest/base.html#gzip-support
Files whose extension is on the blacklist, or that are not >5% smaller
when compressed, are skipped during compression.
Now that the IRC pushbot configs/What's Deployed links have been updated
to use the new revision.txt file location, we can stop creating the file
in the old location.
Since WhiteNoise won't serve files from /media/, so the existing file
in `treeherder/webapp/media/` 404s when accessed via:
https://treeherder.{mozilla,allizom}.org/media/revision
IMO the site root makes more sense for this file anyway, so let's just
save it under `dist/`. Also adds a .txt extension for clarity.
The old file has been left for now to ease the transition, and will be
deleted once the IRC pushbots config and What's Deployed URLs have been
updated.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
For bug 1124278, we're going to want to sprinkle new relic annotations
around the codebase, so by always installing it, we save having to stub
these out in development/on Travis. It also seems wise to have prod
running as close to the same packages as in development.
Since NEW_RELIC_LICENSE_KEY isn't set locally, plus
NEW_RELIC_DEVELOPER_MODE is set to true, the New Relic agent doesn't
submit anything. See:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
The deployment script also had to be updated, so that we set the
prod/stage-specific environment variables before using manage.py, since
dj-database-url cannot rely on what's in the stage/prod local.py config
(which isn't a bad thing, since we're deprecating that file).
Since it only speeds up parsing by a few percent of total runtime, and
is therefore not worth the added complexity for deployment and local
hack-test-debug cycles when working on the log parser.
The .gitignore and update.py entries will be removed in a later commit,
once the stage/prod src directories have been cleaned up.
Since otherwise we may end up with interactive prompts.
Note: When using call_command() we instead have to use 'interactive'
instead of 'noinput' due to https://code.djangoproject.com/ticket/22985,
which is only fixed in Django 1.8+.
There's no need to make multiple calls to peep - we can just combine
them into one. Not changing the puppet instances for Vagrant, since the
calls are made in two separate puppet modules and so would require a bit
of refactoring, which is going to occur in bug 1074151 and friends.
I added a Procfile listing all the different python services treeherder needs.
Heroku provides deployment-specific settings via environment variables, so I had to modify the settings file to listen to them where that wasn't the case. I created an enviroment variable IS_HEROKU which allows to have a heroku-only configuration where needed.
The db service is provided by Amazon RDS, which requires a ssl connection. To enable ssl in the MySQLdb python client I had to modify Datasource (and bump up the version used).
The cache service is provided by the memcachier heroku addon. Heroku recommends to use pylibmc, so I set it up according to the docs here https://devcenter.heroku.com/articles/memcachier#python.
The amqp service is provided by the CloudAMQP addon.
I added a post_compile script that runs every time we deploy. We should run every build step we require in there, like static asset minification, collection, etc.
To share the oauth credentials among the various services I used an environment variable. I also added an option to export_project_credentials so that the credentials can be printed to stdout. This should come handy when we will need to update the environment-stored credentials with the ones in the db.
The Cython build process creates .c and .so files in the log_parser
directory, which only get re-created if the original source changes.
We recently updated the version of Cython, however this doesn't trigger
re-building them. To avoid surprises, we should purge them like we do
for the .pyc files.
In addition, there is no need to delete .pyc files from the UI source
directory, since that does not contain any Python files.
We're passing settings.UPDATE_REF, which is the default ref value, not
the actual value that was passed via the Chief web UI. As such, New
Relic was always displaying the revision deployed as "master", even if
it was a non-master branch deploy. Ideally we'd use the value of 'ref'
from pre_update(), but that's not available during the deploy() task
(the script is run in three independent stages & Chief unhelpfully only
passes 'ref' to the first stage), so we'll have to read it back from the
media/revision file - which will require more mangling in bug 1076886.
For now, seeing as the 'revision' property is optional, lets just omit
it, rather than setting it to a sometimes wrong value.
Prior to this change, on stage/production we didn't use virtualenvs
(unlike dev/the local Vagrant project) and instead pip installed
packages globally (when puppet ran periodically), using requirements
files that are not in the repo.
Now during deployment, a virtualenv is created and then populated using
peep (which uses hashes to verify the contents of packages before pip
installing them). The virtualenv is then made relocatable (as best as it
can, the feature isn't perfect), the lib64 symlinks are made relative,
and then the virtualenv is rsynced to all nodes, along with the source.
The one main remaining limitation of --relocatable is that the bash
activate script will not work on the other nodes - however the wrapper
scripts under treeherder-service/bin/ add venv/bin/ to PATH so using the
activate script is unnecessary for them. This just leaves running
manage.py commands locally on a node, for we can use:
|../venv/bin/python manage.py foo|, an alias or else we can fix up the
activate scripts in a follow-up bug.
The 'treeherder-service' repo has been renamed to 'treeherder', ready
for when the treeherder-ui repo is imported into it. This means the
Github URL, Travis URL and directory name when cloned changes. The Read
The Docs URL cannot be changed, so for now we will leave as-is, and in
the future (once service and UI docs combined) we will create a new
project on RTD with name "treeherder".
This updates doc links and puppet/Vagrant configs, but leaves the
stage/prod deploy script alone, since renaming the directories on our
infra is non-trivial. The dev instance will need some TLC since unlike
stage/prod, it does use the puppet scripts in the repo.
The current values were copied from another project's deploy script, but
they are not working - so let's use what the latest New Relic docs say
we should use.
Apparently the current directory is only sometimes on the path & I
mis-read or understood the docs, sigh. Let's just keep the sys.path
manipulation and use #noqa to keep flake8 quiet.
Anything in the deploy script decorated with @task or @hostgroups ends
up having it's progress displayed in IRC by the bot - eg:
[2015-02-23 20:09:40] Finished deploy_log (0.454s)
However, the "deploy_X" @hostgroup functions were actually only
rsyncing the code (which is fast compared to restarting the processes),
but the name used to imply the whole deploy for that node type was
complete.
The new structure not only reduces code duplication, but makes the IRC
output more accurate, to provide a better idea as to what stage the
deploy is at, and which are the problematic steps in the case of errors.
Previously the revision file was regenerated after it had already been
rsynced to the webheads, and as such the publicly accessible file
displayed the previous revision, not the new revision.
The current fetch/reset/checkout doesn't correctly update the working
directory when switching between branches. This new approach is
recommended by:
http://stackoverflow.com/a/9589927
The directory of the currently running script is already on the path:
https://docs.python.org/2/tutorial/modules.html#the-module-search-path
...so adding it is unnecessary.
In addition, I don't believe we need the treeherder-service directory on
the path, since we don't directly import anything from there.
The @hostgroups decorator was causing us to call restart-jobs multiple
times (once for each host within a host group), when we only want to
call it once per group of hosts.
Abstracting the restart-jobs call to its own method also means the IRC
bot will output "restart_jobs complete" after each restart-jobs call,
making the progress through the deployment task clearer.
Ed Morley
Ed Morley
Ed Morley
MikeLing
Mauro Doglio
Cameron Dawson
mdoglio