- Adds a new "infrastructure" section to the docs, which describes
architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
docs, to make the documentation easier to navigate.
Since it clobbers the settings we need to use on Heroku for making
memcached connections via stunnel. In addition, it made the cache
configuration very opaque, and doesn't use the latest best practices
suggested on:
https://www.memcachier.com/documentation#django
As of pip 8, peep has now been integrated into pip.
Migrating from peep to this native feature has several advantages:
* It avoids the complexity/learning curve of using a wrapper around pip.
* It means we do not need to fork the official Heroku Python buildpack
(which handles pip installation of requirements files) in order to use
hash verification on Heroku. (Once the buildpack updates to pip 8.)
* Omitted sub-dependencies result in install-time errors rather than
the user discovering omissions at run-time.
* pip's native caching is used, and all packages are installed in one
pip invocation, so it's significantly faster.
* It has better handling of errors and corner cases.
Key facts about the native feature:
* hash-checking mode is enabled if at least one hash is found in the
requirements files passed to pip, or can be force enabled by passing
`--requires-hashes` when running `pip install`.
* Once enabled, hash-checking mode enforces that all packages:
- are pinned to a specific version
- have hashes listed
- have all sub-dependencies specified
* Older versions of pip will error out if either `--require-hashes` or
the requirements file `--hash` syntax is used, meaning it's not
possible to accidentally lose hash-checking protection if the pip used
is older than expected.
For more details, see:
https://pip.pypa.io/en/stable/user_guide/#hash-checking-modehttps://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
The pip version on Travis and in the Vagrant virtualenv has been updated
to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519.
The Heroku Python buildpack pip was updated in bug 1241909.
The requirements files hashes were ported using `peep port`, and then
comments/URLs re-added by hand.
We're already using pylibmc on Heroku, since we need its SASL
authentication support. However we were still using python-memcached on
Vagrant, Travis, stage & prod.
To reduce the number of simultaneous changes when we migrate to Heroku,
and to ensure at that point we're testing what we ship, this switches us
to pylibmc pre-emptively for all environments.
Django does have a native pylibmc backend [1], however it doesn't
support SASL authentication, so we have to use the custom django-pylibmc
backend instead [2], which we choose to use everywhere (even though only
Heroku's memcache instances require auth) for consistency.
Installing pylibmc requires libmemcached-dev, which is installed by
default on Travis, has just been installed on stage/prod (bug 1243767),
and as of this change, is now installed in the Vagrant environment too.
The comment mentioning that pylibmc must be present in the root
requirements.txt file no longer applies, since the Python buildpack now
uses pip-grep (rather than regex) to determine whether it is in the
requirements files [3], and so handles included requirements files too.
Example: https://emorley.pastebin.mozilla.org/8858007
I'll be checking for any changes in performance when this is deployed,
however if anything I expect it to be faster since it's not pure Python.
[1] https://github.com/django/django/blob/1.8.7/django/core/cache/backends/memcached.py#L171
[2] https://github.com/django-pylibmc/django-pylibmc/blob/master/django_pylibmc/memcached.py
[3] https://github.com/heroku/heroku-buildpack-python/blob/v75/bin/steps/pylibmc#L22
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
For bug 1124278, we're going to want to sprinkle new relic annotations
around the codebase, so by always installing it, we save having to stub
these out in development/on Travis. It also seems wise to have prod
running as close to the same packages as in development.
Since NEW_RELIC_LICENSE_KEY isn't set locally, plus
NEW_RELIC_DEVELOPER_MODE is set to true, the New Relic agent doesn't
submit anything. See:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode
I added a Procfile listing all the different python services treeherder needs.
Heroku provides deployment-specific settings via environment variables, so I had to modify the settings file to listen to them where that wasn't the case. I created an enviroment variable IS_HEROKU which allows to have a heroku-only configuration where needed.
The db service is provided by Amazon RDS, which requires a ssl connection. To enable ssl in the MySQLdb python client I had to modify Datasource (and bump up the version used).
The cache service is provided by the memcachier heroku addon. Heroku recommends to use pylibmc, so I set it up according to the docs here https://devcenter.heroku.com/articles/memcachier#python.
The amqp service is provided by the CloudAMQP addon.
I added a post_compile script that runs every time we deploy. We should run every build step we require in there, like static asset minification, collection, etc.
To share the oauth credentials among the various services I used an environment variable. I also added an option to export_project_credentials so that the credentials can be printed to stdout. This should come handy when we will need to update the environment-stored credentials with the ones in the db.
Ed Morley
Ed Morley
Jonathan French
Mauro Doglio