WhiteNoise 3.0 now supports serving Brotli-compressed files to browsers
whose `Accept-Encoding` includes `br`. Note: Both Firefox and Chrome
only support Brotli over HTTPS.
To take advantage of this, the Brotli package just needs to be available
when the compression tool (`python -m whitenoise.compress`) is run. See:
http://whitenoise.evans.io/en/latest/changelog.html#brotli-compression-supporthttp://whitenoise.evans.io/en/latest/django.html#brotli-compression
The WhiteNoise docs say to use an unofficial PyPI package (brotlipy),
however this has a dependency on libffi (via cffi) and the official repo
now has it's own Python wrapper that does not. As such, this commit
instead uses the official Brotli package from GitHub, whilst we wait for
the official PyPI release (https://github.com/google/brotli/issues/72).
The Brotli install works fine on stage/prod/Heroku/Travis. The Vagrant
environment was missing g++, which is now installed during provision.
There are some backwards incompatible changes:
http://whitenoise.evans.io/en/latest/changelog.htmlhttps://github.com/evansd/whitenoise/compare/v2.0.6...v3.0
Specifically:
* The CLI compression utility must now be called via
`python -m whitenoise.compress` rather than `python -m whitenoise.gzip`.
* The `whitenoise.django.GzipManifestStaticFilesStorage` storage backend
has moved to `whitenoise.storage.CompressedManifestStaticFilesStorage`.
* The internal `add_files()` method has been split into two and the part
which we need to subclass is now named `update_files_dictionary()`. See:
07f9c0bece
As of pip 8, peep has now been integrated into pip.
Migrating from peep to this native feature has several advantages:
* It avoids the complexity/learning curve of using a wrapper around pip.
* It means we do not need to fork the official Heroku Python buildpack
(which handles pip installation of requirements files) in order to use
hash verification on Heroku. (Once the buildpack updates to pip 8.)
* Omitted sub-dependencies result in install-time errors rather than
the user discovering omissions at run-time.
* pip's native caching is used, and all packages are installed in one
pip invocation, so it's significantly faster.
* It has better handling of errors and corner cases.
Key facts about the native feature:
* hash-checking mode is enabled if at least one hash is found in the
requirements files passed to pip, or can be force enabled by passing
`--requires-hashes` when running `pip install`.
* Once enabled, hash-checking mode enforces that all packages:
- are pinned to a specific version
- have hashes listed
- have all sub-dependencies specified
* Older versions of pip will error out if either `--require-hashes` or
the requirements file `--hash` syntax is used, meaning it's not
possible to accidentally lose hash-checking protection if the pip used
is older than expected.
For more details, see:
https://pip.pypa.io/en/stable/user_guide/#hash-checking-modehttps://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
The pip version on Travis and in the Vagrant virtualenv has been updated
to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519.
The Heroku Python buildpack pip was updated in bug 1241909.
The requirements files hashes were ported using `peep port`, and then
comments/URLs re-added by hand.
We're already using pylibmc on Heroku, since we need its SASL
authentication support. However we were still using python-memcached on
Vagrant, Travis, stage & prod.
To reduce the number of simultaneous changes when we migrate to Heroku,
and to ensure at that point we're testing what we ship, this switches us
to pylibmc pre-emptively for all environments.
Django does have a native pylibmc backend [1], however it doesn't
support SASL authentication, so we have to use the custom django-pylibmc
backend instead [2], which we choose to use everywhere (even though only
Heroku's memcache instances require auth) for consistency.
Installing pylibmc requires libmemcached-dev, which is installed by
default on Travis, has just been installed on stage/prod (bug 1243767),
and as of this change, is now installed in the Vagrant environment too.
The comment mentioning that pylibmc must be present in the root
requirements.txt file no longer applies, since the Python buildpack now
uses pip-grep (rather than regex) to determine whether it is in the
requirements files [3], and so handles included requirements files too.
Example: https://emorley.pastebin.mozilla.org/8858007
I'll be checking for any changes in performance when this is deployed,
however if anything I expect it to be faster since it's not pure Python.
[1] https://github.com/django/django/blob/1.8.7/django/core/cache/backends/memcached.py#L171
[2] https://github.com/django-pylibmc/django-pylibmc/blob/master/django_pylibmc/memcached.py
[3] https://github.com/heroku/heroku-buildpack-python/blob/v75/bin/steps/pylibmc#L22
Since Heroku doesn't use nginx/Apache we must perform this via wsgi
middleware. We cannot use Django's HTTPS/HSTS features since they won't
help with requests that were served by WhiteNoise directly (eg the site
homepage).
Instead we use wsgi-sslify, as recommended by:
https://github.com/evansd/whitenoise/issues/53#issuecomment-166972824
We only enable it when IS_HEROKU is set, since stage/prod is handled by
Apache, and for local development we have to use HTTP.
We don't currently have code coverage enabled, and whilst we want to do
so, it will likely be in a form different from here (eg coveralls.io).
The packages removed by this commit are also older releases - and the
latest versions have changed their dependencies (for example cov-core is
no longer a separate package). By removing them it will reduce the
number of packages we have to install locally & on Travis, as well as
mean fewer warnings for out of date packages from requires.io.
pytest-django doesn't setup a test database for every single test, but
only for those tests that actually require a db. Tests that require a db
need to either be marked with `@pytest.mark.django_db` or use a fixture
that has a dependency on `db` or `transactional_db`.
Using a non transactional db would make tests execution much faster, but
unfortunately it doesn't play well with the treeherder datasource
creation so I used a transactional_db.
pytest-django also allows you to specify a settings file to use for
tests in a pytest.ini file, which is nicer than monkeypatch the original
settings file in the pytest session start function 😃.
These are the settings (which can be overridden by environment variables)
that will be used to specify which Pulse exchanges we ingest data from
This also introduces the ``django-environ`` package which will be used
elsewhere as we move away from ``local.py`` files on the stage/prod
servers.
The latest version of django-browserid removes a view that we used to fetch to retrieve
basig config params for the browserid client initialization. We now have loginUrl and logoutUrl
hardcoded in the client and we fetch the user login status from a dedicated endpoint.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
We had to do at least one deploy after the initial landing of
bug 1169944, before removing prod.txt, to avoid errors during update.py.
That has now occurred, so we can remove the file.
For bug 1124278, we're going to want to sprinkle new relic annotations
around the codebase, so by always installing it, we save having to stub
these out in development/on Travis. It also seems wise to have prod
running as close to the same packages as in development.
Since NEW_RELIC_LICENSE_KEY isn't set locally, plus
NEW_RELIC_DEVELOPER_MODE is set to true, the New Relic agent doesn't
submit anything. See:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode
William Lachance
Cameron Dawson
Ed Morley
Mauro Doglio