Bug 1197796 - Make WhiteNoise serve the static assets gzipped

On Heroku, there is no load balancer or Varnish-like cache in front of
gunicorn, so we must handle gzipping responses in the app.

In order for WhiteNoise to serve gzipped static content, assets must be
gzipped on disk in advance (doing so on-demand in Python would not be
as performant). WhiteNoise will then serve the `.gz` version of files in
preference to the original, if the client indicated it supported gzip.

For assets covered by Django's collectstatic, gzipping the assets only
requires using WhiteNoise's GzipManifestStaticFilesStorage backend,
which wraps Django's ManifestStaticFilesStorage to create hashed+gzipped
versions of static assets:
http://whitenoise.evans.io/en/latest/django.html#add-gzip-and-caching-support

The collectstatic generated files will then contain the file hash in
their filename, so WhiteNoise can also serve them with a large max-age
to avoid further requests if the file contents have not changed.

For the UI files under `dist/`, we cannot rely on the Django storage
backend, since the directory isn't covered by STATICFILES_DIRS (it is
instead made known to WhiteNoise via `WHITENOISE_ROOT`). As such, files
under `dist/` are gzipped via an additional step during deployment. See:
http://whitenoise.evans.io/en/latest/base.html#gzip-support

Files whose extension is on the blacklist, or that are not >5% smaller
when compressed, are skipped during compression.

.gitignore

@@ -42,6 +42,9 @@ htmlcov/
 # Deployed revision file.
 dist/revision.txt
 
+# Gzipped UI files created during deployment, for use with WhiteNoise.
+dist/**/*.gz
+
 #static files in deployment
 treeherder/static/
 

bin/post_compile

@@ -0,0 +1,8 @@
+#!/bin/bash -e
+# Tasks run by the Heroku Python buildpack after the compile step.
+
+# Generate gzipped versions of files that would benefit from compression, that
+# WhiteNoise can then serve in preference to the originals. This is required
+# since WhiteNoise's Django storage backend only gzips assets handled by
+# collectstatic, and so does not affect files in the `dist/` directory.
+python -m whitenoise.gzip dist

deployment/update/update.py

@@ -36,6 +36,8 @@ def pre_update(ctx, ref=settings.UPDATE_REF):
         ctx.local('git fetch --quiet origin %s' % ref)
         ctx.local('git reset --hard FETCH_HEAD')
         ctx.local('find . -type f -name "*.pyc" -delete')
+        # Remove gzipped UI assets in case the uncompressed original no longer exists.
+        ctx.local('find dist/ -type f -name "*.gz" -delete')
         ctx.local('git status -s')
         ctx.local('git rev-parse HEAD > dist/revision.txt')
 
@@ -65,6 +67,11 @@ def update(ctx):
         # collectstatic should be performed on one of the stage/prod specific
         # nodes at the end of the deploy, rather than on the admin node.
 
+        # Generate gzipped versions of files that would benefit from compression, that
+        # WhiteNoise can then serve in preference to the originals. This is required
+        # since WhiteNoise's Django storage backend only gzips assets handled by
+        # collectstatic, and so does not affect files in the `dist/` directory.
+        ctx.local("python2.7 -m whitenoise.gzip dist")
         # Collect the static files (eg for the Persona or Django admin UI)
         run_local_with_env(ctx, "python2.7 manage.py collectstatic --noinput")
         # Update the database schema, if necessary.

treeherder/settings/base.py

@@ -68,6 +68,10 @@ STATICFILES_FINDERS = [
     # "django.contrib.staticfiles.finders.DefaultStorageFinder",
 ]
 
+# Create hashed+gzipped versions of assets during collectstatic,
+# which will then be served by WhiteNoise with a suitable max-age.
+STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
+
 TEMPLATE_LOADERS = [
     "django.template.loaders.filesystem.Loader",
     "django.template.loaders.app_directories.Loader",