Bug 1151803 - Serve the UI using gunicorn/WhiteNoise instead of Apache

In order that we can serve the UI on Heroku, we wrap the Django wsgi app
with WhiteNoise, so both the UI and API requests are served by gunicorn.

In the Vagrant environment, Apache has been removed and Varnish instead
now proxies all requests to gunicorn/Django runserver directly, without
Apache as a go-between.

The UI on production will not be affected by this commit, since the
Apache config there will still intercept requests for the UI assets
rather than proxying them to gunicorn.

It's worth noting too, that we're not able to make use of WhiteNoise's
automatic Django GZip/caching support since that assumes we are using
Django templates and referring to resources using {% static "foo.css" %}

However, we can sub-class WhiteNoise (or more specifically the
DjangoWhiteNoise class) and override the is_immutable_file() method to
add caching support at a later date:
http://whitenoise.evans.io/en/latest/base.html#caching-headers

Documentation for WhiteNoise can be found at:
http://whitenoise.evans.io/

Vagrantfile

@@ -28,13 +28,6 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
     puppet.manifests_path = "puppet/manifests"
     puppet.manifest_file = "vagrant.pp"
 
-    # The UI can either be served as is, or else in minified/processed form,
-    # from the dist/ directory (generated by grunt build). To serve the
-    # minified version, toggle serve_minified_ui to true.
-    puppet.facter = {
-      "serve_minified_ui" => "false"
-    }
-
     # enable this to see verbose and debug puppet output
     #puppet.options = "--verbose --debug"
   end

docs/deployment.rst

@@ -7,16 +7,8 @@ Serving the UI build from the dist directory
 During local development the UI is served in its original, unprocessed form. In
 production, a minified/built version of the UI (generated using grunt) is used instead.
 
-To serve the built version of the UI locally, in ``Vagrantfile`` change
-``serve_minified_ui`` to true:
-
-.. code-block:: ruby
-
-  puppet.facter = {
-    "serve_minified_ui" => "true"
-  }
-
-You will need to run ``vagrant provision`` to pick up those changes, if the Vagrant environment was already created.
+To serve the built version of the UI locally, set ``SERVE_MINIFIED_UI`` to True in
+the environment before starting gunicorn/runserver.
 
 
 Updating the UI build prior to deployment

docs/installation.rst

@@ -73,7 +73,7 @@ Setting up a local Treeherder instance
 Viewing the local server
 ------------------------
 
-* Start a gunicorn instance, to serve API requests:
+* Start a gunicorn instance, to serve the static UI and API requests:
 
   .. code-block:: bash
 

docs/list_of_services.rst

@@ -12,8 +12,8 @@ Follows a description of those services.
 Gunicorn
 --------
 
-A wsgi server in charge of serving the restful api and the django admin.
-All the requests to this server are proxied through varnish and apache.
+A wsgi server in charge of serving the restful api and the static UI assets.
+All the requests to this server are proxied through Varnish.
 
 Celery task worker
 ------------------

puppet/files/apache/treeherder.conf

@@ -1,80 +0,0 @@
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, you can obtain one at http://mozilla.org/MPL/2.0/.
-
-<VirtualHost *:8080>
-  ServerName <%= @APP_URL %>
-
-  <Directory <%= @APP_UI_DIR %>>
-    Require all granted
-  </Directory>
-
-  <Directory <%= @APP_STATIC_DIR %>>
-    Require all granted
-  </Directory>
-
-  ProxyRequests Off
-  <Proxy *>
-    Require all granted
-  </Proxy>
-
-  ###########
-  # Shared locations between production and dev environment
-  ###########
-  Alias /help.html <%= @APP_UI_DIR %>/help.html
-  ProxyPass        /help.html !
-
-  Alias /logviewer.html <%= @APP_UI_DIR %>/logviewer.html
-  ProxyPass        /logviewer.html !
-
-  Alias /perf.html <%= @APP_UI_DIR %>/perf.html
-  ProxyPass        /perf.html !
-
-  Alias /js <%= @APP_UI_DIR %>/js
-  ProxyPass        /js !
-
-  Alias /css <%= @APP_UI_DIR %>/css
-  ProxyPass        /css !
-
-  Alias /img <%= @APP_UI_DIR %>/img
-  ProxyPass        /img !
-
-  Alias /fonts <%= @APP_UI_DIR %>/fonts
-  ProxyPass        /fonts !
-
-  ###########
-  # These locations are to support loading files directly from the non-dist
-  # directories to support local development
-  ###########
-  Alias /vendor <%= @APP_UI_DIR %>/vendor
-  ProxyPass        /vendor !
-
-  Alias /plugins <%= @APP_UI_DIR %>/plugins
-  ProxyPass        /plugins !
-
-  Alias /partials <%= @APP_UI_DIR %>/partials
-  ProxyPass        /partials !
-
-  Alias /icons <%= @APP_UI_DIR %>/icons
-  ProxyPass        /icons !
-
-  ###############
-  # Serve static and media files
-  ###############
-  Alias /static <%= @PROJ_DIR %>/treeherder/webapp/static
-  ProxyPass        /static !
-  Alias /media <%= @PROJ_DIR %>/treeherder/webapp/media
-  ProxyPass        /media !
-
-  Alias / <%= @APP_UI_DIR %>/index.html
-  ProxyPassMatch ^/$ !
-
-  ProxyPass        / http://localhost:8000/ retry=0
-  ProxyPassReverse / http://localhost:8000/
-  ProxyPreserveHost On
-
-  ErrorLog /var/log/<%= @apache_service %>/treeherder_err.log
-  LogLevel warn
-  CustomLog /var/log/<%= @apache_service %>/treeherder_access.log combined
-
-</VirtualHost>

puppet/files/varnish/default.vcl

@@ -2,12 +2,15 @@
  * License, v. 2.0. If a copy of the MPL was not distributed with this
  * file, you can obtain one at http://mozilla.org/MPL/2.0/. */
 
-backend apache {
+# We use Varnish even for development, since gunicorn/runserver default to 127.0.0.1:8000,
+# and to be accessible from outside the VM we'd have to use 0.0.0.0. In addition, to serve
+# on port 80 we'd have to run gunicorn/runserver with sudo or use authbind.
+backend gunicorn {
     .host = "127.0.0.1";
-    .port = "8080";
+    .port = "8000";
 }
 
 sub vcl_recv {
-    set req.backend = apache;
+    set req.backend = gunicorn;
     return (pass);
 }

puppet/manifests/classes/apache.pp

@@ -1,73 +0,0 @@
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, you can obtain one at http://mozilla.org/MPL/2.0/.
-
-$apache_devel = $operatingsystem ? {
-  ubuntu => "apache2-dev",
-  default => "httpd-devel",
-}
-
-$apache_vhost_path = $operatingsystem ? {
-  ubuntu => "/etc/apache2/sites-enabled",
-  default => "/etc/httpd/conf.d",
-}
-
-$apache_service = $operatingsystem ? {
-  ubuntu => "apache2",
-  default => "httpd",
-}
-
-$apache_port_definition_file = $operatingsystem ? {
-  ubuntu => "/etc/apache2/ports.conf",
-  default => "/etc/httpd/conf/httpd.conf",
-}
-
-class apache {
-  package { $apache_devel:
-    ensure => present
-  }
-
-  package { $apache_service:
-    ensure => present
-  }
-
-  file { "${apache_vhost_path}/treeherder.conf":
-    content => template("${PROJ_DIR}/puppet/files/apache/treeherder.conf"),
-    owner => "root", group => "root", mode => 0644,
-    require => Package[$apache_service],
-    notify => Service[$apache_service],
-  }
-
-  exec { "sed -i '/[: ]80$/ s/80/8080/' ${apache_port_definition_file}":
-    require => Package[$apache_service],
-    before => [
-      Service[$apache_service]
-    ]
-  }
-
-  service { $apache_service:
-    ensure => running,
-    enable => true,
-    require => [
-      File["${apache_vhost_path}/treeherder.conf"]
-    ],
-  }
-
-  if $operatingsystem == 'ubuntu'{
-    /*by default ubuntu doesn't have these modules enabled*/
-    exec {
-      'a2enmod rewrite':
-        onlyif => 'test ! -e /etc/apache2/mods-enabled/rewrite.load',
-        require => Package[$apache_service],
-        before => Service[$apache_service];
-      'a2enmod proxy':
-        onlyif => 'test ! -e /etc/apache2/mods-enabled/proxy.load',
-        require => Package[$apache_service],
-        before => Service[$apache_service];
-      'a2enmod proxy_http':
-        onlyif => 'test ! -e /etc/apache2/mods-enabled/proxy_http.load',
-        require => Package[$apache_service],
-        before => Service[$apache_service];
-    }
-  }
-}

puppet/manifests/classes/varnish.pp

@@ -12,7 +12,6 @@ $varnish_port_change = $operatingsystem ? {
   default => "sed -i '/^VARNISH_LISTEN_PORT=6081$/ s/6081/80/' ${varnish_port_file}",
 }
 
-
 class varnish {
   package { "varnish":
     ensure => installed;
@@ -27,20 +26,14 @@ class varnish {
   file {"/etc/varnish/default.vcl":
     content => template("${PROJ_DIR}/puppet/files/varnish/default.vcl"),
     owner => "root", group => "root", mode => 0644,
-    require => [Package["varnish"]],
+    require => Package["varnish"],
     before => Service["varnish"],
     notify => Service["varnish"],
   }
 
   exec { $varnish_port_change:
-    require => [
-      Package[$apache_devel],
-      Package["varnish"],
-    ],
-    before => [
-      Service["varnish"]
-    ],
+    require => Package["varnish"],
+    before => Service["varnish"],
     notify => Service["varnish"],
   }
-
 }

puppet/manifests/vagrant.pp

@@ -5,7 +5,6 @@
 # Playdoh puppet magic for dev boxes
 import "classes/*.pp"
 
-$APP_URL="local.treeherder.mozilla.org"
 $APP_USER="vagrant"
 $APP_GROUP="vagrant"
 $HOME_DIR = "/home/${APP_USER}"
@@ -14,16 +13,6 @@ $VENV_DIR = "${HOME_DIR}/venv"
 $PS1 = '\[\e[0;31m\]\u\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \$ '
 $THELP_TEXT = 'Type \\"thelp\\" to see a list of Treeherder-specific helper aliases'
 
-# The UI can either be served as is, or else in minified/processed form,
-# from the dist/ directory (generated by grunt build). To serve the
-# minified version, toggle serve_minified_ui to true in Vagrantfile.
-$APP_UI_DIR = $serve_minified_ui ? {
-    "true" => "${PROJ_DIR}/dist",
-    default => "${PROJ_DIR}/ui",
-}
-
-$APP_STATIC_DIR = "${PROJ_DIR}/treeherder/webapp/static"
-
 # You can make these less generic if you like, but these are box-specific
 # so it's not required.
 $DB_USER = "treeherder_user"
@@ -65,8 +54,7 @@ class vagrant {
     class {
         init: before => Class["mysql"];
         mysql: before  => Class["python"];
-        python: before => Class["apache"];
-        apache: before => Class["varnish"];
+        python: before => Class["varnish"];
         varnish: before => Class["treeherder"];
         treeherder: before => Class["rabbitmq"];
         rabbitmq: before => Class["dev"];

requirements/common.txt

@@ -7,6 +7,9 @@
 # sha256: pRec3pItK04EXuWxHocyPud9NjrkApT8glLyXWoOrwY
 gunicorn==19.3.0
 
+# sha256: Pk2AGZlglZuCiVGqJMv6o2zr7RSdzXKMEehVeYsV-HA
+whitenoise==1.0.6
+
 # sha256: 7sV9MhlQHsbmhWRoJvLrjndoepP-N0p-aZRJBSDbCT4
 Django==1.7.7
 

treeherder/settings/base.py

@@ -62,6 +62,8 @@ USE_I18N = False
 USE_L10N = True
 USE_TZ = False
 
+SERVE_MINIFIED_UI = os.environ.get("SERVE_MINIFIED_UI") == "True"
+UI_ROOT = path("..", "dist" if SERVE_MINIFIED_UI else "ui")
 
 STATIC_ROOT = path("webapp", "static")
 STATIC_URL = "/static/"

treeherder/webapp/urls.py

@@ -2,9 +2,11 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, you can obtain one at http://mozilla.org/MPL/2.0/.
 
+from django.conf import settings
 from django.conf.urls import patterns, include, url
 from django.views.generic import RedirectView
 from django.contrib import admin
+from django.contrib.staticfiles.urls import staticfiles_urlpatterns
 
 from .api import urls as api_urls
 from treeherder.embed import urls as embed_urls
@@ -21,6 +23,12 @@ urlpatterns += patterns('',
                         url(r'^admin/', include(browserid_admin.urls)),
                         url(r'^docs/', include('rest_framework_swagger.urls')),
                         url(r'', include('django_browserid.urls')),
-                        # by default redirect all request on / to /ui/
-                        url(r'^$', RedirectView.as_view(url='/ui/'))
+                        # Redirect all requests on / to /index.html, where they
+                        # will be served by WhiteNoise.
+                        url(r'^$', RedirectView.as_view(url='index.html'))
                         )
+
+if settings.DEBUG:
+    # Add the patterns needed so static files can be viewed without running
+    # collectstatic, even when using gunicorn instead of runserver.
+    urlpatterns += staticfiles_urlpatterns()

treeherder/webapp/wsgi.py

@@ -18,7 +18,16 @@ framework.
 
 """
 import os
+
+# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
+# if running multiple sites in the same mod_wsgi process. To fix this, use
+# mod_wsgi daemon mode with each site in its own daemon process, or use
+# os.environ["DJANGO_SETTINGS_MODULE"] = "webapp.settings"
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "treeherder.settings")
+
+from django.conf import settings
 from django.core.wsgi import get_wsgi_application
+from whitenoise.django import DjangoWhiteNoise
 
 try:
     import newrelic.agent
@@ -28,17 +37,16 @@ except ImportError:
 if newrelic:
     newrelic.agent.initialize()
 
-# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
-# if running multiple sites in the same mod_wsgi process. To fix this, use
-# mod_wsgi daemon mode with each site in its own daemon process, or use
-# os.environ["DJANGO_SETTINGS_MODULE"] = "webapp.settings"
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "treeherder.settings")
-
 # This application object is used by any WSGI server configured to use this
 # file. This includes Django's development server, if the WSGI_APPLICATION
 # setting points here.
 application = get_wsgi_application()
 
+# Wrap the Django WSGI app with WhiteNoise so the UI can be served by gunicorn
+# in production, avoiding the need for Apache/nginx on Heroku.
+application = DjangoWhiteNoise(application)
+application.add_files(settings.UI_ROOT)
+
 if newrelic:
     application = newrelic.agent.wsgi_application()(application)