The individual RABBITMQ_* variables are never used on their own, so it
makes more sense to switch to a URL variable that combines all of them.
Travis/Vagrant have also had BROKER URL explicitly set, since we're
generally moving away from having testing defaults set in settings.py.
Once stage/prod have BROKER_URL set, we can remove the fallback to the
old variable names, and also remove defaults entirely, making missing
settings fail fast.
Now that we're using django-environ's env.bool() in settings.py, the
value of TREEHERDER_DEBUG set in the Vagrant environment is correctly
picked up, so the local settings file DEBUG define is unnecessary.
This means rather than just returning this to the client:
'{"detail":"Hawk authentication failed"}'
We'll output more information to the stage/prod/Vagrant console and thus
logs. eg:
WARNING [hawkrest:81] access denied: MacMismatch: MACs do not match;
ours: 8HqvrjA7MO0CmJ3c6Ij2VfEuZxkchEL+pOL6Cw7At2c=; theirs:
CyDv9vnmqP6yGqT+LknMA+FOwJvSLATJwvVgG0OTl8A=
Thanks to the log.warn() here:
https://github.com/kumar303/hawkrest/blob/0.0.8/hawkrest/__init__.py#L80
I've removed the log level set on the console handler, since I think
it's clearer to set them on the loggers themselves, but it's not
entirely clear what Django logging best practices are, since the Django
docs are not very specific.
Since they're not specific to the Django app 'webapp'.
Whilst we're there, the local & example settings files have been
renamed. In the future I'd like to combine settings_local.example.py
with puppet/files/treeherder/local.vagrant.py, but I'll do that in
another bug.
Ideally we'd remove local.py entirely, in favour of environment
variables. However at least for now, it's still needed for setting the
logging preferences/enabling debugging apps for the Vagrant environment.
As such, it's being kept but disabled everywhere other than for local
development.
The stage/prod local.py files have been vetted, and (a) the contents
confirmed to match that in the environment, and (b) all affected
settings in base.py checked to ensure they read the correct environment
variable. Heroku doesn't have a local.py. As such, this should be a
no-op other than locally, where a Vagrant provision will be required.
I added a create_credentials command to help setting up the initial
development environment. The puppet setup now creates a new user and set
it as the owner of the treeherder-etl credentials.
Previously if TREEHERDER_DJANGO_SECRET_KEY was not set, we'd silently
fall back to a default value for SECRET_KEY, meaning we wouldn't realise
we were using an insecure key on a live deployment instance.
With this change, TREEHERDER_DJANGO_SECRET_KEY being missing from the
environment is fatal, resulting in:
"ImproperlyConfigured: The SECRET_KEY setting must not be empty."
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
For bug 1124278, we're going to want to sprinkle new relic annotations
around the codebase, so by always installing it, we save having to stub
these out in development/on Travis. It also seems wise to have prod
running as close to the same packages as in development.
Since NEW_RELIC_LICENSE_KEY isn't set locally, plus
NEW_RELIC_DEVELOPER_MODE is set to true, the New Relic agent doesn't
submit anything. See:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode
Removes the default value for DATABASE_URL and DATABASE_URL_RO, so that
if either are not set, there is a clearer upfront Django error about
Databases not being configured, rather than timeouts trying to connect
to localhost.
This makes it more obvious when setting up a new instance that the
variables are not present in the environment (and would have avoided
some confusion on Heroku today, when DATABASE_URL was set, but
DATABASE_URL_RO was not) - in exchange for now needing two extra lines
in the puppet config for local testing.
Since it only speeds up parsing by a few percent of total runtime, and
is therefore not worth the added complexity for deployment and local
hack-test-debug cycles when working on the log parser.
The .gitignore and update.py entries will be removed in a later commit,
once the stage/prod src directories have been cleaned up.
In order that we can serve the UI on Heroku, we wrap the Django wsgi app
with WhiteNoise, so both the UI and API requests are served by gunicorn.
In the Vagrant environment, Apache has been removed and Varnish instead
now proxies all requests to gunicorn/Django runserver directly, without
Apache as a go-between.
The UI on production will not be affected by this commit, since the
Apache config there will still intercept requests for the UI assets
rather than proxying them to gunicorn.
It's worth noting too, that we're not able to make use of WhiteNoise's
automatic Django GZip/caching support since that assumes we are using
Django templates and referring to resources using {% static "foo.css" %}
However, we can sub-class WhiteNoise (or more specifically the
DjangoWhiteNoise class) and override the is_immutable_file() method to
add caching support at a later date:
http://whitenoise.evans.io/en/latest/base.html#caching-headers
Documentation for WhiteNoise can be found at:
http://whitenoise.evans.io/
Since we should be doing this using Django's GZip middleware, not in
Varnish, which is only used in development. At least with this removed,
we won't check a local instance and believe we have everything set up
correctly if its not.
This removes the puppet workaround added by bug 1144805, since the
Ubuntu 14.04 vagrant image comes with puppet 3.4.3, and so includes the
fix for:
https://projects.puppetlabs.com/issues/10907
This uses the newer image from:
https://vagrantcloud.com/ubuntu/boxes/trusty32
This also updates us from apache 2.2 to apache 2.4, which means the
apache config must be changed to be compatible with apache 2.4:
http://httpd.apache.org/docs/2.4/upgrading.html
(The syntax for "Allow from all" has changed, plus the default config
now specifies a "Require all denied" which we have to override for the
UI directory)
A few puppet steps were either depending on the wrong package (ie
apache2-dev instead of apache2) or else did not have a 'requires'
property set at all. This was causing errors during provision due to
races - particularly after updating to Ubuntu 14.04.
Sets the default values (and now also those used by Vagrant) to the same
as those used by Travis, so we can avoid specifying different values all
over the place.
The only remaining difference between production.pp and vagrant.pp is
that the former uses a different local settings file
(local.production.py), which also outputs to the syslog. However we
don't need this for the local development environment, and nothing in
production is using the in-repo puppet, so it is no longer required.
One of the few differences between vagrant.pp and production.pp is from
which directory the UI is served. This change adds the ability to toggle
the directory from the Vagantfile directly, thereby bringing
production.pp (and the duplication it brings) closer to being obsolete.
As part of merging the UI repo into this one, the following directory
moves were performed:
webapp/app/ -> ui/
webapp/test/ -> tests/ui/
webapp/config/ -> tests/ui/config/
webapp/scripts/ -> tests/ui/scripts/
webapp/scripts/web-server.js -> web-server.js
All packages in the vendor directory are now installed in the virtualenv
site-packages, so there's no need to add the in-repo vendor directory to
the Python path.
This differentiation was only useful when explaining which packages
could be listed in which requirements file (since compiled packages
could not be added to checked-in.txt). Now that all packages are peep
installed, common.txt contains both pure and compiled packages.
Ed Morley
Cameron Dawson
Mauro Doglio
William Lachance
Jonathan French