Templates directories inside Django apps are automatically detected, so
we shouldn't set the global directory inside a Django app. 404.html is
generic enough to be in the global directory, persona_login.html is only
used by the 'webapp' Django app, so is being left as-is.
Ideally we'd remove local.py entirely, in favour of environment
variables. However at least for now, it's still needed for setting the
logging preferences/enabling debugging apps for the Vagrant environment.
As such, it's being kept but disabled everywhere other than for local
development.
The stage/prod local.py files have been vetted, and (a) the contents
confirmed to match that in the environment, and (b) all affected
settings in base.py checked to ensure they read the correct environment
variable. Heroku doesn't have a local.py. As such, this should be a
no-op other than locally, where a Vagrant provision will be required.
On stage/prod the environment variable set is SITE_URL rather than
TREEHERDER_SITE_URL (this doesn't affect anything at the moment, since
local.py is being used, but we want to remove it). Instead of changing
the puppet configs its probably easier to just adjust what variable we
look at, particularly since the non-prefixed form is more concise and
matches what it actually controls.
I've added SITE_URL to Heroku in preparation.
So it can still be set on stage/prod, where we want to stop using
local.py. (It's already set in the environment there, we just need to
start using it.)
I added a create_credentials command to help setting up the initial
development environment. The puppet setup now creates a new user and set
it as the owner of the treeherder-etl credentials.
By storing the duration as an int rather than a timedelta, we can set it
via the environment, rather than having to rely on local.py.
The variable has also been renamed to ease the transition and also make
its purpose clearer. Prod/Heroku use the default duration, and stage has
already had DATA_CYCLE_DAYS set to 45 in its environment, for parity
with the old stage local.py DATA_CYCLE_INTERVAL setting.
So we can more easily track out of date dependencies.
Note that since we pin our nodejs packages, the Node badges will remain
green even if dependencies are out of date, due to:
https://github.com/alanshaw/david-www/issues/48
...however the badge is still useful since it's just one click to see
the table.
To get the requires.io badge working a GitHub hook has to be set up,
which I've already done.
Heroku now generates it on deploy, and for stage/prod we generate it
fresh on the stage/prod branch and force push each time. As such, we
have no need for the directory on master, and by removing it we avoid
confusion when new contributors grep the repo.
As an added bonus, the stage/prod deploy script should fail if the dist
directory is missing, so the grunt build cannot be forgotten prior to
deploying. (Currently if it's forgotten, we end up deploying the ancient
dist directory from master that was last updated prior to us switching
to the new deployment strategy.)
This documentation instructs a user on how to setup their local
machine to ingest data from existing exchanges as well as posting
to their own to test their jobs.
This ingests jobs over a pulse stream if the resultset for the job
already exists. There is a new management command called
``ingest_from_pulse`` which reads from a set of exchanges specified
in ``base.py`` or environment variables. These will be loaded in a
local celery queue for ingestion into the DB.
The user is expected to create their own Pulse user with Pulse Guardian
to be able to ingest jobs as well as post to their own queue.
A JSON Schema in the form of a YML file is included to validate the jobs
prior to ingestion. The resultset/push for a job must
already exist in Treeherder or the job will be skipped.
This commit relies on the nodejs buildpack being added to the list of
buildpacks for the app, and prior to the Python buildpack. See:
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
The nodejs buildpack will automatically install the packages listed in
`dependencies` in package.json, so that we have the requirements for
the grunt build. We don't actually need node or all of the files in
node_modules after we've run the grunt build, so in the future could try
and remove them to reduce the resultant slug size (though it only
increased from 55MB to 70MB, so it's not urgent).
The dist directory has been added to `.slugignore` to prevent the
in-repo directory from being uploaded, since we'll be generating a new
one as part of the deploy. Once `dist/` is deleted from master, that
entry can be removed.
Normally grunt-cli (a wrapper around grunt to allow it to be easily run
from the command line) would be installed globally, so that it is on the
PATH. However to save having to do this on prod/Heroku/Travis, we
install it locally - and it will then be available via:
`./node_modules/.bin/grunt`
...or else via just `grunt`, if referenced inside a package.json scripts
section.
Ed Morley
Mauro Doglio
Ed Morley
Cameron Dawson
William Lachance
Tapesh