Since it's faster, deterministic and doesn't given obscure errors when
using `--no-bin-links` (which is required for both npm and yarn on
Windows hosts), and as such unblocks the work in bug 1343624.
Many of the commands are the same as with npm. See:
https://yarnpkg.com/en/docs/usage
Routing commands via npm/yarn is preferred, since it avoids
having to do global installs of grunt-cli, which simplifies contributor
setup, and means less effort when we switch to Yarn (since it requires
manual PATH setup for globally installed packages).
Since hopefully now we're no longer using datasource, the leaks should
have gone. The gunicorn processes will now only be restarted at the
daily Heroku dyno restart, rather than multiple times per minute,
improving performance.
Outputting to the console rather than a log file:
* is more user-friendly during development
* is more consistent with Heroku
* means the Vagrant-specific Django LOGGING config is now closer to the
one in settings.py, and so more easily combined with it
Both gunicorn and celery default to outputting to stdout/stderr, so the
`logfile` options can be omitted entirely.
The packages have been moved under the archives path, since there are
newer releases now available. The official download link has remained
unchanged, however it's a 302 redirect to a non-HTTPS version of these
links, so we intentionally deep-link, even if it means the URLs will
occasionally need updating.
Fixes the 404 whilst Vagrant provisioning, and failures that would occur
with a clean cache on Heroku/Travis.
This changes ingestion, the API endpoints, and the frontend to match
the new structure. For now we continue to store text_log_summary artifacts,
though they don't do anything anymore.
Previously `<site-root>/revision.txt` would 404 if `SERVE_MINIFIED_UI`
was unset on Heroku, which would then cause the next deploy to fail.
It's now made available in both the `ui/` and `dist/` directories, so
it can be found regardless of the value of `SERVE_MINIFIED_UI`.
* Bug 1264074 - Move to_timestamp function to a reusable location
* Bug 1264074 - Refactor JobConsumer to have a PulseConsumer super class
Much of what was in the JobConsumer is reusable by the upcoming
ResultsetConsumer. So refactor those parts out so that each specific
consumer can reuse code as much as possible.
* Bug 1264074 - Add ability to ingest Github Resultsets via Pulse
This introduces a ResultsetConsumer and a read_pulse_resultsets
management command to ingest resultsets from the TaskCluster
github exchanges.
When a supported Github repo has a Pull Request created or
updated, or a push is made to master, then it will kick off a
Pulse message. We will receive it and then fetch any additional
information we need from github's API and store the Resultset.
This follows a very similar pattern to the Job Pulse ingestion.
* Bug 1264074 - Old code/comments cleanup
* Bug 1264074 - Tests for the Github resultset pulse loader
We're using Bash errexit mode (`set -e`) so cannot directly use a
command that returns non-zero inside `[` style brackets (unless in a
`$(...)`), however the `(` form does work with errexit and is cleaner.
There's a strange bug when creating relative symlinks on top of an
already existing symlink, in that the target resolves to the wrong file.
As such, we now explicitly check for an existing symlink first, to avoid
overwriting.
The SCL3 system check disabling has also been tweaked since #1770, since
the `HOSTNAME` environment variable doesn't appear to be set during
deploy, even though it appears in `env` on the same machine.
Yey for deployment code that can only really be tested by deploying \o/
The Heroku pre_compile script is currently run prior to the cache being
restored (https://github.com/heroku/heroku-buildpack-python/pull/321),
which means we have to tweak PATH so vendor-libmysqlclient.sh can find
the binaries from the cache instead of the app directory.
However the workaround added in #1770 only added one of the two extra
required PATHs, this adds the other.
Prior to this the buildpack compile would output:
> ./bin/vendor-libmysqlclient.sh: line 65: pip: command not found
...and so wouldn't purge the old mysqlclient package, which is needed to
force recompilation against the newer libmysqlclient.
Once the PR against heroku-buildpack-python is merged, these workarounds
can be removed.
The latest versions of libmysqlclient 5.5/5.6 (used by mysqlclient) are
still vulnerable to TLS stripping, even after last year's backports of
5.7.x fixes:
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3152
- http://bugs.mysql.com/bug.php?id=82383
Ideally we'd just use the standalone Connector/C library instead of the
libmysqlclient packages, however the latest release is too old:
- http://bugs.mysql.com/bug.php?id=82448
Heroku's cedar-14 stack comes with libmysqlclient 5.5.x, so until it is
updated to 5.7.x (see https://github.com/heroku/stack-images/pull/38) we
must manually vendor 5.7.x ourselves, so that connections between the
Heroku dynos and our public RDS instances are secure. We can do this and
still remain on MySQL server 5.6, since newer client releases are
backwards compatible with older server versions.
Whilst the Vagrant/Travis MySQL instances don't use TLS (and so aren't
affected), we still want them to use libmysqlclient 5.7, to be
consistent with production.
Installing the newer libmysqlclient isn't sufficient on it's own. Any
packages compiled against the older version (in our case mysqlclient)
need to be recompiled. We ensure this happens by pip uninstalling the
existing package if it was already installed.
Rename ``ingest_from_pulse`` management command to ``read_pulse_jobs`` to
indicate that this step does not actually do any ingesting. It just populates
the celery queue ``store_pulse_jobs`` that DOES do the actual ingesting.
`manage.py check --deploy` is now run during Travis testing and as part
of stage/prod/Heroku deployment. It checks for a number of common
configuration mistakes & ensures security best practices are being
followed:
https://docs.djangoproject.com/en/1.8/ref/checks/
...by defining 'SKIP_PREDEPLOY' in the environment.
This is useful in cases where we want to avoid potentially destructive
actions (such as running migrations) from taking place, or know that the
migration is just going to time out unless run by hand.
Previously if the `bin/` directory scripts were run in the Vagrant
environment, the processes run would not have been started via the
New Relic wrapper command, since the New Relic licence key is not set.
For greater consistency between Vagrant and production, the wrapper
command is now always used.
This works since `NEW_RELIC_DEVELOPER_MODE` is defined in the Vagrant
environment (thanks to a previous commit in the same bug), which
prevents the agent from trying to submit real data:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode
There are some backwards incompatible changes:
http://whitenoise.evans.io/en/latest/changelog.htmlhttps://github.com/evansd/whitenoise/compare/v2.0.6...v3.0
Specifically:
* The CLI compression utility must now be called via
`python -m whitenoise.compress` rather than `python -m whitenoise.gzip`.
* The `whitenoise.django.GzipManifestStaticFilesStorage` storage backend
has moved to `whitenoise.storage.CompressedManifestStaticFilesStorage`.
* The internal `add_files()` method has been split into two and the part
which we need to subclass is now named `update_files_dictionary()`. See:
07f9c0bece
When first setting up a new app on Heroku, things like reporting the
deploy to New Relic will fail, since it requires that the app exist on
New Relic. However the app will only be created there once the Python
agent first reports app metadata, which won't happen until after the
deploy (there is no way to create the app via the web interface).
In addition, there may be cases in the future when stage/prod is broken,
and the pre-deploy tasks therefore fail, however we still want the
deploy to proceed.
To avoid needing to constantly edit this file, the environment variable
`IGNORE_PREDEPLOY_ERRORS` can now be set, in cases where the deploy
should continue even if there were errors. (Note this uses the bash 4.2+
`-v` option, see http://stackoverflow.com/a/18448624).
Requires that `NEW_RELIC_APP_NAME` and `NEW_RELIC_API_KEY` be set in the
environment. NB: `NEW_RELIC_API_KEY` is different from the existing
`NEW_RELIC_LICENSE_KEY`.
We're also making use of the runtime-dyno-metadata labs feature, which
sets the slug/release related environment variables used in this PR:
https://devcenter.heroku.com/articles/dyno-metadata
Since we'll soon be adding reporting deploys to New Relic, which will be
too verbose to include in the Procfile. Also adds additional log output
(which follows the buildpack compile log formatting convention) to make
it easier to find & follow the release tasks on Papertrail.
Uses the `set -euo pipefail` recommendation from:
http://redsymbol.net/articles/unofficial-bash-strict-mode/
The Python buildpack has now rewritten the automatic collectstatic
feature, such that it no longer does an unnecessary (and time-consuming)
dry-run every time. As such, we can switch back to the buildpack's
automatic collectstatic:
https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/collectstatic
Prior to this landing, I'll update us to the latest version of the
buildpack (using the `heroku buildpacks:set -i X ...` command) and
remove the `DISABLE_COLLECTSTATIC` environment variable.
[ci skip]
Since once the `grunt build` has run they are no longer required, and
only serve to bloat the slug size, increase attack surface & overwrite
the Python .profile.d script's environment variables.
Since we're not calculating ETAs (the UI does that once it knows the
start time and expected duraction), we're calculating recent average
durations instead.
This creates a 'runnable_job' table in the database, as well as an API
endpoint at /api/project/{branch}/runnable_jobs listing all existing
buildbot jobs and their symbols. A new daily task 'fetch_allthethings' is
added to update the this table.
Ed Morley
jgraham
William Lachance
Armen Zambrano G
William Lachance
Cameron Dawson
Alice Scarpa