* Bug 1395254 - Consume Taskcluster Pulse messages from standard queue exchanges
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 4 terminal windows and follow these steps:
* On the first two windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the third window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* On the last window run `docker-compose up`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
Now that no submissions are using revision_hash, it can be removed.
This removes everything but the model field, which will be handled
later.
I've removed revision_hash from the Pulse jobs schema without bumping
the version, which wouldn't normally be ok, but no one is still using
it, and I'd rather have explicit failures later than if we left the
schema unchanged.
This sets the field lengths to what they will be in a later PR for
the job_details model. But these are still within the constraints
of the current field lengths for that table.
Several existing jobs are already out of compliance with these patterns
and there is no existing way to tell task definition developers how to
comply with our required patterns.
created bug 1283866 in Taskcluster for that tool/workflow
This removes the pattern requirements. If we ever decide that we DO
need these patterns, we can create that tool and then fix old task
definitions to comply.
Some repos are longer-lived and do not yet have the Task Cluster
code that allows them to submit tasks with a revision. They only
have the older code to submit revision_hash. This prevents the
jobs from being ingested via Pulse. This commit adds support
for revision_hash until a time when it's no longer needed.
This contains several tweaks and fixes that allow us to ingest data from
a real Task Cluster owned exchange.
One of the main fixes is the way I was binding to the exchange and
routing keys with the same Queue. Before, it was re-creating the Queue,
so would miss some of the bindings.
This will also prune the durable queue if the config has removed some
exchanges and/or routing keys.
These changes were discovered to be needed after direct testing
against Task Cluster Pulse exchanges.
-Made some JSON/YML schema changes to be more precise for several fields
-Modified to job_loader to be more resilient to optional data being
missing
We store both long and short, but only utilize the short (as before). We
need to populate all the short and long revision records before we can
start using them. So after this commit, we will begin backfilling the
old records that don't yet have those values populated. Once they all
are, we can move to using the long_revision primarily in Bug 1199364.
Cameron Dawson
Sarah Clements
Armen Zambrano G
Armen Zambrano
Ed Morley
Simon Sapin
Dustin J. Mitchell
Pete Moore
Dave Hunt