Work to support running Python tests outside of the Docker container.
This also helps with integration with IDEs and is prework for using Git hooks for precommit/prepush features.
* Support filtering tasks by test path
For every push, it fetches the artifact `manifests-by-task.json` produced by the Gecko decision task. For every job it adds the `test_paths` property which allows the filtering.
Click on the "Filter by a job field" (the funnel icon), select "test path" from the
dropdown and you can insert a path like `devtools/client/inspector/changes/test/browser.ini` (You can use substrings).
* Use Django's json()
* Skip test that only times out on Travis
* replace absolute positioned tooltip with custom Victory flyout tooltip
* remove use of x, y and pushId in query param
* move createGraphData into helpers and set up tooltip test
* update selected param in TH details panel
change css to allow scrollTo so header or row doesn't hide behind navbar, prevent updates with hash location prop change to address slowness onPermaLinkClick, and extract table row into CompareTableRow to utilize refs
Without this change we will get invalid schema changes in Papertrail without knowing
what log has this problem. This change creates a warning line that will lead us to
the log that has missing performance data.
* Bug 1574651 - remove unused JobConsumer and related code
The `update_pulse_test_fixtures` management command listens for job
messages, so is of no use anymore.
* Bug 1574651 - refactor pulse listening to support multiple AMQP servers
This looks forward to supporting ingesting jobs and tasks from multiple
Taskcluster depoyments, each of which is on its own AMQP server (or, at
least, a vhost).
* Bug 1574651 - pass rootUrl from pulse to celery, verify against repository
When jobs and pushes are loaded, the repo's root URL is known. This
just serves to ensure that the rootUrl for the repo and the rootUrl for
the event match up.
* Bug 1574651 - use root_url from message to make in-job URLs
* Bug 1574651 - update ingest-and-push-tasks to take --root-url
* Bug 1574651 - add tc_root_url to each repository
* Fix issues introduced since initial creation of this feature
* Fix unit tests
* Fix retriggers in Compare Chooser of Perfherder
Switch Perfherder to react-router
Use top-level of app as a cache for projects, frameworks, alerts data and compare data
Cleanup files and move constants to dedicated perfherder file
Remove angular-related libraries and bump up the neutrino entry and asset limits
This fixes the issues with retriggering and add-new-jobs that the
original PR introduced. It also adds a few unit tests to catch
this potential mistake in the future.
This reverts commit ffd871ae34.
`jest-dom` has moved to `@testing-library/jest-dom`. Please uninstall jest-dom and install `@testing-library/jest-dom` instead,
or use an older version of `jest-dom`. If you do upgrade to `@testing-library/jest-dom`, make sure to update your usage of `jest-dom`
to use `@testing-library/jest-dom/extend-expect` rather than simply `jest-dom/extend-expect`. Learn more about this change here:
https://github.com/testing-library/dom-testing-library/issues/260 Thanks!
* Rename details panel ``selectedJob`` to ``selectedJobFull``
The job that's passed in the DetailsPanel has a bunch of extra fields
that are not in the normal downloaded list of jobs. So I wanted to
depict that ``selectedJob`` is not the same thing as what you see
in the DetailsPanel.
* Stop using Redux where not necessary
I was using Redux to assign the selectedJob in a few details
classes when I should have just passed it where it was needed.
* New addAggregateFields function
Instead of using a more heavy weight JobModel for each job,
we just persist some fields that were getting constantly calculated
over and over. This was especially true during filtering and re-rendering.
* Remove some cruft leftover from Buildbot.
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 5 terminal windows and follow these steps:
* On the first window run `docker-compose up`
* On the next three windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_tasks`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the last window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Extra script =
Script that permits comparing pushes from two different Treeherder instances.
```
usage: Compare a push from a Treeherder instance to the production instance.
[-h] [--host HOST] --revision REVISION [--project PROJECT]
optional arguments:
-h, --help show this help message and exit
--host HOST Host to compare. It defaults to localhost
--revision REVISION Revision to compare
--project PROJECT Project to compare. It defaults to mozilla-central
```
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
* Bug 1395254 - Consume Taskcluster Pulse messages from standard queue exchanges
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 4 terminal windows and follow these steps:
* On the first two windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the third window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* On the last window run `docker-compose up`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
Modify performance/summary endpoint to accommodate perfherder graphing needs:
* modify logic so signature query param does not filter on parent_signature__isnull
and add all_data param to return performance data as a list of objects with additional
data like PerformanceDatum
* add condition to all_data and return revision, repo name and repository_id
* accept multiple signatures
* make startday and endday optional if interval is provided
Bugfiler used an undocumented bugzilla api to search for
product/component pairs, and bugzilla recently moved the api endpoint,
breaking the bug filer. This patch points to the correct endpoint.
This updates the bug filer to use markdown in the bugzilla bug
description. It adds bolded labels to the `Filed by` entry and any URLs
specified. Addtionally the comment is surrounded in a code fence so
that long lines aren't wrapped and lines that also happen to have
markdown formatting in them are ignored.
Example output:
**Filed by:** foo [@] bar.com
**Parsed log:** http://.../parsed.log.html
**Full log:** http://.../full.log.html
---
```
[task 2019-03-06T03:54:26.459Z] 03:54:26 INFO - TEST-FAIL...
[task 2019-03-06T03:54:26.460Z] 03:54:26 INFO - INFO | LeakSanitize
...
```
Kyle Lahnakoski
octavian-negru
Mellina Yonashiro
ionutgoldan
Armen Zambrano
Cameron Dawson
Alexandru Irimovici
Sarah Clements
renovate[bot]
Shavin Fonseka
Octavian Negru
Shoaib Shaikh
Armen Zambrano G
Musa Talluzi
Sebastian Hengst
Dustin J. Mitchell
Kohei Yoshino
Sarah Clements
KWierso
Eric Rahm