This makes sure that serving the documentation via poetry does not regress
and we have a single way of generating them.
Setting the build backend in `pyproject.toml` makes `pip` to use `poetry` to install the dependencies. By doing so, readthedocs will be able to install the required dependencies to generate the docs (since they don't support poetry directly).
* Configuration for black
* changes congiguration
* change pyproject's directory
* add files to be excluded and skip string normalization
* removed isort from pre-commit
* remove version locks for black
* fix
* remove all isort
* update requirements
Co-authored-by: SuyashSalampuria <suyash546@gmail.com>
Co-authored-by: Kyle Lahnakoski <kyle@lahnakoski.com>
* Add Travis job to run Python tests outside of Docker
* `runtests.sh` is renamed to `runchecks.sh` and it does not run Python tests
* `manage.py check --deploy` was duplicated in Travis
* Update testing documentation
* Remove `-bb` since it is not needed since Python 3.5
Git based projects can list pushes in the UI incorrectly. This can be caused by commits having been ammended on a PR or merges of old commits.
Using the committer's date (the date when the PR gets merged) instead of the author's date to determine push time fixes the sorting problem.
This change also includes:
* Support for manual ingestion of Git pushes
* Support for ingesting the latest commits for a repo
* Script to compare pushes between Treeherder instances
## Script to compare pushes between Treeherder instances
`compare_pushes.py` compares the last 50 pushes of various projects for different Treeherder instances. The output generates links to each instance and revision to visually compare.
```console
% ./misc/compare_pushes.py --projects android-components,fenix,reference-browser,servo-master,servo-auto,servo-try
Comparing android-components against production.
Comparing fenix against production.
Comparing reference-browser against production.
{"values_changed": {"root['push_timestamp']": {"new_value": 1582580346, "old_value": 1582581477}}}
https://treeherder.allizom.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4ehttps://treeherder.mozilla.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4e
Comparing servo-master against production.
Comparing servo-auto against production.
Comparing servo-try against production.
```
Github based projects can list pushes in the UI incorrectly. This is caused by commits that have been ammended.
This code change switches to grab the push time from the `timestamp` field of the head commit from the Pulse event rather that Github's APIs. This fixes the problem of push sorting.
Note, that the field `timestamp` only exists in the Push event that the Pulse message contains. This field can be seen in the `events` API, however, that API contains all sorts of events and holds a maximum of 300 events.
This change also includes partial support for manual ingestion of Git pushes.
* CSS Cleanup
* Use darker-info and darker-secondary for a11y
* Put Parent Push metric at end of list and clean it up
* Add a `scrollToLine` when clicking/expanding metric names
Git based projects can list pushes in the UI incorrectly. This is caused by commits that have been ammended.
This code change switches to use the `timestamp` field of head commit to determine push time rather a commit's authorship date. This fixes the problem of push sorting.
This change also includes support for manual ingestion of Git pushes.
Add style specs in case darker-secondary and darker-info color props are used in other reactstrap components in the future. Update docs and reorganize these classes in the stylesheet.
* Properly configure LOGGING_LEVEL
`LOGGING_LEVEL` was being cast down to False rather than "INFO" or "DEBUG".
There were no fallouts from it but it was preventing setting the right logging
level for the Docker container or the Heroku Review Apps.
* Validate preseed entries and fail if issues are found
Without validation some invalid data job priorities would be inserted into the table. No issues
would be noticed for SETA since the web API sanitizes the data before presenting it.
This section was confusing for a contributor since it does not actually work (The MySql container is not ready) and the Django migrations are now automatically applied when running `docker-compose up`.
* Development Docker container to initialize data
`docker-compose up` will now initialize the data for the local database.
This will also include initializing SETA's data.
* Change SETA's logging level and preseed information
The preseed information was all for Buildbot
The logging level changes is in preparation for when production switching to INFO messages instead of WARNING
* Docker container to log INFO messages instead of DEBUG messages
`LOGGING_LEVEL` is now set for the Docker container and it sets the logging
to `INFO` instead of `DEBUG`. The Django app will still be initialize in
debug mode but the default logging will be informative messages.
You can still log debugging messages by setting `LOGGING_LEVEL=DEBUG` before
initializing the Docker container.
Defining PULSE_TASK_SOURCES and PULSE_PUSH_SOURCES is a bit of a nightmare
because `app.json` is a Json file and using double and single quotes for a
value in the file is impossible.
- update repositories fixture with new tc_root_urls for firefox-ci and community-ci
- update getRunnableJobsUrl and RunnableJobModel to access currentRepo and tc_root_url;
refactor getInspectTaskUrl and JobInfo component to have currentRepo prop
- add new rooturls to connect-src, update rooturl for seta, update docs and
remove header in make_request,
* Bug 1574651 - remove unused JobConsumer and related code
The `update_pulse_test_fixtures` management command listens for job
messages, so is of no use anymore.
* Bug 1574651 - refactor pulse listening to support multiple AMQP servers
This looks forward to supporting ingesting jobs and tasks from multiple
Taskcluster depoyments, each of which is on its own AMQP server (or, at
least, a vhost).
* Bug 1574651 - pass rootUrl from pulse to celery, verify against repository
When jobs and pushes are loaded, the repo's root URL is known. This
just serves to ensure that the rootUrl for the repo and the rootUrl for
the event match up.
* Bug 1574651 - use root_url from message to make in-job URLs
* Bug 1574651 - update ingest-and-push-tasks to take --root-url
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 5 terminal windows and follow these steps:
* On the first window run `docker-compose up`
* On the next three windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_tasks`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the last window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Extra script =
Script that permits comparing pushes from two different Treeherder instances.
```
usage: Compare a push from a Treeherder instance to the production instance.
[-h] [--host HOST] --revision REVISION [--project PROJECT]
optional arguments:
-h, --help show this help message and exit
--host HOST Host to compare. It defaults to localhost
--revision REVISION Revision to compare
--project PROJECT Project to compare. It defaults to mozilla-central
```
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
* Bug 1395254 - Consume Taskcluster Pulse messages from standard queue exchanges
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 4 terminal windows and follow these steps:
* On the first two windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the third window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* On the last window run `docker-compose up`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
Armen Zambrano
Kimberly Sereduck
Shubhank Saxena
Cameron Dawson
Sarah Clements
Kyle Lahnakoski
Suyash Salampuria
Shubham Kumar
ionutgoldan
Armen Zambrano G
Mellina Yonashiro
Dustin J. Mitchell