This change changes the Docker set up to use the new Pulse listener command that listens to both tasks and pushes instead of having two different ones.
This also adds a unique queue name. When running `docker-compose up` the Pulse listener will use a queue name that is the same for every developer running the listener. This means that all developers would be consuming from the same queue and stealing each other's Pulse messages. This change creates a unique name for each developer.
Co-authored-by: Armen Zambrano <armenzg@users.noreply.github.com>
This permits setting up the complete ingestion pipeline with a single command.
It only ingests from autoland and tries to use minimal resources on a localhost.
Co-authored-by: Armen Zambrano <armenzg@users.noreply.github.com>
`initialize_data.sh` calls exec in the last line, thus, it was trying to execute `&&`
```
backend | /app/initialize_data.sh: line 11: exec: &&: not found
```
* Add Travis job to run Python tests outside of Docker
* `runtests.sh` is renamed to `runchecks.sh` and it does not run Python tests
* `manage.py check --deploy` was duplicated in Travis
* Update testing documentation
* Remove `-bb` since it is not needed since Python 3.5
Git based projects can list pushes in the UI incorrectly. This can be caused by commits having been ammended on a PR or merges of old commits.
Using the committer's date (the date when the PR gets merged) instead of the author's date to determine push time fixes the sorting problem.
This change also includes:
* Support for manual ingestion of Git pushes
* Support for ingesting the latest commits for a repo
* Script to compare pushes between Treeherder instances
## Script to compare pushes between Treeherder instances
`compare_pushes.py` compares the last 50 pushes of various projects for different Treeherder instances. The output generates links to each instance and revision to visually compare.
```console
% ./misc/compare_pushes.py --projects android-components,fenix,reference-browser,servo-master,servo-auto,servo-try
Comparing android-components against production.
Comparing fenix against production.
Comparing reference-browser against production.
{"values_changed": {"root['push_timestamp']": {"new_value": 1582580346, "old_value": 1582581477}}}
https://treeherder.allizom.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4ehttps://treeherder.mozilla.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4e
Comparing servo-master against production.
Comparing servo-auto against production.
Comparing servo-try against production.
```
Work to support running Python tests outside of the Docker container.
This also helps with integration with IDEs and is prework for using Git hooks for precommit/prepush features.
* Development Docker container to initialize data
`docker-compose up` will now initialize the data for the local database.
This will also include initializing SETA's data.
* Change SETA's logging level and preseed information
The preseed information was all for Buildbot
The logging level changes is in preparation for when production switching to INFO messages instead of WARNING
* Docker container to log INFO messages instead of DEBUG messages
`LOGGING_LEVEL` is now set for the Docker container and it sets the logging
to `INFO` instead of `DEBUG`. The Django app will still be initialize in
debug mode but the default logging will be informative messages.
You can still log debugging messages by setting `LOGGING_LEVEL=DEBUG` before
initializing the Docker container.
Defining PULSE_TASK_SOURCES and PULSE_PUSH_SOURCES is a bit of a nightmare
because `app.json` is a Json file and using double and single quotes for a
value in the file is impossible.
* Bug 1574651 - remove unused JobConsumer and related code
The `update_pulse_test_fixtures` management command listens for job
messages, so is of no use anymore.
* Bug 1574651 - refactor pulse listening to support multiple AMQP servers
This looks forward to supporting ingesting jobs and tasks from multiple
Taskcluster depoyments, each of which is on its own AMQP server (or, at
least, a vhost).
* Bug 1574651 - pass rootUrl from pulse to celery, verify against repository
When jobs and pushes are loaded, the repo's root URL is known. This
just serves to ensure that the rootUrl for the repo and the rootUrl for
the event match up.
* Bug 1574651 - use root_url from message to make in-job URLs
* Bug 1574651 - update ingest-and-push-tasks to take --root-url
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
Renovate Bot
Shubhank Saxena
Cameron Dawson
Shubham Kumar
Suyash Salampuria
Armen Zambrano
Sarah Clements
octavian-negru
Kyle Lahnakoski
renovate[bot]
Armen Zambrano G
Dustin J. Mitchell