* Make changes to docs
* Add cert for prototype connections
* Add TLS_CERT_PATH variable to docker yaml file
* Change troubleshooting and database sections of docs
* Update docker-shared-user for pulse_url and add PROJECTS_TO_INGEST to backend container
* Update docs to make them clearer
* Fix exception caught in pytest.raises
* Switch test logic to use mozci for regressions
* bump timeout on web node to give leeway for longer-running queries
* make changes to mozci config file
* fix duplicate jobs in unstructured logs bug
* Use mozci labels to increase accuracy of build failures
* Combine duplicate health and health_summary API logic into helper function
* Expand mozci to query groups and labels
* clean up get_commit_history and CommitHistory component
* refactor lints and builds to use same jobs
* get_test_failure_jobs refactor to remove extra query
* refactor get_test_failures to utilize existing data
* only return what we need from Job query
* fix JS tests and cleanup console warnings
fix python push health lint, test and build tests
* expand exception handling and add logging in get_commit_history
add logging to _get_failure_data
* Add separate mozci config file for local development
* upgrade mozci to 2.0.0
Co-authored-by: Cameron Dawson <cdawson@mozilla.com>
Co-authored-by: Sarah Clements <sclements313@gmail.com>
This change changes the Docker set up to use the new Pulse listener command that listens to both tasks and pushes instead of having two different ones.
This also adds a unique queue name. When running `docker-compose up` the Pulse listener will use a queue name that is the same for every developer running the listener. This means that all developers would be consuming from the same queue and stealing each other's Pulse messages. This change creates a unique name for each developer.
Co-authored-by: Armen Zambrano <armenzg@users.noreply.github.com>
This permits setting up the complete ingestion pipeline with a single command.
It only ingests from autoland and tries to use minimal resources on a localhost.
Co-authored-by: Armen Zambrano <armenzg@users.noreply.github.com>
`initialize_data.sh` calls exec in the last line, thus, it was trying to execute `&&`
```
backend | /app/initialize_data.sh: line 11: exec: &&: not found
```
* Add Travis job to run Python tests outside of Docker
* `runtests.sh` is renamed to `runchecks.sh` and it does not run Python tests
* `manage.py check --deploy` was duplicated in Travis
* Update testing documentation
* Remove `-bb` since it is not needed since Python 3.5
Git based projects can list pushes in the UI incorrectly. This can be caused by commits having been ammended on a PR or merges of old commits.
Using the committer's date (the date when the PR gets merged) instead of the author's date to determine push time fixes the sorting problem.
This change also includes:
* Support for manual ingestion of Git pushes
* Support for ingesting the latest commits for a repo
* Script to compare pushes between Treeherder instances
## Script to compare pushes between Treeherder instances
`compare_pushes.py` compares the last 50 pushes of various projects for different Treeherder instances. The output generates links to each instance and revision to visually compare.
```console
% ./misc/compare_pushes.py --projects android-components,fenix,reference-browser,servo-master,servo-auto,servo-try
Comparing android-components against production.
Comparing fenix against production.
Comparing reference-browser against production.
{"values_changed": {"root['push_timestamp']": {"new_value": 1582580346, "old_value": 1582581477}}}
https://treeherder.allizom.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4ehttps://treeherder.mozilla.org/#/jobs?repo=reference-browser&revision=547a18b97534b237fa87bd22650f342836014c4e
Comparing servo-master against production.
Comparing servo-auto against production.
Comparing servo-try against production.
```
Work to support running Python tests outside of the Docker container.
This also helps with integration with IDEs and is prework for using Git hooks for precommit/prepush features.
* Development Docker container to initialize data
`docker-compose up` will now initialize the data for the local database.
This will also include initializing SETA's data.
* Change SETA's logging level and preseed information
The preseed information was all for Buildbot
The logging level changes is in preparation for when production switching to INFO messages instead of WARNING
* Docker container to log INFO messages instead of DEBUG messages
`LOGGING_LEVEL` is now set for the Docker container and it sets the logging
to `INFO` instead of `DEBUG`. The Django app will still be initialize in
debug mode but the default logging will be informative messages.
You can still log debugging messages by setting `LOGGING_LEVEL=DEBUG` before
initializing the Docker container.
Defining PULSE_TASK_SOURCES and PULSE_PUSH_SOURCES is a bit of a nightmare
because `app.json` is a Json file and using double and single quotes for a
value in the file is impossible.
* Bug 1574651 - remove unused JobConsumer and related code
The `update_pulse_test_fixtures` management command listens for job
messages, so is of no use anymore.
* Bug 1574651 - refactor pulse listening to support multiple AMQP servers
This looks forward to supporting ingesting jobs and tasks from multiple
Taskcluster depoyments, each of which is on its own AMQP server (or, at
least, a vhost).
* Bug 1574651 - pass rootUrl from pulse to celery, verify against repository
When jobs and pushes are loaded, the repo's root URL is known. This
just serves to ensure that the rootUrl for the repo and the rootUrl for
the event match up.
* Bug 1574651 - use root_url from message to make in-job URLs
* Bug 1574651 - update ingest-and-push-tasks to take --root-url
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
renovate[bot]
Sarah Clements
Cameron Dawson
Renovate Bot
ionutgoldan
Cameron Dawson
Armen Zambrano
Shubhank Saxena
Shubham Kumar
Suyash Salampuria
octavian-negru
Kyle Lahnakoski
Armen Zambrano G
Dustin J. Mitchell