Github based projects can list pushes in the UI incorrectly. This is caused by commits that have been ammended.
This code change switches to grab the push time from the `timestamp` field of the head commit from the Pulse event rather that Github's APIs. This fixes the problem of push sorting.
Note, that the field `timestamp` only exists in the Push event that the Pulse message contains. This field can be seen in the `events` API, however, that API contains all sorts of events and holds a maximum of 300 events.
This change also includes partial support for manual ingestion of Git pushes.
* CSS Cleanup
* Use darker-info and darker-secondary for a11y
* Put Parent Push metric at end of list and clean it up
* Add a `scrollToLine` when clicking/expanding metric names
Git based projects can list pushes in the UI incorrectly. This is caused by commits that have been ammended.
This code change switches to use the `timestamp` field of head commit to determine push time rather a commit's authorship date. This fixes the problem of push sorting.
This change also includes support for manual ingestion of Git pushes.
Add style specs in case darker-secondary and darker-info color props are used in other reactstrap components in the future. Update docs and reorganize these classes in the stylesheet.
* Properly configure LOGGING_LEVEL
`LOGGING_LEVEL` was being cast down to False rather than "INFO" or "DEBUG".
There were no fallouts from it but it was preventing setting the right logging
level for the Docker container or the Heroku Review Apps.
* Validate preseed entries and fail if issues are found
Without validation some invalid data job priorities would be inserted into the table. No issues
would be noticed for SETA since the web API sanitizes the data before presenting it.
This section was confusing for a contributor since it does not actually work (The MySql container is not ready) and the Django migrations are now automatically applied when running `docker-compose up`.
* Development Docker container to initialize data
`docker-compose up` will now initialize the data for the local database.
This will also include initializing SETA's data.
* Change SETA's logging level and preseed information
The preseed information was all for Buildbot
The logging level changes is in preparation for when production switching to INFO messages instead of WARNING
* Docker container to log INFO messages instead of DEBUG messages
`LOGGING_LEVEL` is now set for the Docker container and it sets the logging
to `INFO` instead of `DEBUG`. The Django app will still be initialize in
debug mode but the default logging will be informative messages.
You can still log debugging messages by setting `LOGGING_LEVEL=DEBUG` before
initializing the Docker container.
Defining PULSE_TASK_SOURCES and PULSE_PUSH_SOURCES is a bit of a nightmare
because `app.json` is a Json file and using double and single quotes for a
value in the file is impossible.
- update repositories fixture with new tc_root_urls for firefox-ci and community-ci
- update getRunnableJobsUrl and RunnableJobModel to access currentRepo and tc_root_url;
refactor getInspectTaskUrl and JobInfo component to have currentRepo prop
- add new rooturls to connect-src, update rooturl for seta, update docs and
remove header in make_request,
* Bug 1574651 - remove unused JobConsumer and related code
The `update_pulse_test_fixtures` management command listens for job
messages, so is of no use anymore.
* Bug 1574651 - refactor pulse listening to support multiple AMQP servers
This looks forward to supporting ingesting jobs and tasks from multiple
Taskcluster depoyments, each of which is on its own AMQP server (or, at
least, a vhost).
* Bug 1574651 - pass rootUrl from pulse to celery, verify against repository
When jobs and pushes are loaded, the repo's root URL is known. This
just serves to ensure that the rootUrl for the repo and the rootUrl for
the event match up.
* Bug 1574651 - use root_url from message to make in-job URLs
* Bug 1574651 - update ingest-and-push-tasks to take --root-url
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 5 terminal windows and follow these steps:
* On the first window run `docker-compose up`
* On the next three windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_tasks`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the last window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Extra script =
Script that permits comparing pushes from two different Treeherder instances.
```
usage: Compare a push from a Treeherder instance to the production instance.
[-h] [--host HOST] --revision REVISION [--project PROJECT]
optional arguments:
-h, --help show this help message and exit
--host HOST Host to compare. It defaults to localhost
--revision REVISION Revision to compare
--project PROJECT Project to compare. It defaults to mozilla-central
```
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
* Bug 1395254 - Consume Taskcluster Pulse messages from standard queue exchanges
Currently, Treeherder consumes Pulse messages from an intermediary service called `taskcluster-treeherder`.
Such service needs to be shut down and its functionality imported into Treeherder.
In order to do this we need to switch to the standard Taskcluster exchanges as defined in here:
https://docs.taskcluster.net/docs/reference/platform/queue/exchanges
On a first pass we are only including the code from `taskcluster-treeherder` without changing
much of Treeherder's code. The code is translated from Javascript to Python and only some minor
code changes were done to reduce the difficulty on porting the code without introducing bugs.
Internally, on this first pass, we will still have an intermediary data structure representing
what `taskcluster-treeherder` is emitting, however, we will stop consuming the messages
from it and be able to shut it down.
Instead of consuming from one single exchange we will be consuming multiple ones. Each one representing
a different kind of task (e.g. pending vs running).
In order to test this change you need to open 4 terminal windows and follow these steps:
* On the first two windows `export PULSE_URL="amqp://foo:bar@pulse.mozilla.org:5671/?ssl=1"` and run the following commands:
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_jobs`
* `docker-compose run -e PULSE_URL backend ./manage.py pulse_listener_pushes`
* On the third window run `docker-compose run backend celery -A treeherder worker -B --concurrency 5`
* On the last window run `docker-compose up`
* Open on your browser `http://localhost:5000`
This is just a summary from [the docs](https://treeherder.readthedocs.io/pulseload.html).
= ETL management commands =
This change also introduces two ETL management command that can be executed like this:
== Ingest push and tasks ==
This script can ingest into Treeherder all tasks associated to a push.
It uses Python's asyncio to speed up the ingestion of tasks.
```bash
./manage.py ingest_push_and_tasks
```
== Update Pulse test fixtures ==
```bash
./manage.py update_pulse_test_fixtures
```
This command will read 100 Taskcluster Pulse messages, process them and store them as test fixtures
under these two files: `tests/sample_data/pulse_consumer/taskcluster_{jobs,metadata}.json`
Following this work would be to get rid of the intermediary job representation ([bug 1560596](https://bugzilla.mozilla.org/show_bug.cgi?id=1560596) which will
clean up some of the code and some of the old tests.
= Other changes =
Other changes included:
* Import `taskcluster-treeherder`'s validation to ensure we're not fed garbage.
* Change `yaml.load(f)` for `yaml.load(f, Loader=yaml.FullLoader)`. Read [this](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) for details
* Introduce `taskcluster` and `taskcluster-urls` as dependencies
* The test `test_retry_missing_revision_never_succeeds` makes no sense because
we make Json validation on the Pulse message
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
- Adds a new "infrastructure" section to the docs, which describes
architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
docs, to make the documentation easier to navigate.
Since they are out of sync with `Procfile` and long term are best replaced
with one of the solutions in bug 1318020.
The development setup instructions have been updated to stop recommending
gunicorn, since Django's runserver is more appropriate for beginners
(and I'm pretty sure the memory leaks seen previously no longer occur).
The latest policy used in the report-only header has been working well
on production (the violation reports logged to New Relic are only from
scripts injected by browser addons), so we're ready to start enforcing
the policy by using the real `Content-Security-Policy` header name.
NB: When features are added in the future, PR authors and reviewers will
need to remember to update the policy if needed (for example to add domains
to the `connect-src` directive). The CSP header is not enabled when using
`webpack-dev-server` (it would break dev source maps and react-hot-loader)
so if in doubt test locally (using `yarn build` and serving via Django
runserver) or on prototype first.
See:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CSPhttps://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy
To ensure the CI and local environments match stage/production (where
Elasticsearch is not configured) and don't run differing code-paths due
to the various `ELASTICSEARCH_URL` conditionals used in auto-classify.
At such time when work on the fuzzy auto-classify matcher continues,
this can be reverted.
Also remove all Karma support and update the docs to only mention ``Jest``.
One of the test files was testing some AngularJS filters. I converted these
tests to test the equivalent helper functions.
Since it's only used by the frontend as a fallback for when it cannot
find a non-gzipped version of `runnable-jobs.json`, and enough time has
now passed for all jobs to have that file.
Switches from Sphinx to MkDocs, since it:
* supports Markdown natively without requiring hacks like `eval_rst`
* validates inline links, ensuring that they are not broken
* has a more pleasant live-reloading dev-server
* supports the nicer looking mkdocs-material theme
* is a third of the size of Sphinx (including deps)
The theme change is now possible since Read the Docs have just started
supporting use of custom themes (previously they would override the theme
and use `readthedocs` theme regardless).
Currently Prettier is used via the ESLint integration, which only supports
JS/JSX and not the other filetypes Prettier is able to format.
For now these additional filetypes are excluded by `.prettierignore`
entries, however these will be removed/fixes applied in later commits.
So that their purposes are clearer and it's easier to differentiate
between generated content and files committed to the repository.
* Neutrino build: `build/` -> `.build/`
* Sphinx build: `_build/` -> `.build-docs/`
* Django collectstatic: `treeherder/static/` -> `.django-static/`
Since it's more reliable (and strict) at code formatting than ESLint.
We use it via an ESLint plugin, and so disable the style-related AirBnB
preset rules, leaving the AirBnB guide to handle only correctness and
best practices rules.
It's highly encouraged to use an IDE integration or Git commit hook
to run Prettier (or `yarn lint --fix`) automatically. See:
* https://prettier.io/docs/en/editors.html
* https://prettier.io/docs/en/precommit.html
We may consider enabling a git commit hook out of the box (using eg
Husky) in the future, however they have previously been known to
interfere with partial-staging workflows, so would need to test the
fixes they have made for them thoroughly first.
In future PRs we may also want to start formatting JSON/CSS/Markdown
using Prettier too.
Imports must now be grouped like so (with newlines between each):
```
// "external" modules
import _ from 'lodash';
import chalk from 'chalk';
// modules from a "parent" directory
import foo from '../foo';
import qux from '../../foo/qux';
// "sibling" modules from the same or a sibling's directory
import bar from './bar';
import baz from './bar/baz';
```
The `import/order` rule has auto-fix support, so any errors can be
resolved using `yarn lint --fix`.
See:
https://github.com/benmosher/eslint-plugin-import/blob/master/docs/rules/order.md
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.
Highlights of the upgrade are:
* Major version updates to the individual tools within (such as webpack,
Babel and ESLint), significantly improving performance, fixing
transpilation/minification correctness bugs, adding support for newer
ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
automatic shared/vendor code chunk splitting (no need for the manually
maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
frontend needs, reducing maintenance burden and allowing for easier
future Neutrino upgrades.
In addition this PR:
* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
compilation errors in the browser, saving having to switch back
to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
which saves having to manually navigate to `localhost:5000` after
running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
correctly on Windows, even when setting environment variables.
Performance comparison:
* Local `yarn build`:
- Cached: 2m34s -> 23s
- Uncached: 2m34s -> 58s
* Local `yarn start`:
- Cached: 34.5s -> 13.6s
- Uncached: 34.5s -> 31.3s
* Local `yarn test`
- Cached: 61.5s -> 19.8s
- Uncached: 61.5s -> 22.0s
* Local `yarn lint`
- Cached: 3.8s -> 1.8s
- Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
14 minutes -> 9 minutes
* Enables the display of skipped test/expected fail reasons, in
the pytest summary.
* Skips the Selenium tests with a clear reason message, unless the
built UI is found (preventing the annoying/confusing test timeouts).
* Removes the disabling of the `pytest-html` and `pytest-metadata`
plugins, since they are required when passing the `--html` option
to generate an HTML report.
* Updates the docs to mention `yarn build` and `--html`.
Now that autophone and AWFY have migrated to Taskcluster, there are
no more submitters of jobs to the REST API (confirmed via New Relic
Insights). As such, this deprecated data ingestion method can now be
removed, along with support for API Hawk auth, API POST throttling
and `treeherder-client` job submission capability.
After this lands we'll need to manually drop the `credentials` table.
Runnable jobs for buildbot were calculated via a celerybeat task
(that was disabled in #4007) and the results stored in the
`runnable_jobs` table. This can all be removed now that buildbot is
EOL, since the remaining support for Taskcluster runnable jobs does
not use that celery task/Django model.
We don't currently change the push ingestion sources in any of our envs
(prod, staging, prototype, test, dev) from their defaults. This
hardcodes those unchanging values and documents where to change them
should an engineer need to.
This reverts commit 1728dd306c (#3884), due to:
```
Pulse warning: queue "queue/treeherder-staging/jobs" is overgrowing
Warning: your queue "queue/treeherder-staging/jobs" on exchange "could not be determined" is
overgrowing (4083 ready messages, 4083 total messages).
...
```
Renaming as a separate commit since otherwise Git was struggling to
identify the renames, making the diff harder to review.
`index.rst` has been left unchanged, since we still want to use the
`toctree` directive for now.
We want to switch from reStructuredText to Markdown, since:
* it's more user-friendly
* it is used in multiple places on GitHub, so people are more familiar
with it / context switching between the two is annoying
* the disadvantages of markdown typically raised as reasons to prefer
reST either don't apply to our use-case, or are no longer accurate
with recent tooling/extensions
* switching means we can also later start using mkdocs (which doesn't
support reST) instead of Sphinx. That change has been kept separate
to reduce the size of the PR, and to make the comparison between
the two more fair.
See:
https://docs.readthedocs.io/en/latest/getting_started.html#in-markdownhttps://recommonmark.readthedocs.io/en/latest/index.html
Previously the mixture of heading types was confusing the RTD theme,
causing the navigation sidebar to display at the top level not just
the page links, but links to subsections within those pages too.
The subsection names taken out of context were pretty confusing,
making it hard to navigate the docs.
The architecture pages are significantly out of date and so have
been removed until such time as new ones are created (this will be
simpler once Buildbot/REST API support is removed later this year).
In addition, the deployment page duplicated the UI installation page.
The `--provision` flag is unnecessary for new instances, since Vagrant
runs `provision` as part of the initial `up` regardless. However by
including `--provision` in the suggested command, it will mean
returning contributors will re-run provision when they re-use the
same steps.
An alternative approach would be to mark the provision block in
`Vagrantfile` as `run: "always"`, however that would slow down the
`vagrant up` of core developers too (by 40s), with no way to opt-out.
Previously it was not possible to test features that required an
authenticated user when:
* using `yarn start` with Vagrant (bug 1363722), which meant slower
watch builds
* pointing the UI at the prod/stage API (bug 1317752), which was
extremely limiting
Now login works in all environments, since the frontend no longer uses a
URL prefix, but instead webpack-dev-server proxies non-webpack URLs to
the chosen `BACKEND_DOMAIN` - avoiding cross-domain issues. Cookies are
rewritten to remove any `secure` directive (which is set on production),
so that they can still be read from HTTP localhost. The `Referer` has to
also be changed to stop Django's CSRF checks from rejecting request.
The slower "build into `dist` and watch" mode is therefore no longer
necessary, so `yarn start:local` instead invokes webpack-dev-server just
like `yarn start` - and the `local-watch.js` workaround has been
removed.
Support for the "publish to GitHub with hardcoded `SERVICE_DOMAIN`"
workflow has been dropped, since it was already rarely used and there is
no way to make it support login.
The API domain environment variable was renamed to `BACKEND_DOMAIN` to
avoid potential confusion given it no longer behaves the same as
`SERVICE_DOMAIN` used to.
NB: For full stack Vagrant workflows users must now connect to port
*5000* on localhost, not 8000.
This helps prevent:
https://www.owasp.org/index.php/Reverse_Tabnabbing
We're not also using `noreferrer`, since most browsers now support
`noopener` (https://caniuse.com/#search=noopener) and the link targets
are all Mozilla properties where the referrer may be useful.
The auth.js `window.open()` has not been changed, since the login
callback makes use of `window.opener`.
With ES6, the `'use strict'` directives are unnecessary:
https://eslint.org/docs/rules/strict
The directives have been left in the Neutrino configs, since they
are used by node directly, which doesn't yet support ES6 modules.
Now that we're using MySQL 5.7, we can specify `REQUIRE SSL` on the
`CREATE USER` statement, rather than having to do so on the individual
GRANTs. Compare:
https://dev.mysql.com/doc/refman/5.6/en/create-user.htmlhttps://dev.mysql.com/doc/refman/5.7/en/create-user.html
Prevents:
```
1 warning(s): 1287 Using GRANT statement to modify existing user's
properties other than privileges is deprecated and will be removed
in future release. Use ALTER USER statement for this operation.
```
Generated using the approach documented at the end of the page:
https://treeherder.readthedocs.io/admin.html#direct-database-access
The changes are required since bug 1373008 added the `group` and
`group_failure_lines` tables and #2532 removed `text_log_summary`
and `text_log_summary_line`.
The UI has already been removed. This cleans up the data ingestion
and removes the `JobDuration` model, however leaves the `running_eta`
field on the `Job` model for the next time that table is touched (since
the table is large, so altering the schema would likely require
downtime).
Most of the existing config (generated by `sphinx-quickstart`) was
either commented out, identical to the defaults or else for build
modes we do not use - and only end up making the config hard to read.
* Removes the unnecessary initial `make html`, since sphinx-autobuild
performs the initial build itself.
* Enables `--poll` watch mode, since file change detection otherwise
doesn't work in Vagrant.
* Updates the docs to recommend the livereload web server over the
one-off build when working on docs locally.
The build previously contained warnings like:
```
docs/rest_api.rst:144: WARNING: Title underline too short.
docs/admin.rst:18: WARNING: Could not lex literal_block as "sql". Highlighting skipped.
WARNING: html_static_path entry u'.../docs/_static' does not exist
```
Now that no submissions are using revision_hash, it can be removed.
This removes everything but the model field, which will be handled
later.
I've removed revision_hash from the Pulse jobs schema without bumping
the version, which wouldn't normally be ok, but no one is still using
it, and I'd rather have explicit failures later than if we left the
schema unchanged.
Since for people installing node/yarn on their own (outside the
Vagrant environment) it can be confusing to work out which version is
appropriate, given node always has both an LTS and current release.
Shubhank Saxena
Armen Zambrano
Cameron Dawson
Armen Zambrano G
Mellina Yonashiro
Kyle Lahnakoski
Sarah Clements
Dustin J. Mitchell
Ed Morley
KWierso
George Hickman
George Hickman
ionutgoldan
Jonathan French