Since it only speeds up parsing by a few percent of total runtime, and
is therefore not worth the added complexity for deployment and local
hack-test-debug cycles when working on the log parser.
The .gitignore and update.py entries will be removed in a later commit,
once the stage/prod src directories have been cleaned up.
In order that we can serve the UI on Heroku, we wrap the Django wsgi app
with WhiteNoise, so both the UI and API requests are served by gunicorn.
In the Vagrant environment, Apache has been removed and Varnish instead
now proxies all requests to gunicorn/Django runserver directly, without
Apache as a go-between.
The UI on production will not be affected by this commit, since the
Apache config there will still intercept requests for the UI assets
rather than proxying them to gunicorn.
It's worth noting too, that we're not able to make use of WhiteNoise's
automatic Django GZip/caching support since that assumes we are using
Django templates and referring to resources using {% static "foo.css" %}
However, we can sub-class WhiteNoise (or more specifically the
DjangoWhiteNoise class) and override the is_immutable_file() method to
add caching support at a later date:
http://whitenoise.evans.io/en/latest/base.html#caching-headers
Documentation for WhiteNoise can be found at:
http://whitenoise.evans.io/
The Varnish config uses |return (pass)| unconditionally, so never
caches anything, so there's no need to suggest restarting it after
making changes to the UI.
It's sometimes useful to be able to spin up an additional Vagrant
environment without affecting the first. To do this, we create two named
machines, both identical (since they inherit the main Vagrantfile
configuration) - one called default (to match the machine name used up
until now, so we don't force people to recreate their existing VMs when
we land this) and one called scratch.
The former is set to be the primary, so that single-machine commands
(eg `vagrant ssh`) work without having to append the machine name every
time.
The scratch machine has autostart set to false, so that it does not spin
up without explicitly using: `vagrant up scratch`
The name `scratch` is entirely arbitrary, and we can add additional
temporary machine names later, if people wish to have more than two
environments saved simultaneously.
For more information, see:
http://docs.vagrantup.com/v2/multi-machine/
These steps are already performed by puppet on provision, so don't need
to be performed again manually. They are mentioned on the "common tasks"
docs page if people need to run them again later.
Since gunicorn is only running on port 8000 inside the VM, it's then
proxied via varnish/apache and accessible on port 80 outside the VM, so
mentioning port 8000 is more confusing than helpful.
We're no longer using the in-repo puppet config in production, nor would I
recommend anyone do so again in the future. In addition, copying and pasting
default configs into the docs isn't great, since they are a pain to maintain.
Let's just remove this from the docs.
One of the few differences between vagrant.pp and production.pp is from
which directory the UI is served. This change adds the ability to toggle
the directory from the Vagantfile directly, thereby bringing
production.pp (and the duplication it brings) closer to being obsolete.
Instead of always displaying the directory file listing, if index.html
is present we serve that instead. This not only fixes the remaining
broken URLs when using web-server.js, but also means people can click
the "Starting web server at <URL>" link in the console and immediately
see Treeherder, without having to then manually navigate to index.html
themselves.
This introduces two new ways to generate ``Bug suggestions`` artifacts from
a ``text_log_summary`` artifact
1. POST a ``text_log_summary`` on the ``/artifact`` endpoint
2. POST a ``text_log_summary`` with a job on the ``/jobs`` endpoint.
Both of these cases will schedule an asynchronous task to generate the
``Bug suggestions`` artifact with ``celery``.
Artifact generation scenarios:
JobCollections
^^^^^^^^^^^^^^
Via the ``/jobs`` endpoint:
1. Submit a Log URL with no ``parse_status`` or ``parse_status`` set to "pending"
* This will generate ``text_log_summary`` and ``Bug suggestions`` artifacts
* Current *Buildbot* workflow
2. Submit a Log URL with ``parse_status`` set to "parsed" and a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact only
* Desired future state of *Task Cluster*
3. Submit a Log URL with ``parse_status`` of "parsed", with ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
ArtifactCollections
^^^^^^^^^^^^^^^^^^^
Via the ``/artifact`` endpoint:
1. Submit a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact if it does not already exist for that job.
2. Submit ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
* This is *Treeherder's* current internal log parser workflow
As part of merging the UI repo into this one, the following directory
moves were performed:
webapp/app/ -> ui/
webapp/test/ -> tests/ui/
webapp/config/ -> tests/ui/config/
webapp/scripts/ -> tests/ui/scripts/
webapp/scripts/web-server.js -> web-server.js
The UI docs are now being built at the same time as the main service
docs. Until we combine them, let's at least make sure the UI parts are
still discoverable, by adding them to the main table of contents.
* Create a generic TreeherderClient class
* Add a single method called `post_collection` which takes care of all
details of validation, submitting stuff and raising errors
* Also add a new update_parse_status method, for updating status (replaces
manual calls to post information on raw TreeherderRequest)
The 'treeherder-service' repo has been renamed to 'treeherder', ready
for when the treeherder-ui repo is imported into it. This means the
Github URL, Travis URL and directory name when cloned changes. The Read
The Docs URL cannot be changed, so for now we will leave as-is, and in
the future (once service and UI docs combined) we will create a new
project on RTD with name "treeherder".
This updates doc links and puppet/Vagrant configs, but leaves the
stage/prod deploy script alone, since renaming the directories on our
infra is non-trivial. The dev instance will need some TLC since unlike
stage/prod, it does use the puppet scripts in the repo.
At some point in the future, we may break the "Running the tests"
section out to its own file, but for now "Common Tasks" seems like a
better home than the installation instructions.
We're not currently using socketio - and if we start doing so in the
future we'll likely want to update to a newer version/adjust the
implementation anyway. Removing the dependencies from common.txt speeds
up the pip install on Travis. The old files will still be in version
control should we wish to refer to them :-)
Generated using:
autopep8 --in-place --recursive .
Before:
$ pep8 | wc -l
1686
After:
$ pep8 | wc -l
57
A later autopep8 run will be performed using --aggressive, which makes
non-whitespace changes too.
We only pass it in some cases, so setting it to anything other than the
default will break all the places where we don't use it. Django doesn't
require it is set, since it will fall back to the default port for that
engine type anyway.
* Separates out the steps required for running the tests, from those for
setting up a local instance.
* The running ingestion tasks step now explains what they are, that the
API server must be running already, and how to ingest just a single
revision for testing.
* The log parser compile step is moved inline, so it's harder to forget.
Ed Morley
William Lachance
Cameron Dawson
Jonathan French
Maja Frydrychowicz
Mauro Doglio
William Lachance
deshraj
AutomatedTester
Mauro Doglio
Jonathan Eads