I added a create_credentials command to help setting up the initial
development environment. The puppet setup now creates a new user and set
it as the owner of the treeherder-etl credentials.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
Since bug 1140349, the objectstore endpoint has been deprecated, and
performs the same function as the jobs endpoint. Now that there are no
remaining submitters to it, let's remove it.
Since we use Celery for queueing job ingestion, the objectstore is
now irrelevant. This code is the first step. This will bypass
the Objectstore and ingest jobs directly to our ``jobs`` database.
Phase 2 is to remove all the Objectstore code (in a later commit)
Phase 3 is to delete the Objectstore databases and related fields in
other tables.
This introduces two new ways to generate ``Bug suggestions`` artifacts from
a ``text_log_summary`` artifact
1. POST a ``text_log_summary`` on the ``/artifact`` endpoint
2. POST a ``text_log_summary`` with a job on the ``/jobs`` endpoint.
Both of these cases will schedule an asynchronous task to generate the
``Bug suggestions`` artifact with ``celery``.
Artifact generation scenarios:
JobCollections
^^^^^^^^^^^^^^
Via the ``/jobs`` endpoint:
1. Submit a Log URL with no ``parse_status`` or ``parse_status`` set to "pending"
* This will generate ``text_log_summary`` and ``Bug suggestions`` artifacts
* Current *Buildbot* workflow
2. Submit a Log URL with ``parse_status`` set to "parsed" and a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact only
* Desired future state of *Task Cluster*
3. Submit a Log URL with ``parse_status`` of "parsed", with ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
ArtifactCollections
^^^^^^^^^^^^^^^^^^^
Via the ``/artifact`` endpoint:
1. Submit a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact if it does not already exist for that job.
2. Submit ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
* This is *Treeherder's* current internal log parser workflow
* Create a generic TreeherderClient class
* Add a single method called `post_collection` which takes care of all
details of validation, submitting stuff and raising errors
* Also add a new update_parse_status method, for updating status (replaces
manual calls to post information on raw TreeherderRequest)
This supports ingesting job ``log_references`` that have a
``parse_status`` value. This is so that external tools can submit jobs
that don’t require our internal log parsing. They will then submit
their own log summary artifact.
Mauro Doglio
Ed Morley
Cameron Dawson
William Lachance