This adds a new FailureClassification for autoclassified
intermittent. When a job is completely classified by the
autoclassifer, and it has the same number of structured and
unstructured error lines, it is marked as an autoclassified
intermittent.
Conversely, when there is exactly one structured and one unstructured
error line, the autoclassifier did not match the job, but has a
detector that could match the job, and the job is marked as
intermittent by a human, add a new autoclassification target
corresponding to the error line.
Since they're not specific to the Django app 'webapp'.
Whilst we're there, the local & example settings files have been
renamed. In the future I'd like to combine settings_local.example.py
with puppet/files/treeherder/local.vagrant.py, but I'll do that in
another bug.
I added a create_credentials command to help setting up the initial
development environment. The puppet setup now creates a new user and set
it as the owner of the treeherder-etl credentials.
This adds an autoclassify command and a detect_intermittents command.
The former is designed to take an incoming job with an error summary
and look for existing results marked as intermittent that are a close
match for the new result. At present only one matcher is implemented;
this requires an exact match in terms of test name, result and error
message. Matching is also constrained to be based on single lines; it
is anticipated that future iterations may add support for matching on
groups of lines.
The detect_intermittents command is designed to take a group of jobs
running on the same push and with the same build job (i.e. same
testsuite, same chunk, etc.) and look for new intermittents to add to
the database. This currently only looks for test failures where there
is at least one green job and one non-green job.
There is currently no UI for seeing matches or for adding new
prototypical intermittents as match candidates. There is also no
integration with bugzilla; future development should add association
of frequent intermittents with bugs.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
urllib isn't handling the unicode found in some log lines correctly,
whereas requests does. This prevents UnicodeEncodeError exceptions when
making the request to the bugscache API to find the bug suggestions for
these log lines.
After the previous commit, the Objectstore is effectively "dead code".
So this commit removes all the dead code after anything left over in
the Objectstore has been drained and added to the DB.
Having the ability to use different DB hosts for each project sounded
like a good idea, but in reality, we have no need for it.
This switches us to using the global read-write and read-only database
host names rather than the fields on the datasource table. As such, the
'host', 'read_only_host' and 'type' (eg 'mysql') fields can be removed.
The Django model had a unique_together on host+name, so we now need to
make 'name' (ie database name) a unique key on it's own.
In addition, this removes the 'creation_date' field, since we don't use
it anywhere, and we can just look at the commit history to see when a
repo was created. (I imagine it may have had more use if we actually had
started partitioning the databases uses the old 'dataset' count field).
In a future bug, I'll remove the redundant substitution of 'engine' for
'InnoDB' in the template schema, given that engine is now always InnoDB
in create_db().
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
Before this is deployed, we'll need to update the stage/prod puppet
configs & Heroku settings to add the new environment variable.
Since otherwise we may end up with interactive prompts.
Note: When using call_command() we instead have to use 'interactive'
instead of 'noinput' due to https://code.djangoproject.com/ticket/22985,
which is only fixed in Django 1.8+.
The datasource table has a 'dataset' field, to allow for multiple
datasources of the same type (for partitioning; eg the "1" in
`mozilla-central_jobs_1`). However we have never used it, so let's just
remove it.
This reverts commit e71e781565.
This commit caused pending and running jobs to be put into the objectstore.
This causes their completed versions not to be ingested.
* Create a generic TreeherderClient class
* Add a single method called `post_collection` which takes care of all
details of validation, submitting stuff and raising errors
* Also add a new update_parse_status method, for updating status (replaces
manual calls to post information on raw TreeherderRequest)
The treeherder client is in the vendor directory, however that doesn't
get added to the sys.path until settings/base.py is loaded, so defer the
import until we need it.
Generated using:
autopep8 --in-place --recursive --aggressive --aggressive
--max-line-length 999 --exclude='.git,__pycache__,.vagrant,build,vendor,
0001_initial.py,models.py,test_note_api.py,test_bug_job_map_api.py' .
autopep8's aggressive mode, unlike standard mode, makes non-whitespace
changes. It also uses lib2to3 to correct deprecated code (W690), some of
which aren't pep8 failures. Some of these changes are more dubious, but
rather than disable W690 completely, I've just excluded the files where
the unwanted changes would have been made, so we can benefit from the
rest.
Generated using:
autopep8 --in-place --recursive .
Before:
$ pep8 | wc -l
1686
After:
$ pep8 | wc -l
57
A later autopep8 run will be performed using --aggressive, which makes
non-whitespace changes too.
James Graham
Ed Morley
Mauro Doglio
Cameron Dawson
William Lachance
Vaibhav Agrawal
William Lachance
James Lal
Jonathan French
mdoglio
Jonathan Eads
vagrant