Add support for matching test failures where the test, subtest, status,
and expected status are all exact matches, but the message is not an
exact match. The matching uses ElasticSearch and is initially optimised
for cases where the messages differ only in numeric values since this is
a relatively common case.
This commit also adds ElasticSearch to the travis environment.
* Bug 1275589 - Prune runnable jobs
Prior to this fix, the list of runnable jobs would be pruned only when
the normal cycle-data process ran. This is not fast enough, however.
Now, defunct buildernames are removed every time the runnable jobs are
updated.
* test fix
* fix
This is an obscure case (that we should probably handle better), but I
believe this patch should at least fix the new relic exception in the
common cases (specifically when we get the same retry with a different
guid in the same batch of jobs). This patch also adds a bunch of unit
tests to test ingesting batches of jobs containing retries. There are
many cases which still fail (which would probably never occur in real
life), but I'm going to put off fixing those for now -- see comments
in the unit tests for details.
When setting the bug number for a classified failure, first check for an
classified failure that already has that bug number, and if one exists,
update all references to the classified failure being changed to instead
point at the existing classification
Not really useful in-and-of-itself, but should be helpful by acting
as a centralized place to relate and map job information from what
is stored in the main treeherder database to what's in the per-project
databases. Eventually, we can extend this into a full-fledged jobs model.
New resultsets will still store a value in their ``revision_hash`` field, but it will
just be the same value as their ``long_revision`` field.
This will log an exception in New Relic when a new resultset or job is posted
to the API with only a ``revision_hash``and not a ``revision`` value.
This also switches to using the longer 40 char revisions along side the
12 char revisions. But we leverage the longer ones for most actions. The
short revisions are stored and used so that people and the UI can support
locating a resultset (or setting ranges) with short revisions.
It's almost entirely unnecessary (the few times we need base data
we can generate it ourselves) and can overwrite other test data
if we're not careful. Better just to remove it.
For future Python 3 compatibility.
(Whilst for 100% Python 3 equivalence we should be using
`from __future__ import print_function`, it doesn't seem worth it.)
This test checks that the `calculate_duration` command correctly
populates the `job_duration` table both on an initial run and when new
jobs cause an existing average duration to change.
It also checks that once the task has been run, any subsequent ingestion
of jobs with matching signatures cause those jobs to have the correct
`running_eta` value when inserted into the `job` table.
At the moment tests that store jobs can get away with not having the
test project listed in the repository table. However with later commits,
the new job_duration table will have a foreign key on the repository
table, which will break those tests unless they use the test_repository
fixture, which creates the repository record for them.
pytest-django doesn't setup a test database for every single test, but
only for those tests that actually require a db. Tests that require a db
need to either be marked with `@pytest.mark.django_db` or use a fixture
that has a dependency on `db` or `transactional_db`.
Using a non transactional db would make tests execution much faster, but
unfortunately it doesn't play well with the treeherder datasource
creation so I used a transactional_db.
pytest-django also allows you to specify a settings file to use for
tests in a pytest.ini file, which is nicer than monkeypatch the original
settings file in the pytest session start function 😃.
We store both long and short, but only utilize the short (as before). We
need to populate all the short and long revision records before we can
start using them. So after this commit, we will begin backfilling the
old records that don't yet have those values populated. Once they all
are, we can move to using the long_revision primarily in Bug 1199364.
By storing the duration as an int rather than a timedelta, we can set it
via the environment, rather than having to rely on local.py.
The variable has also been renamed to ease the transition and also make
its purpose clearer. Prod/Heroku use the default duration, and stage has
already had DATA_CYCLE_DAYS set to 45 in its environment, for parity
with the old stage local.py DATA_CYCLE_INTERVAL setting.
It's unused in the UI and doesn't add any value, since we're
normally much more interested in the push_timestamp.
This can land without causing errors, since the field is DEFAULT NULL,
so can be dropped at our leisure later.
They were never used for anything and take up a lot of space. They
also don't fit into the new performance model we're working on. We
can always bring back the useful bits later.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
After the previous commit, the Objectstore is effectively "dead code".
So this commit removes all the dead code after anything left over in
the Objectstore has been drained and added to the DB.
It appears that on occasion we parse a log more than once, which
resulted in duplicate performance series going into the database.
Let's be resilient about this by not inserting duplicate jobs into the
database (we always attach a unique job id to every datapoint, so there's
no chance of accidentally removing entries which happen to have the same
performance numbers)
Having the ability to use different DB hosts for each project sounded
like a good idea, but in reality, we have no need for it.
This switches us to using the global read-write and read-only database
host names rather than the fields on the datasource table. As such, the
'host', 'read_only_host' and 'type' (eg 'mysql') fields can be removed.
The Django model had a unique_together on host+name, so we now need to
make 'name' (ie database name) a unique key on it's own.
In addition, this removes the 'creation_date' field, since we don't use
it anywhere, and we can just look at the commit history to see when a
repo was created. (I imagine it may have had more use if we actually had
started partitioning the databases uses the old 'dataset' count field).
In a future bug, I'll remove the redundant substitution of 'engine' for
'InnoDB' in the template schema, given that engine is now always InnoDB
in create_db().
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
Before this is deployed, we'll need to update the stage/prod puppet
configs & Heroku settings to add the new environment variable.
Since it's unused, and hg.mozilla.org now has this information available
via its API.
Note: This commit depends on bug 1178719, to prevent issues during
deployment. Also, due to https://code.djangoproject.com/ticket/25036 a
migrate will need to be run interactively after deployment, to clean up
the old repositoryversion content type.
The datasource table has a 'dataset' field, to allow for multiple
datasources of the same type (for partitioning; eg the "1" in
`mozilla-central_jobs_1`). However we have never used it, so let's just
remove it.
Before if two celery jobs were updating the same series, one would overwrite
the other because the locking code did not actually work (it just always
unconditonally got a new lock without checking if anything was using it).
This fixes that.
This introduces two new ways to generate ``Bug suggestions`` artifacts from
a ``text_log_summary`` artifact
1. POST a ``text_log_summary`` on the ``/artifact`` endpoint
2. POST a ``text_log_summary`` with a job on the ``/jobs`` endpoint.
Both of these cases will schedule an asynchronous task to generate the
``Bug suggestions`` artifact with ``celery``.
Artifact generation scenarios:
JobCollections
^^^^^^^^^^^^^^
Via the ``/jobs`` endpoint:
1. Submit a Log URL with no ``parse_status`` or ``parse_status`` set to "pending"
* This will generate ``text_log_summary`` and ``Bug suggestions`` artifacts
* Current *Buildbot* workflow
2. Submit a Log URL with ``parse_status`` set to "parsed" and a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact only
* Desired future state of *Task Cluster*
3. Submit a Log URL with ``parse_status`` of "parsed", with ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
ArtifactCollections
^^^^^^^^^^^^^^^^^^^
Via the ``/artifact`` endpoint:
1. Submit a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact if it does not already exist for that job.
2. Submit ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
* This is *Treeherder's* current internal log parser workflow
Fixes:
tests/model/derived/test_jobs_model.py:343:23: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
treeherder/__init__.py:11:1: E731 do not assign a lambda expression, use a def
treeherder/etl/buildapi.py:107:16: E713 test for membership should be 'not in'
treeherder/log_parser/utils.py:183:33: W503 line break before binary operator
treeherder/model/derived/base.py:73:12: E713 test for membership should be 'not in'
treeherder/model/derived/base.py:82:12: E713 test for membership should be 'not in'
treeherder/model/derived/jobs.py:1998:26: W503 line break before binary operator
Generated using:
autopep8 --in-place --recursive .
Before:
$ pep8 | wc -l
1686
After:
$ pep8 | wc -l
57
A later autopep8 run will be performed using --aggressive, which makes
non-whitespace changes too.
Added several parameters to the cycle_data shell command: cycle-interval (in days),
chunk-size (in number of result sets), sleep-time (in seconds).
I made the cycle_data task a very thin wrapper around the shell command,
there is no more logic in it.
All the queries for data cycling are executed with the retry logic to
handle db deadlocks
We divide bug suggestions for a search term into 'open_recent' and
'all_others'. The former is supposed to be group #1 below, and the
latter groups 2-4, with the summation of the two groups corresponding
to every bug whose summary matches the search term.
1) Open + recently modified
2) Open + not recently modified
3) Resolved + recently modified
4) Resolved + not recently modified
However prior to this patch group #3 was not being returned at all, when
it should have been included in all_others.
Test that we are treating the search term literally in the LIKE
statement, and so have correctly escaped any underscores, percent signs
or escape symbols.
William Lachance
jgraham
camd
autolander
Ed Morley
Tiramisu 1993
Mauro Doglio
mdoglio
Jonathan French
Jonathan Eads
pnispel
vagrant