Change of new environment variable `PULSE_PUSH_SOURCES`.
Keep old `publish-resultset-runnable-job-action` task name by creating a
method that points to `publish_push_runnable_job_action`.
Process output lines start with a prefix like GECKO(1234) or PID 1234
and then a pipe. This is an extra pipe symbol compared to other lines,
so our code that splits on | and assumes specific data in specific
positions is broken. Try to detect the process output case and discard
the first token so that we have the same fields as other
pipe-delimited data.
Fixes:
tests/autoclassify/test_classify_failures.py:7:1: F401 'treeherder.model.models.TextLogErrorMetadata' imported but unused
tests/etl/test_job_loader.py:7:1: F401 'treeherder.model.models.Repository' imported but unused
tests/model/test_classified_failure.py:6:1: F401 'treeherder.model.models.FailureLine' imported but unused
tests/seta/conftest.py:2:1: F401 'django.utils.timezone' imported but unused
tests/seta/test_job_priorities.py:8:1: F401 'treeherder.seta.settings.SETA_LOW_VALUE_PRIORITY' imported but unused
tests/webapp/api/test_text_log_summary_lines.py:4:1: F401 'treeherder.model.models.TextLogError' imported but unused
treeherder/auth/backends.py:13:5: F401 'django.utils.encoding.smart_str as smart_bytes' imported but unused
treeherder/autoclassify/tasks.py:4:1: F401 'django.conf.settings' imported but unused
treeherder/autoclassify/tasks.py:6:1: F401 'treeherder.celery_app' imported but unused
treeherder/perfalert/__init__.py:1:1: F401 '.perfalert.*' imported but unused
treeherder/seta/analyze_failures.py:7:1: F401 'treeherder.etl.seta.valid_platform' imported but unused
treeherder/seta/job_priorities.py:10:1: F401 'treeherder.model.models.Repository' imported but unused
treeherder/seta/models.py:7:1: F401 'treeherder.model.models.Repository' imported but unused
The seta migrations file change is due to the seta models no longer
depending on `model` (since the unnecessary `Repository` import has
been removed).
In most cases, but not all, we want to expire performance datum on the
same cadence as job data. Add some code to do this, with an optional
override to keep it around indefinitely for some cases.
tests/client/test_treeherder_client.py:514:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:66:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:94:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:112:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:148:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:166:1: E305 expected 2 blank lines after class or function definition, found 1
tests/perfalert/test_analyze.py:94:1: E305 expected 2 blank lines after class or function definition, found 1
tests/webapp/api/test_auth.py:23:1: E305 expected 2 blank lines after class or function definition, found 1
tests/webapp/api/test_version.py:12:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/config/settings.py:21:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/credentials/admin.py:10:1: E305 expected 2 blank lines after class or function definition, found 0
treeherder/etl/schema.py:16:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/model/search.py:56:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/model/tasks.py:42:1: E305 expected 2 blank lines after class or function definition, found 1
We were always resetting the failure classification to an invalid value
when the last job note was deleted for a job, but we didn't notice it
before because we didn't have foreign key validation.
* Can no longer store raw artifacts (anything treeherder doesn't understand
is ignored)
* Attempting to retrieve an artifact now returns a 405 (not allowed)
It appears that the intent of this code is to to a phrase match of the
search string against the bug summary for relevance matching. However
the code incorrectly tried to quote the string and as a result failed
to handle special characters in the AGAINST clause (e.g. + - ~ >
etc.). This change simply removes any existing quote characters from
the string and places the entire thing in quotes. Per the MySQL
documentations:
> A phrase that is enclosed within double quote (") characters
matches only rows that contain the phrase literally, as it was
typed
Instead, generate the data when required. We will store the return value
of this in memcache for a day to ensure things are responsive for the sheriffs
when classifying recent failures.
* Bug 1286578 - Retry job task if resultset doesn't exist
This removes the logic which creates `skeleton resultsets`
when a job is ingested that we don't have a resultset for yet.
The new approach is to fail and wait for the task to retry.
The buildbot job ingestion already skips and retries later if
it encounters a job for which it has no resultset.
This adds a similar check to the Pulse Job ingestion. If
a job comes in with a revision that doesn't have a resultset
yet, then this will raise a ValueError. That will invoke the
retryable_task actions which will wait a bit, then retry. Each
time it will wait a little longer to retry. After 9 retries it
waits something like 3900 seconds which should be plenty of time
for the resultset ingestion to complete.
This changes ingestion, the API endpoints, and the frontend to match
the new structure. For now we continue to store text_log_summary artifacts,
though they don't do anything anymore.
Previously test_new_job_in_exclusion_profile was attempting to download
logs from ftp.mozilla.org, due to the log parser not being mocked, which
caused intermittent test timeouts on Travis.
This is required in order to create a unique index on title,
value and job_id to prevent duplicates. The index will be
created in a later PR.
This also uses update_or_create instead of get_or_create as
this will be the mechanism going forward to prevent duplicates.
A prior commit removed the ability to use "-Infinity" for the last_modified
query param. However, the fix accidentally stripped the param entirely.
This change ensures that the value is a valid date string. The range is
not limited.
This also adds some new tests to ensure the param of `last_modified` is
working correctly when included.
Add support for matching test failures where the test, subtest, status,
and expected status are all exact matches, but the message is not an
exact match. The matching uses ElasticSearch and is initially optimised
for cases where the messages differ only in numeric values since this is
a relatively common case.
This commit also adds ElasticSearch to the travis environment.
* Bug 1275589 - Prune runnable jobs
Prior to this fix, the list of runnable jobs would be pruned only when
the normal cycle-data process ran. This is not fast enough, however.
Now, defunct buildernames are removed every time the runnable jobs are
updated.
* test fix
* fix
This is an obscure case (that we should probably handle better), but I
believe this patch should at least fix the new relic exception in the
common cases (specifically when we get the same retry with a different
guid in the same batch of jobs). This patch also adds a bunch of unit
tests to test ingesting batches of jobs containing retries. There are
many cases which still fail (which would probably never occur in real
life), but I'm going to put off fixing those for now -- see comments
in the unit tests for details.
When setting the bug number for a classified failure, first check for an
classified failure that already has that bug number, and if one exists,
update all references to the classified failure being changed to instead
point at the existing classification
Not really useful in-and-of-itself, but should be helpful by acting
as a centralized place to relate and map job information from what
is stored in the main treeherder database to what's in the per-project
databases. Eventually, we can extend this into a full-fledged jobs model.
New resultsets will still store a value in their ``revision_hash`` field, but it will
just be the same value as their ``long_revision`` field.
This will log an exception in New Relic when a new resultset or job is posted
to the API with only a ``revision_hash``and not a ``revision`` value.
This also switches to using the longer 40 char revisions along side the
12 char revisions. But we leverage the longer ones for most actions. The
short revisions are stored and used so that people and the UI can support
locating a resultset (or setting ranges) with short revisions.
It's almost entirely unnecessary (the few times we need base data
we can generate it ourselves) and can overwrite other test data
if we're not careful. Better just to remove it.
For future Python 3 compatibility.
(Whilst for 100% Python 3 equivalence we should be using
`from __future__ import print_function`, it doesn't seem worth it.)
This test checks that the `calculate_duration` command correctly
populates the `job_duration` table both on an initial run and when new
jobs cause an existing average duration to change.
It also checks that once the task has been run, any subsequent ingestion
of jobs with matching signatures cause those jobs to have the correct
`running_eta` value when inserted into the `job` table.
At the moment tests that store jobs can get away with not having the
test project listed in the repository table. However with later commits,
the new job_duration table will have a foreign key on the repository
table, which will break those tests unless they use the test_repository
fixture, which creates the repository record for them.
pytest-django doesn't setup a test database for every single test, but
only for those tests that actually require a db. Tests that require a db
need to either be marked with `@pytest.mark.django_db` or use a fixture
that has a dependency on `db` or `transactional_db`.
Using a non transactional db would make tests execution much faster, but
unfortunately it doesn't play well with the treeherder datasource
creation so I used a transactional_db.
pytest-django also allows you to specify a settings file to use for
tests in a pytest.ini file, which is nicer than monkeypatch the original
settings file in the pytest session start function 😃.
We store both long and short, but only utilize the short (as before). We
need to populate all the short and long revision records before we can
start using them. So after this commit, we will begin backfilling the
old records that don't yet have those values populated. Once they all
are, we can move to using the long_revision primarily in Bug 1199364.
By storing the duration as an int rather than a timedelta, we can set it
via the environment, rather than having to rely on local.py.
The variable has also been renamed to ease the transition and also make
its purpose clearer. Prod/Heroku use the default duration, and stage has
already had DATA_CYCLE_DAYS set to 45 in its environment, for parity
with the old stage local.py DATA_CYCLE_INTERVAL setting.
It's unused in the UI and doesn't add any value, since we're
normally much more interested in the push_timestamp.
This can land without causing errors, since the field is DEFAULT NULL,
so can be dropped at our leisure later.
They were never used for anything and take up a lot of space. They
also don't fit into the new performance model we're working on. We
can always bring back the useful bits later.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
After the previous commit, the Objectstore is effectively "dead code".
So this commit removes all the dead code after anything left over in
the Objectstore has been drained and added to the DB.
It appears that on occasion we parse a log more than once, which
resulted in duplicate performance series going into the database.
Let's be resilient about this by not inserting duplicate jobs into the
database (we always attach a unique job id to every datapoint, so there's
no chance of accidentally removing entries which happen to have the same
performance numbers)
Having the ability to use different DB hosts for each project sounded
like a good idea, but in reality, we have no need for it.
This switches us to using the global read-write and read-only database
host names rather than the fields on the datasource table. As such, the
'host', 'read_only_host' and 'type' (eg 'mysql') fields can be removed.
The Django model had a unique_together on host+name, so we now need to
make 'name' (ie database name) a unique key on it's own.
In addition, this removes the 'creation_date' field, since we don't use
it anywhere, and we can just look at the commit history to see when a
repo was created. (I imagine it may have had more use if we actually had
started partitioning the databases uses the old 'dataset' count field).
In a future bug, I'll remove the redundant substitution of 'engine' for
'InnoDB' in the template schema, given that engine is now always InnoDB
in create_db().
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
Before this is deployed, we'll need to update the stage/prod puppet
configs & Heroku settings to add the new environment variable.
Since it's unused, and hg.mozilla.org now has this information available
via its API.
Note: This commit depends on bug 1178719, to prevent issues during
deployment. Also, due to https://code.djangoproject.com/ticket/25036 a
migrate will need to be run interactively after deployment, to clean up
the old repositoryversion content type.
The datasource table has a 'dataset' field, to allow for multiple
datasources of the same type (for partitioning; eg the "1" in
`mozilla-central_jobs_1`). However we have never used it, so let's just
remove it.
Before if two celery jobs were updating the same series, one would overwrite
the other because the locking code did not actually work (it just always
unconditonally got a new lock without checking if anything was using it).
This fixes that.
This introduces two new ways to generate ``Bug suggestions`` artifacts from
a ``text_log_summary`` artifact
1. POST a ``text_log_summary`` on the ``/artifact`` endpoint
2. POST a ``text_log_summary`` with a job on the ``/jobs`` endpoint.
Both of these cases will schedule an asynchronous task to generate the
``Bug suggestions`` artifact with ``celery``.
Artifact generation scenarios:
JobCollections
^^^^^^^^^^^^^^
Via the ``/jobs`` endpoint:
1. Submit a Log URL with no ``parse_status`` or ``parse_status`` set to "pending"
* This will generate ``text_log_summary`` and ``Bug suggestions`` artifacts
* Current *Buildbot* workflow
2. Submit a Log URL with ``parse_status`` set to "parsed" and a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact only
* Desired future state of *Task Cluster*
3. Submit a Log URL with ``parse_status`` of "parsed", with ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
ArtifactCollections
^^^^^^^^^^^^^^^^^^^
Via the ``/artifact`` endpoint:
1. Submit a ``text_log_summary`` artifact
* Will generate a ``Bug suggestions`` artifact if it does not already exist for that job.
2. Submit ``text_log_summary`` and ``Bug suggestions`` artifacts
* Will generate nothing
* This is *Treeherder's* current internal log parser workflow
Fixes:
tests/model/derived/test_jobs_model.py:343:23: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
treeherder/__init__.py:11:1: E731 do not assign a lambda expression, use a def
treeherder/etl/buildapi.py:107:16: E713 test for membership should be 'not in'
treeherder/log_parser/utils.py:183:33: W503 line break before binary operator
treeherder/model/derived/base.py:73:12: E713 test for membership should be 'not in'
treeherder/model/derived/base.py:82:12: E713 test for membership should be 'not in'
treeherder/model/derived/jobs.py:1998:26: W503 line break before binary operator
Generated using:
autopep8 --in-place --recursive .
Before:
$ pep8 | wc -l
1686
After:
$ pep8 | wc -l
57
A later autopep8 run will be performed using --aggressive, which makes
non-whitespace changes too.
Added several parameters to the cycle_data shell command: cycle-interval (in days),
chunk-size (in number of result sets), sleep-time (in seconds).
I made the cycle_data task a very thin wrapper around the shell command,
there is no more logic in it.
All the queries for data cycling are executed with the retry logic to
handle db deadlocks
We divide bug suggestions for a search term into 'open_recent' and
'all_others'. The former is supposed to be group #1 below, and the
latter groups 2-4, with the summation of the two groups corresponding
to every bug whose summary matches the search term.
1) Open + recently modified
2) Open + not recently modified
3) Resolved + recently modified
4) Resolved + not recently modified
However prior to this patch group #3 was not being returned at all, when
it should have been included in all_others.
Test that we are treating the search term literally in the LIKE
statement, and so have correctly escaped any underscores, percent signs
or escape symbols.
Max Chehab
KWierso
Ed Morley
James Graham
William Lachance
William Lachance
camd
Kalpesh Krishna
ka7
Shruti Jasoria
autolander
Tiramisu 1993
Mauro Doglio
mdoglio
Jonathan French
Jonathan Eads