This reverts commit fa533ff25a.
Ends up there's a bug with this change. The plan is for @ionutgoldan to create a new PR with just the migration and model change for PerformanceDatum which we can deploy. Then he can re-implement his code changes from this PR and we'll already be done with the lengthy migration to add the index.
Add index to performance_datum.push_timestamp field
Add more expire parameters for Perfherder data cycler
Don't keep perf data indefinitely
Have a debugger which logs even on production
Whenever we create a BugJobMap with a User we want to update its Job's
best FailureClassification's Bug number with the one the BugJobMap was
created with. This keeps the autoclassify data (FailureClassification)
in sync with BugJobMap.
However, doing this via an overridden `.save()` effectively hides the
functionality from anyone reading through the code. Since the
`.update_autoclassification_bug()` method is only called from that one
place this moves all the functionality into the classmethod `.create()`.
This makes it explicit that creating a BugJobMap involves more than a
simple DB row creation.
This method was exclusively used in auto-classification making it better
suited to live there. As an added benefit we can remove some complexity
because we know all TextLogError instances in auto-classification have
related FailureLines.
* Remove unused methods
* Use TextLogError.set_classification instead of FailureLine's version since they're the same effective code it's better to have a single entry point.
* Remove FailureLine.mark_best_classification_verified for TextLogError's to create a single place to mark a classification as verified.
* Get related FailureLine in a clearer way
* Only save changed fields
* Use decorator form of transaction.atomic to avoid extra indentation
* Use count on QuerySets instead of len since len() triggers a DB query and then counts the objects in memory while .count() triggers a query returning the result.
* Pull classified failure we're comparing against out into a variable
* Get objects from the db instead of refreshing in place
* Remove use of FailureMatch object/relation in favour of TLEMatch
* Manually build the matches output for the failure lines API
* Pull expected object out of list
* Manually build the classified failures output for the failure lines API
change BugJobMap bug_id field to foreign key field to create a join on
Bugscache model; add whiteboard to bugzilla, add index to Push time
field for search by date range; update associated tests
The UI has already been removed. This cleans up the data ingestion
and removes the `JobDuration` model, however leaves the `running_eta`
field on the `Job` model for the next time that table is touched (since
the table is large, so altering the schema would likely require
downtime).
pytest treats objects starting with the string "Test" as tests, so an
underscore prefix has been added to prevent warnings of form:
```
WC1 .../test_detect_intermittents.py cannot collect test class 'TestFailureDetector' because it has a __init__ constructor
```
Change of new environment variable `PULSE_PUSH_SOURCES`.
Keep old `publish-resultset-runnable-job-action` task name by creating a
method that points to `publish_push_runnable_job_action`.
Process output lines start with a prefix like GECKO(1234) or PID 1234
and then a pipe. This is an extra pipe symbol compared to other lines,
so our code that splits on | and assumes specific data in specific
positions is broken. Try to detect the process output case and discard
the first token so that we have the same fields as other
pipe-delimited data.
Fixes:
tests/autoclassify/test_classify_failures.py:7:1: F401 'treeherder.model.models.TextLogErrorMetadata' imported but unused
tests/etl/test_job_loader.py:7:1: F401 'treeherder.model.models.Repository' imported but unused
tests/model/test_classified_failure.py:6:1: F401 'treeherder.model.models.FailureLine' imported but unused
tests/seta/conftest.py:2:1: F401 'django.utils.timezone' imported but unused
tests/seta/test_job_priorities.py:8:1: F401 'treeherder.seta.settings.SETA_LOW_VALUE_PRIORITY' imported but unused
tests/webapp/api/test_text_log_summary_lines.py:4:1: F401 'treeherder.model.models.TextLogError' imported but unused
treeherder/auth/backends.py:13:5: F401 'django.utils.encoding.smart_str as smart_bytes' imported but unused
treeherder/autoclassify/tasks.py:4:1: F401 'django.conf.settings' imported but unused
treeherder/autoclassify/tasks.py:6:1: F401 'treeherder.celery_app' imported but unused
treeherder/perfalert/__init__.py:1:1: F401 '.perfalert.*' imported but unused
treeherder/seta/analyze_failures.py:7:1: F401 'treeherder.etl.seta.valid_platform' imported but unused
treeherder/seta/job_priorities.py:10:1: F401 'treeherder.model.models.Repository' imported but unused
treeherder/seta/models.py:7:1: F401 'treeherder.model.models.Repository' imported but unused
The seta migrations file change is due to the seta models no longer
depending on `model` (since the unnecessary `Repository` import has
been removed).
In most cases, but not all, we want to expire performance datum on the
same cadence as job data. Add some code to do this, with an optional
override to keep it around indefinitely for some cases.
tests/client/test_treeherder_client.py:514:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:66:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:94:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:112:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:148:1: E305 expected 2 blank lines after class or function definition, found 1
tests/model/test_error_summary.py:166:1: E305 expected 2 blank lines after class or function definition, found 1
tests/perfalert/test_analyze.py:94:1: E305 expected 2 blank lines after class or function definition, found 1
tests/webapp/api/test_auth.py:23:1: E305 expected 2 blank lines after class or function definition, found 1
tests/webapp/api/test_version.py:12:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/config/settings.py:21:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/credentials/admin.py:10:1: E305 expected 2 blank lines after class or function definition, found 0
treeherder/etl/schema.py:16:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/model/search.py:56:1: E305 expected 2 blank lines after class or function definition, found 1
treeherder/model/tasks.py:42:1: E305 expected 2 blank lines after class or function definition, found 1
Cameron Dawson
ionutgoldan
Sebastian Hengst
Ed Morley
George Hickman
George Hickman
Sarah Clements
Alisha Aneja
Max Chehab
KWierso
Ed Morley
James Graham
William Lachance
William Lachance