зеркало из https://github.com/mozilla/treeherder.git
Bug 1720181 - migration to remove SETA table (#7371)
This commit is contained in:
Sebastian Hengst
GitHub
Родитель
3064c54faa
Коммит
5541f65cef
|
|
@ -54,7 +54,7 @@ elif [ "$1" == "worker_log_parser_fail_json_unsheriffed" ]; then
|
|||
# Tasks that don't need a dedicated worker.
|
||||
elif [ "$1" == "worker_misc" ]; then
|
||||
export REMAP_SIGTERM=SIGQUIT
|
||||
exec newrelic-admin run-program celery worker -A treeherder --without-gossip --without-mingle --without-heartbeat -Q default,generate_perf_alerts,pushlog,seta_analyze_failures --concurrency=3
|
||||
exec newrelic-admin run-program celery worker -A treeherder --without-gossip --without-mingle --without-heartbeat -Q default,generate_perf_alerts,pushlog --concurrency=3
|
||||
|
||||
# Cron jobs
|
||||
elif [ "$1" == "run_intermittents_commenter" ]; then
|
||||
|
|
|
|||
85
docs/seta.md
85
docs/seta.md
|
|
@ -1,85 +0,0 @@
|
|||
# SETA
|
||||
|
||||
SETA finds the minimum set of jobs to run in order to catch all failures that our automation has found in the recent past on Firefox development repositories.
|
||||
There's one main API that SETA offers consumers (e.g. the Gecko decision task) in order to show which jobs are consider low value
|
||||
(less likely to catch a regression). After a certain number of calls, the API will return all jobs that need to be run.
|
||||
|
||||
SETA creates job priorities for all jobs found in the runnable-jobs API for that repository.
|
||||
Initially all jobs will be treated as low value, however, once we run the test to analyze past
|
||||
failures we will mark certain jobs as high value (priority=1).
|
||||
|
||||
Also note that jobs from different platforms (linux64 vs win7) or different CI systems (Buildbot vs TaskCluster)
|
||||
will be treated the same (use the same job priority). In other words, a job priority can represent a multiple
|
||||
number of jobs.
|
||||
|
||||
Jobs that appear on Treeherder for the first time will be treated as a job with high priority for a couple of
|
||||
weeks since we don't have historical data to determine how likely they're to catch a code regression.
|
||||
|
||||
In order to find open bugs for SETA visit list of [SETA bugs].
|
||||
|
||||
[seta bugs]: https://bugzilla.mozilla.org/buglist.cgi?product=Tree%20Management&component=Treeherder%3A%20SETA&resolution=---
|
||||
|
||||
## API
|
||||
|
||||
- `/api/project/{project}/seta/job-priorities/`
|
||||
- This is the API that consumers like the Gecko decision task will use
|
||||
- Currently only available for `autoland` and `try`
|
||||
|
||||
## Local set up
|
||||
|
||||
- Follow the steps at [Starting a local Treeherder instance].
|
||||
- Basically `docker-compose up`. This will initialize SETA's data
|
||||
|
||||
- Try out the various APIs:
|
||||
- <http://localhost:8000/api/project/autoland/seta/job-priorities/>
|
||||
|
||||
[starting a local treeherder instance]: installation.md#starting-a-local-treeherder-instance
|
||||
|
||||
## Local development
|
||||
|
||||
If you have ingested invalid `preseed.json` data you can clear like this:
|
||||
|
||||
```bash
|
||||
./manage.py initialize_seta --clear-job-priority-table
|
||||
```
|
||||
|
||||
If you want to validate `preseed.json` you can do so like this:
|
||||
|
||||
```bash
|
||||
./manage.py load_preseed --validate
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
Sometimes the default behaviour of SETA is not adequate (e.g. new jobs noticed get a 2 week expiration date & a high priority) when adding new platforms (e.g. stylo).
|
||||
Instead of investing more on accommodating for various scenarios we’ve decided to document how to make changes in the DB when we have to.
|
||||
|
||||
If you want to inspect the priorities for various jobs and platforms you can query the JobPriority table from reDash.
|
||||
Use this query as a starting point:
|
||||
|
||||
<https://sql.telemetry.mozilla.org/queries/14771/source#table>
|
||||
|
||||
### Steps for adjusting jobs
|
||||
|
||||
Sometimes, before you can adjust priorities of the jobs, you need to make sure they make it into the JobPriority table.
|
||||
In order to do so we need to update the job priority table from the shell. You will need cloudOps to do this for you:
|
||||
|
||||
Open the Python shell using `./manage.py shell`, then enter:
|
||||
|
||||
```python
|
||||
from treeherder.seta.update_job_priority import update_job_priority_table
|
||||
update_job_priority_table()
|
||||
```
|
||||
|
||||
If you want to remove the 2 week grace period and make the job low priority (priority=5) do something similar to this:
|
||||
|
||||
```python
|
||||
from treeherder.seta.models import JobPriority;
|
||||
# Inspect the jobs you want to change
|
||||
# Change the values appropriately
|
||||
JobPriority.objects.filter(platform="windows7-32-stylo", priority=1)
|
||||
JobPriority.objects.filter(platform="windows7-32-stylo", expiration_date__isnull=False)
|
||||
# Once satisfied
|
||||
JobPriority.objects.filter(platform="windows7-32-stylo", priority=1).update(priority=5);
|
||||
JobPriority.objects.filter(platform="windows7-32-stylo", expiration_date__isnull=False).update(expiration_date=None)
|
||||
```
|
||||
|
|
@ -10,8 +10,6 @@ if [ "${DATABASE_URL}" == "mysql://root@mysql/treeherder" ] ||
|
|||
echo '-----> Running Django migrations and loading reference data'
|
||||
./manage.py migrate --noinput
|
||||
./manage.py load_initial_data
|
||||
echo '-----> Initialize SETA'
|
||||
./manage.py initialize_seta
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
|
|
@ -47,5 +47,4 @@ nav:
|
|||
- Accessing data: 'accessing_data.md'
|
||||
- Data retention: 'data_cycling.md'
|
||||
- Submitting data: 'submitting_data.md'
|
||||
- SETA: 'seta.md'
|
||||
- Manual test cases: 'testcases.md'
|
||||
|
|
|
|||
|
|
@ -679,7 +679,7 @@
|
|||
},
|
||||
{
|
||||
"context": [],
|
||||
"description": "Run tests in the selected push that were optimized away, usually by SETA.\nThis action is for use on pushes that will be merged into another branch,to check that optimization hasn't hidden any failures.",
|
||||
"description": "Run tests in the selected push that were optimized away.\nThis action is for use on pushes that will be merged into another branch, to check that optimization hasn't hidden any failures.",
|
||||
"extra": {
|
||||
"actionPerm": "generic"
|
||||
},
|
||||
|
|
@ -689,7 +689,7 @@
|
|||
"decision": {
|
||||
"action": {
|
||||
"cb_name": "run-missing-tests",
|
||||
"description": "Run tests in the selected push that were optimized away, usually by SETA.\nThis action is for use on pushes that will be merged into another branch,to check that optimization hasn't hidden any failures.",
|
||||
"description": "Run tests in the selected push that were optimized away.\nThis action is for use on pushes that will be merged into another branch, to check that optimization hasn't hidden any failures.",
|
||||
"name": "run-missing-tests",
|
||||
"symbol": "rmt",
|
||||
"taskGroupId": "f7Jj_h6MTEKr5Ln_7aFqbw",
|
||||
|
|
|
|||
|
|
@ -1,206 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from treeherder.model.models import Job, JobNote
|
||||
from treeherder.seta.common import job_priority_index
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import SETA_HIGH_VALUE_PRIORITY, SETA_LOW_VALUE_PRIORITY
|
||||
from treeherder.seta.update_job_priority import _sanitize_data
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def runnable_jobs_data():
|
||||
repository_name = 'test_treeherder_jobs'
|
||||
runnable_jobs = [
|
||||
{
|
||||
"build_system_type": "buildbot",
|
||||
"job_type_name": "W3C Web Platform Tests",
|
||||
"platform": "windows8-64",
|
||||
"platform_option": "debug",
|
||||
"ref_data_name": "Windows 8 64-bit {} debug test web-platform-tests-1".format(
|
||||
repository_name
|
||||
),
|
||||
},
|
||||
{
|
||||
"build_system_type": "buildbot",
|
||||
"job_type_name": "Reftest e10s",
|
||||
"platform": "linux32",
|
||||
"platform_option": "opt",
|
||||
"ref_data_name": "Ubuntu VM 12.04 {} opt test reftest-e10s-1".format(repository_name),
|
||||
},
|
||||
{
|
||||
"build_system_type": "buildbot",
|
||||
"job_type_name": "Build",
|
||||
"platform": "osx-10-7",
|
||||
"platform_option": "opt",
|
||||
"ref_data_name": "OS X 10.7 {} build".format(repository_name),
|
||||
},
|
||||
{
|
||||
"build_system_type": "taskcluster",
|
||||
"job_type_name": "test-linux32/opt-reftest-e10s-1",
|
||||
"platform": "linux32",
|
||||
"platform_option": "opt",
|
||||
"ref_data_name": "test-linux32/opt-reftest-e10s-1",
|
||||
},
|
||||
{
|
||||
"build_system_type": "taskcluster",
|
||||
"job_type_name": "test-linux64/opt-reftest-e10s-2",
|
||||
"platform": "linux64",
|
||||
"platform_option": "opt",
|
||||
"ref_data_name": "test-linux64/opt-reftest-e10s-2",
|
||||
},
|
||||
]
|
||||
|
||||
return runnable_jobs
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tc_latest_gecko_decision_index(test_repository):
|
||||
return {
|
||||
"namespace": "gecko.v2.{}.latest.taskgraph.decision".format(test_repository),
|
||||
"taskId": "XVDNiP07RNaaEghhvkZJWg",
|
||||
"rank": 0,
|
||||
"data": {},
|
||||
"expires": "2018-01-04T20:36:11.375Z",
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sanitized_data(runnable_jobs_data):
|
||||
return _sanitize_data(runnable_jobs_data)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def all_job_priorities_stored(job_priority_list):
|
||||
"""Stores sample job priorities
|
||||
|
||||
If you include this fixture in your tests it will guarantee
|
||||
to insert job priority data into the temporary database.
|
||||
"""
|
||||
for jp in job_priority_list:
|
||||
jp.save()
|
||||
|
||||
return job_priority_list
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def job_priority_list(sanitized_data):
|
||||
jp_list = []
|
||||
for datum in sanitized_data:
|
||||
jp_list.append(
|
||||
JobPriority(
|
||||
testtype=datum['testtype'],
|
||||
buildtype=datum['platform_option'],
|
||||
platform=datum['platform'],
|
||||
buildsystem=datum['build_system_type'],
|
||||
priority=SETA_HIGH_VALUE_PRIORITY,
|
||||
)
|
||||
)
|
||||
# Mark the reftest-e10s-2 TC job as low priority (unique to TC)
|
||||
if datum['testtype'] == 'reftest-e10s-2':
|
||||
jp_list[-1].priority = SETA_LOW_VALUE_PRIORITY
|
||||
# Mark the web-platform-tests-1 BB job as low priority (unique to BB)
|
||||
if datum['testtype'] == 'web-platform-tests-1':
|
||||
jp_list[-1].priority = SETA_LOW_VALUE_PRIORITY
|
||||
|
||||
return jp_list
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jp_index_fixture(job_priority_list):
|
||||
return job_priority_index(job_priority_list)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fifteen_jobs_with_notes(
|
||||
eleven_jobs_stored, taskcluster_jobs_stored, test_user, failure_classifications
|
||||
):
|
||||
"""provide 15 jobs with job notes."""
|
||||
counter = 0
|
||||
for job in Job.objects.all():
|
||||
counter += 1
|
||||
|
||||
# add 5 valid job notes related to 'this is revision x'
|
||||
if counter < 6:
|
||||
JobNote.objects.create(
|
||||
job=job, failure_classification_id=2, user=test_user, text="this is revision x"
|
||||
)
|
||||
continue
|
||||
|
||||
# add 3 valid job notes with raw revision 31415926535
|
||||
if counter < 9:
|
||||
JobNote.objects.create(
|
||||
job=job, failure_classification_id=2, user=test_user, text="314159265358"
|
||||
)
|
||||
continue
|
||||
|
||||
# Add 3 job notes with full url to revision, expected to map to 31415926535
|
||||
if counter < 12:
|
||||
JobNote.objects.create(
|
||||
job=job,
|
||||
failure_classification_id=2,
|
||||
user=test_user,
|
||||
text="http://hg.mozilla.org/mozilla-central/314159265358",
|
||||
)
|
||||
continue
|
||||
|
||||
# Add 1 valid job with trailing slash, expected to map to 31415926535
|
||||
if counter < 13:
|
||||
JobNote.objects.create(
|
||||
job=job,
|
||||
failure_classification_id=2,
|
||||
user=test_user,
|
||||
text="http://hg.mozilla.org/mozilla-central/314159265358/",
|
||||
)
|
||||
continue
|
||||
|
||||
# Add 1 job with invalid revision text, expect it to be ignored
|
||||
if counter < 14:
|
||||
# We will ignore this based on text length
|
||||
JobNote.objects.create(
|
||||
job=job, failure_classification_id=2, user=test_user, text="too short"
|
||||
)
|
||||
continue
|
||||
|
||||
# Add 1 job with no revision text, expect it to be ignored
|
||||
if counter < 15:
|
||||
# We will ignore this based on blank note
|
||||
JobNote.objects.create(job=job, failure_classification_id=2, user=test_user, text="")
|
||||
continue
|
||||
|
||||
# Add 1 more job with invalid revision text, expect it to be ignored
|
||||
if counter < 16:
|
||||
# We will ignore this based on effectively blank note
|
||||
JobNote.objects.create(job=job, failure_classification_id=2, user=test_user, text="/")
|
||||
continue
|
||||
|
||||
# if we have any more jobs defined it will break this test, ignore
|
||||
continue
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def failures_fixed_by_commit():
|
||||
# We expect 'windows8-32', 'windowsxp' and 'osx-10-7' jobs to be excluded because those
|
||||
# platforms are currently unsupported SETA platforms (see treeherder/seta/settings.py)
|
||||
# Also we expect any jobs with invalid job notes to be excluded
|
||||
return {
|
||||
u'this is revision x': [
|
||||
(u'b2g_mozilla-release_emulator-jb-debug_dep', u'debug', u'b2g-emu-jb'),
|
||||
(u'b2g_mozilla-release_emulator-jb_dep', u'opt', u'b2g-emu-jb'),
|
||||
(u'mochitest-browser-chrome', u'debug', u'osx-10-6'),
|
||||
(u'mochitest-browser-chrome', u'debug', u'windows7-32'),
|
||||
],
|
||||
u'314159265358': [
|
||||
(u'b2g_mozilla-release_emulator-jb-debug_dep', u'debug', u'b2g-emu-jb'),
|
||||
(u'b2g_mozilla-release_emulator-debug_dep', u'debug', u'b2g-emu-ics'),
|
||||
(u'b2g_mozilla-release_inari_dep', u'opt', u'b2g-device-image'),
|
||||
(u'b2g_mozilla-release_nexus-4_dep', u'opt', u'b2g-device-image'),
|
||||
(u'mochitest-devtools-chrome-3', u'debug', u'linux64'),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def patched_seta_fixed_by_commit_repos(monkeypatch, test_repository):
|
||||
patched = [test_repository.name]
|
||||
monkeypatch.setattr("treeherder.seta.analyze_failures.SETA_FIXED_BY_COMMIT_REPOS", patched)
|
||||
return patched
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from treeherder.seta.analyze_failures import get_failures_fixed_by_commit
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_analyze_failures(
|
||||
fifteen_jobs_with_notes, failures_fixed_by_commit, patched_seta_fixed_by_commit_repos
|
||||
):
|
||||
ret = get_failures_fixed_by_commit()
|
||||
exp = failures_fixed_by_commit
|
||||
|
||||
assert sorted(ret.keys()) == sorted(exp.keys())
|
||||
|
||||
for key in exp:
|
||||
assert sorted(ret[key]) == sorted(exp[key])
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from treeherder.seta.high_value_jobs import get_high_value_jobs
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_get_high_value_jobs(fifteen_jobs_with_notes, failures_fixed_by_commit):
|
||||
get_high_value_jobs(failures_fixed_by_commit)
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
import datetime
|
||||
|
||||
import pytest
|
||||
from mock import patch
|
||||
|
||||
from treeherder.seta.job_priorities import SetaError, seta_job_scheduling
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
@patch('treeherder.seta.job_priorities.SETAJobPriorities._validate_request', return_value=None)
|
||||
@patch('treeherder.etl.seta.list_runnable_jobs')
|
||||
def test_gecko_decision_task(
|
||||
runnable_jobs_list,
|
||||
validate_request,
|
||||
test_repository,
|
||||
runnable_jobs_data,
|
||||
all_job_priorities_stored,
|
||||
):
|
||||
"""
|
||||
When the Gecko decision task calls SETA it will return all jobs that are less likely to catch
|
||||
a regression (low value jobs).
|
||||
"""
|
||||
runnable_jobs_list.return_value = runnable_jobs_data
|
||||
jobs = seta_job_scheduling(project=test_repository.name, build_system_type='taskcluster')
|
||||
assert len(jobs['jobtypes'][str(datetime.date.today())]) == 1
|
||||
|
||||
|
||||
def test_gecko_decision_task_invalid_repo():
|
||||
"""
|
||||
When the Gecko decision task calls SETA it will return all jobs that are less likely to catch
|
||||
a regression (low value jobs).
|
||||
"""
|
||||
with pytest.raises(SetaError) as exception_info:
|
||||
seta_job_scheduling(project='mozilla-repo-x', build_system_type='taskcluster')
|
||||
|
||||
assert (
|
||||
str(exception_info.value) == "The specified project repo 'mozilla-repo-x' "
|
||||
"is not supported by SETA."
|
||||
)
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
import datetime
|
||||
|
||||
import pytest
|
||||
from django.db.utils import IntegrityError
|
||||
from django.utils import timezone
|
||||
|
||||
from treeherder.seta.models import JobPriority
|
||||
|
||||
TOMORROW = timezone.now() + datetime.timedelta(days=1)
|
||||
YESTERDAY = timezone.now() - datetime.timedelta(days=1)
|
||||
|
||||
|
||||
# JobPriority tests
|
||||
def test_expired_job_priority():
|
||||
jp = JobPriority(
|
||||
testtype='web-platform-tests-1',
|
||||
buildtype='opt',
|
||||
platform='windows8-64',
|
||||
priority=1,
|
||||
expiration_date=YESTERDAY,
|
||||
buildsystem='taskcluster',
|
||||
)
|
||||
assert jp.has_expired()
|
||||
|
||||
|
||||
def test_not_expired_job_priority():
|
||||
jp = JobPriority(
|
||||
testtype='web-platform-tests-1',
|
||||
buildtype='opt',
|
||||
platform='windows8-64',
|
||||
priority=1,
|
||||
expiration_date=TOMORROW,
|
||||
buildsystem='taskcluster',
|
||||
)
|
||||
assert not jp.has_expired()
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_null_testtype():
|
||||
'''The expiration date accepts null values'''
|
||||
with pytest.raises(IntegrityError):
|
||||
JobPriority.objects.create(
|
||||
testtype=None,
|
||||
buildtype='opt',
|
||||
platform='windows8-64',
|
||||
priority=1,
|
||||
expiration_date=TOMORROW,
|
||||
buildsystem='taskcluster',
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_null_expiration_date():
|
||||
'''The expiration date accepts null values'''
|
||||
jp = JobPriority.objects.create(
|
||||
testtype='web-platform-tests-2',
|
||||
buildtype='opt',
|
||||
platform='windows8-64',
|
||||
priority=1,
|
||||
expiration_date=None,
|
||||
buildsystem='taskcluster',
|
||||
)
|
||||
assert jp.expiration_date is None
|
||||
|
|
@ -1,98 +0,0 @@
|
|||
import pytest
|
||||
from mock import patch
|
||||
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.update_job_priority import (
|
||||
_initialize_values,
|
||||
_sanitize_data,
|
||||
_unique_key,
|
||||
_update_table,
|
||||
query_sanitized_data,
|
||||
)
|
||||
|
||||
|
||||
def test_unique_key():
|
||||
new_job = {
|
||||
'build_system_type': 'buildbot',
|
||||
'platform': 'windows8-64',
|
||||
'platform_option': 'opt',
|
||||
'testtype': 'web-platform-tests-1',
|
||||
}
|
||||
assert _unique_key(new_job), ('web-platform-tests-1', 'opt', 'windows8-64')
|
||||
|
||||
|
||||
def test_sanitize_data(runnable_jobs_data):
|
||||
data = _sanitize_data(runnable_jobs_data)
|
||||
bb_jobs = 0
|
||||
tc_jobs = 0
|
||||
for datum in data:
|
||||
if datum['build_system_type'] in ('taskcluster', '*'):
|
||||
tc_jobs += 1
|
||||
if datum['build_system_type'] in ('buildbot', '*'):
|
||||
bb_jobs += 1
|
||||
|
||||
assert bb_jobs == 2
|
||||
assert tc_jobs == 2
|
||||
|
||||
|
||||
@patch('treeherder.seta.update_job_priority.list_runnable_jobs')
|
||||
def test_query_sanitized_data(list_runnable_jobs, runnable_jobs_data, sanitized_data):
|
||||
list_runnable_jobs.return_value = runnable_jobs_data
|
||||
data = query_sanitized_data()
|
||||
assert data == sanitized_data
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_initialize_values_no_data():
|
||||
results = _initialize_values()
|
||||
assert results == ({}, 5, None)
|
||||
|
||||
|
||||
@patch.object(JobPriority, 'save')
|
||||
@patch('treeherder.seta.update_job_priority._initialize_values')
|
||||
def test_update_table_empty_table(initial_values, jp_save, sanitized_data):
|
||||
"""
|
||||
We test that starting from an empty table
|
||||
"""
|
||||
# This set of values is when we're bootstrapping the service (aka empty table)
|
||||
initial_values.return_value = {}, 5, None
|
||||
jp_save.return_value = None # Since we don't want to write to the DB
|
||||
assert _update_table(sanitized_data) == (3, 0, 0)
|
||||
|
||||
|
||||
@pytest.mark.django_db()
|
||||
def test_update_table_job_from_other_buildsysten(all_job_priorities_stored):
|
||||
# We already have a TaskCluster job like this in the DB
|
||||
# The DB entry should be changed to '*'
|
||||
data = {
|
||||
'build_system_type': 'buildbot',
|
||||
'platform': 'linux64',
|
||||
'platform_option': 'opt',
|
||||
'testtype': 'reftest-e10s-2',
|
||||
}
|
||||
# Before calling update_table the priority is only for TaskCluster
|
||||
assert (
|
||||
len(
|
||||
JobPriority.objects.filter(
|
||||
buildsystem='taskcluster',
|
||||
buildtype=data['platform_option'],
|
||||
platform=data['platform'],
|
||||
testtype=data['testtype'],
|
||||
)
|
||||
)
|
||||
== 1
|
||||
)
|
||||
# We are checking that only 1 job was updated
|
||||
ret_val = _update_table([data])
|
||||
assert ret_val == (0, 0, 1)
|
||||
assert (
|
||||
len(
|
||||
JobPriority.objects.filter(
|
||||
buildsystem='*',
|
||||
buildtype=data['platform_option'],
|
||||
platform=data['platform'],
|
||||
testtype=data['testtype'],
|
||||
)
|
||||
)
|
||||
== 1
|
||||
)
|
||||
|
|
@ -77,7 +77,6 @@ INSTALLED_APPS = [
|
|||
'treeherder.log_parser',
|
||||
'treeherder.etl',
|
||||
'treeherder.perf',
|
||||
'treeherder.seta',
|
||||
'treeherder.intermittents_commenter',
|
||||
'treeherder.changelog',
|
||||
]
|
||||
|
|
@ -320,7 +319,6 @@ CELERY_TASK_QUEUES = [
|
|||
Queue('generate_perf_alerts', Exchange('default'), routing_key='generate_perf_alerts'),
|
||||
Queue('store_pulse_tasks', Exchange('default'), routing_key='store_pulse_tasks'),
|
||||
Queue('store_pulse_pushes', Exchange('default'), routing_key='store_pulse_pushes'),
|
||||
Queue('seta_analyze_failures', Exchange('default'), routing_key='seta_analyze_failures'),
|
||||
]
|
||||
|
||||
# Force all queues to be explicitly listed in `CELERY_TASK_QUEUES` to help prevent typos
|
||||
|
|
@ -365,12 +363,6 @@ CELERY_BEAT_SCHEDULE = {
|
|||
'relative': True,
|
||||
'options': {"queue": "pushlog"},
|
||||
},
|
||||
'seta-analyze-failures': {
|
||||
'task': 'seta-analyze-failures',
|
||||
'schedule': timedelta(days=1),
|
||||
'relative': True,
|
||||
'options': {'queue': "seta_analyze_failures"},
|
||||
},
|
||||
}
|
||||
|
||||
# CORS Headers
|
||||
|
|
|
|||
|
|
@ -1,120 +0,0 @@
|
|||
import logging
|
||||
|
||||
from django.core.cache import cache
|
||||
|
||||
from treeherder.etl.runnable_jobs import list_runnable_jobs
|
||||
from treeherder.seta.common import convert_job_type_name_to_testtype, unique_key
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import (
|
||||
SETA_REF_DATA_NAMES_CACHE_TIMEOUT,
|
||||
SETA_SUPPORTED_TC_JOBTYPES,
|
||||
SETA_UNSUPPORTED_PLATFORMS,
|
||||
SETA_UNSUPPORTED_TESTTYPES,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_job_blacklisted(testtype):
|
||||
if not testtype:
|
||||
return True
|
||||
return testtype in SETA_UNSUPPORTED_TESTTYPES
|
||||
|
||||
|
||||
def parse_testtype(build_system_type, job_type_name, platform_option, ref_data_name):
|
||||
"""
|
||||
Buildbot Taskcluster
|
||||
----------- -----------
|
||||
build_system_type buildbot taskcluster
|
||||
job_type_name Mochitest task label
|
||||
platform_option debug,opt,pgo debug,opt,pgo
|
||||
ref_data_name buildername task label OR signature hash
|
||||
"""
|
||||
# XXX: Figure out how to ignore build, lint, etc. jobs
|
||||
# https://bugzilla.mozilla.org/show_bug.cgi?id=1318659
|
||||
testtype = None
|
||||
if build_system_type == 'buildbot':
|
||||
# The testtype of builbot job can been found in 'ref_data_name'
|
||||
# like web-platform-tests-4 in "Ubuntu VM 12.04 x64 mozilla-inbound
|
||||
# opt test web-platform-tests-4"
|
||||
testtype = ref_data_name.split(' ')[-1]
|
||||
else:
|
||||
if job_type_name.startswith(tuple(SETA_SUPPORTED_TC_JOBTYPES)):
|
||||
# we should get "jittest-3" as testtype for a job_type_name like
|
||||
# test-linux64/debug-jittest-3
|
||||
testtype = convert_job_type_name_to_testtype(job_type_name)
|
||||
return testtype
|
||||
|
||||
|
||||
def valid_platform(platform):
|
||||
# We only care about in-tree scheduled tests and ignore out of band system like autophone.
|
||||
return platform not in SETA_UNSUPPORTED_PLATFORMS
|
||||
|
||||
|
||||
def job_priorities_to_jobtypes():
|
||||
jobtypes = []
|
||||
for jp in JobPriority.objects.all():
|
||||
jobtypes.append(jp.unique_identifier())
|
||||
|
||||
return jobtypes
|
||||
|
||||
|
||||
# The only difference between projects is that their list will be based
|
||||
# on their own specific runnable_jobs.json artifact
|
||||
def get_reference_data_names(project="autoland", build_system="taskcluster"):
|
||||
"""
|
||||
We want all reference data names for every task that runs on a specific project.
|
||||
|
||||
For example: "test-linux64/opt-mochitest-webgl-e10s-1"
|
||||
"""
|
||||
# we cache the reference data names in order to reduce API calls
|
||||
cache_key = '{}-{}-ref_data_names_cache'.format(project, build_system)
|
||||
ref_data_names_map = cache.get(cache_key)
|
||||
if ref_data_names_map:
|
||||
return ref_data_names_map
|
||||
|
||||
logger.debug("We did not hit the cache.")
|
||||
# cache expired so re-build the reference data names map; the map
|
||||
# contains the ref_data_name of every Treeherder task for this project
|
||||
ignored_jobs = []
|
||||
ref_data_names = {}
|
||||
|
||||
runnable_jobs = list_runnable_jobs(project)
|
||||
|
||||
for job in runnable_jobs:
|
||||
# get testtype e.g. web-platform-tests-4
|
||||
testtype = parse_testtype(
|
||||
build_system_type=job['build_system_type'],
|
||||
job_type_name=job['job_type_name'],
|
||||
platform_option=job['platform_option'],
|
||||
ref_data_name=job['ref_data_name'],
|
||||
)
|
||||
|
||||
if not valid_platform(job['platform']):
|
||||
continue
|
||||
|
||||
if is_job_blacklisted(testtype):
|
||||
ignored_jobs.append(job['ref_data_name'])
|
||||
if testtype:
|
||||
logger.debug(
|
||||
'get_reference_data_names: blacklisted testtype {} for job {}'.format(
|
||||
testtype, job
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
key = unique_key(
|
||||
testtype=testtype, buildtype=job['platform_option'], platform=job['platform']
|
||||
)
|
||||
|
||||
if build_system == '*':
|
||||
ref_data_names[key] = job['ref_data_name']
|
||||
elif job['build_system_type'] == build_system:
|
||||
ref_data_names[key] = job['ref_data_name']
|
||||
|
||||
logger.debug('Ignoring %s', ', '.join(sorted(ignored_jobs)))
|
||||
|
||||
# update the cache
|
||||
cache.set(cache_key, ref_data_names_map, SETA_REF_DATA_NAMES_CACHE_TIMEOUT)
|
||||
|
||||
return ref_data_names
|
||||
|
|
@ -16,4 +16,3 @@ class Command(BaseCommand):
|
|||
'performance_bug_templates',
|
||||
'performance_tag',
|
||||
)
|
||||
call_command('load_preseed')
|
||||
|
|
|
|||
|
|
@ -1,149 +0,0 @@
|
|||
import logging
|
||||
from collections import defaultdict
|
||||
from datetime import timedelta
|
||||
|
||||
from django.utils import timezone
|
||||
|
||||
from treeherder.etl.seta import is_job_blacklisted, parse_testtype
|
||||
from treeherder.model import models
|
||||
from treeherder.seta.common import unique_key
|
||||
from treeherder.seta.high_value_jobs import get_high_value_jobs
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import (
|
||||
SETA_FIXED_BY_COMMIT_DAYS,
|
||||
SETA_FIXED_BY_COMMIT_REPOS,
|
||||
SETA_SUPPORTED_TC_JOBTYPES,
|
||||
SETA_UNSUPPORTED_PLATFORMS,
|
||||
)
|
||||
from treeherder.seta.update_job_priority import update_job_priority_table
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AnalyzeFailures:
|
||||
def __init__(self, **options):
|
||||
self.dry_run = options.get('dry_run', False)
|
||||
|
||||
def run(self):
|
||||
fixed_by_commit_jobs = get_failures_fixed_by_commit()
|
||||
if fixed_by_commit_jobs:
|
||||
# We need to update the job priority table before we can call get_high_value_jobs()
|
||||
update_job_priority_table()
|
||||
high_value_jobs = get_high_value_jobs(fixed_by_commit_jobs)
|
||||
|
||||
if not self.dry_run:
|
||||
logger.warning("Let's see if we need to increase the priority of any job")
|
||||
JobPriority.objects.clear_expiration_field_for_expired_jobs()
|
||||
JobPriority.objects.adjust_jobs_priority(high_value_jobs)
|
||||
|
||||
|
||||
def get_failures_fixed_by_commit():
|
||||
"""Return all job failures annotated with "fixed by commit" grouped by reason given for annotation.
|
||||
|
||||
It returns a dictionary with a revision or bug ID as the key (bug ID is used for
|
||||
intermittent failures and the revision is used for real failures). For SETA's purposes
|
||||
we only care about revisions (real failures).
|
||||
The failures for *real failures* will contain all jobs that have been starred as "fixed by commit".
|
||||
|
||||
Notice that the data does not tell you on which repository a root failure was fixed.
|
||||
|
||||
For instance, in the raw data you might see a reference to 9fa614d8310d which is a back out
|
||||
and it is referenced by 12 starred jobs:
|
||||
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=android%20debug%20cpp&tochange=9fa614d8310db9aabe85cc3c3cff6281fe1edb0c
|
||||
The raw data will show those 12 jobs.
|
||||
|
||||
The returned data will look like this:
|
||||
{
|
||||
"44d29bac3654": [
|
||||
["android-4-0-armv7-api15", "opt", "android-lint"],
|
||||
["android-4-0-armv7-api15", "opt", "android-api-15-gradle-dependencies"],
|
||||
]
|
||||
}
|
||||
"""
|
||||
failures = defaultdict(list)
|
||||
option_collection_map = models.OptionCollection.objects.get_option_collection_map()
|
||||
|
||||
fixed_by_commit_data_set = (
|
||||
models.JobNote.objects.filter(
|
||||
failure_classification=2,
|
||||
created__gt=timezone.now() - timedelta(days=SETA_FIXED_BY_COMMIT_DAYS),
|
||||
text__isnull=False,
|
||||
job__repository__name__in=SETA_FIXED_BY_COMMIT_REPOS,
|
||||
)
|
||||
.exclude(job__signature__build_platform__in=SETA_UNSUPPORTED_PLATFORMS)
|
||||
.exclude(text="")
|
||||
.select_related('job', 'job__signature', 'job__job_type')
|
||||
)
|
||||
|
||||
# check if at least one fixed by commit job meets our requirements without populating queryset
|
||||
if not fixed_by_commit_data_set.exists():
|
||||
logger.warning("We couldn't find any fixed-by-commit jobs")
|
||||
return failures
|
||||
|
||||
# now process the fixed by commit jobs in batches using django's queryset iterator
|
||||
for job_note in fixed_by_commit_data_set.iterator():
|
||||
# if we have http://hg.mozilla.org/rev/<rev> and <rev>, we will only use <rev>
|
||||
revision_id = job_note.text.strip('/')
|
||||
revision_id = revision_id.split('/')[-1]
|
||||
|
||||
# This prevents the empty string case and ignores bug ids
|
||||
if not revision_id or len(revision_id) < 12:
|
||||
continue
|
||||
|
||||
# We currently don't guarantee that text is actually a revision
|
||||
# Even if not perfect the main idea is that a bunch of jobs were annotated with
|
||||
# a unique identifier. The assumption is that the text is unique
|
||||
#
|
||||
# I've seen these values being used:
|
||||
# * 12 char revision
|
||||
# * 40 char revision
|
||||
# * link to revision on hg
|
||||
# * revisionA & revisionB
|
||||
# * should be fixed by <revision>
|
||||
# * bug id
|
||||
#
|
||||
# Note that if some jobs are annotated with the 12char revision and others with the
|
||||
# 40char revision we will have two disjunct set of failures
|
||||
#
|
||||
# Some of this will be improved in https://bugzilla.mozilla.org/show_bug.cgi?id=1323536
|
||||
|
||||
try:
|
||||
# check if jobtype is supported by SETA (see treeherder/seta/settings.py)
|
||||
if job_note.job.signature.build_system_type != 'buildbot':
|
||||
if not job_note.job.job_type.name.startswith(tuple(SETA_SUPPORTED_TC_JOBTYPES)):
|
||||
continue
|
||||
|
||||
testtype = parse_testtype(
|
||||
build_system_type=job_note.job.signature.build_system_type, # e.g. taskcluster
|
||||
job_type_name=job_note.job.job_type.name, # e.g. Mochitest
|
||||
platform_option=job_note.job.get_platform_option(
|
||||
option_collection_map
|
||||
), # e.g. 'opt'
|
||||
ref_data_name=job_note.job.signature.name, # buildername or task label
|
||||
)
|
||||
|
||||
if testtype:
|
||||
if is_job_blacklisted(testtype):
|
||||
continue
|
||||
else:
|
||||
logger.warning(
|
||||
'We were unable to parse %s/%s',
|
||||
job_note.job.job_type.name,
|
||||
job_note.job.signature.name,
|
||||
)
|
||||
continue
|
||||
|
||||
# we now have a legit fixed-by-commit job failure
|
||||
failures[revision_id].append(
|
||||
unique_key(
|
||||
testtype=testtype,
|
||||
buildtype=job_note.job.get_platform_option(option_collection_map), # e.g. 'opt'
|
||||
platform=job_note.job.signature.build_platform,
|
||||
)
|
||||
)
|
||||
except models.Job.DoesNotExist:
|
||||
logger.warning('job_note %s has no job associated to it', job_note.id)
|
||||
continue
|
||||
|
||||
logger.warning("Number of fixed_by_commit revisions: %s", len(failures))
|
||||
return failures
|
||||
|
|
@ -1,79 +0,0 @@
|
|||
import logging
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DuplicateKeyError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def unique_key(testtype, buildtype, platform):
|
||||
'''This makes sure that we order consistently this unique identifier'''
|
||||
return (testtype, buildtype, platform)
|
||||
|
||||
|
||||
def job_priority_index(job_priorities):
|
||||
'''This structure helps with finding data from the job priorities table'''
|
||||
jp_index = {}
|
||||
# Creating this data structure which reduces how many times we iterate through the DB rows
|
||||
for jp in job_priorities:
|
||||
key = jp.unique_identifier()
|
||||
|
||||
# This is guaranteed by a unique composite index for these 3 fields in models.py
|
||||
if key in jp_index:
|
||||
msg = '"{}" should be a unique job priority and that is unexpected.'.format(key)
|
||||
raise DuplicateKeyError(msg)
|
||||
|
||||
# (testtype, buildtype, platform)
|
||||
jp_index[key] = {'pk': jp.id, 'build_system_type': jp.buildsystem}
|
||||
|
||||
return jp_index
|
||||
|
||||
|
||||
# The order of this is list is important as the more specific patterns
|
||||
# will be processed before the less specific ones. This must be kept up
|
||||
# to date with SETA_SUPPORTED_TC_JOBTYPES in settings.py.
|
||||
RE_JOB_TYPE_NAMES = [
|
||||
{'name': 'test', 'pattern': re.compile('test-[^/]+/[^-]+-(.*)$')},
|
||||
{'name': 'desktop-test', 'pattern': re.compile('desktop-test-[^/]+/[^-]+-(.*)$')},
|
||||
{'name': 'android-test', 'pattern': re.compile('android-test-[^/]+/[^-]+-(.*)$')},
|
||||
{'name': 'source-test', 'pattern': re.compile('(source-test-[^/]+)(?:/.*)?$')},
|
||||
{'name': 'build', 'pattern': re.compile('(build-[^/]+)/[^-]+$')},
|
||||
{'name': 'spidermonkey', 'pattern': re.compile('(spidermonkey-[^/]+)/[^-]+$')},
|
||||
{'name': 'iris', 'pattern': re.compile('(iris-[^/]+)/[^-]+$')},
|
||||
{'name': 'webrender', 'pattern': re.compile('(webrender-.*)-(?:opt|debug|pgo)$')},
|
||||
]
|
||||
|
||||
|
||||
def convert_job_type_name_to_testtype(job_type_name):
|
||||
"""job_type_names are essentially free form though there are
|
||||
several patterns used in job_type_names.
|
||||
|
||||
test-<platform>/<buildtype>-<testtype> test-linux1804-64-shippable-qr/opt-reftest-e10s-5
|
||||
build-<platform>/<buildtype> build-linux64-asan-fuzzing/opt
|
||||
<testtype>-<buildtype> webrender-android-hw-p2-debug
|
||||
|
||||
Prior to Bug 1608427, only non-build tasks were eligible for
|
||||
optimization using seta strategies and Treeherder's handling of
|
||||
possible task labels failed to properly account for the different
|
||||
job_type_names possible with build tasks. While investigating this
|
||||
failure to support build tasks, it was discovered that other test
|
||||
tasks did not match the expected job_type_name pattern. This
|
||||
function ensures that job_type_names are converted to seta
|
||||
testtypes in a consistent fashion.
|
||||
"""
|
||||
testtype = None
|
||||
if not job_type_name.startswith('[funsize'):
|
||||
for re_job_type_name in RE_JOB_TYPE_NAMES:
|
||||
m = re_job_type_name['pattern'].match(job_type_name)
|
||||
if m:
|
||||
testtype = m.group(1)
|
||||
break
|
||||
if not testtype:
|
||||
logger.warning(
|
||||
'convert_job_type_name_to_testtype("{}") not matched. '
|
||||
'Using job_type_name as is.'.format(job_type_name)
|
||||
)
|
||||
testtype = job_type_name
|
||||
return testtype
|
||||
|
|
@ -1,102 +0,0 @@
|
|||
import logging
|
||||
|
||||
from treeherder.etl.seta import job_priorities_to_jobtypes
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_matched(failure, removals):
|
||||
found = False
|
||||
if failure in removals:
|
||||
found = True
|
||||
|
||||
return found
|
||||
|
||||
|
||||
def check_removal(failures, removals):
|
||||
results = {}
|
||||
for failure in failures:
|
||||
results[failure] = []
|
||||
for failure_job in failures[failure]:
|
||||
found = is_matched(failure_job, removals)
|
||||
|
||||
# we will add the test to the resulting structure unless we find a match
|
||||
# in the jobtype we are trying to ignore.
|
||||
if not found:
|
||||
results[failure].append(failure_job)
|
||||
|
||||
if not results[failure]:
|
||||
del results[failure]
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def build_removals(active_jobs, failures, target):
|
||||
"""
|
||||
active_jobs - all possible desktop & android jobs on Treeherder (no PGO)
|
||||
failures - list of all failures
|
||||
target - percentage of failures we're going to process
|
||||
|
||||
Return list of jobs to remove and list of revisions that are regressed
|
||||
"""
|
||||
# Determine the number of failures we're going to process
|
||||
# A failure is a revision + all of the jobs that were fixed by it
|
||||
number_of_failures = int((target / 100) * len(failures))
|
||||
low_value_jobs = []
|
||||
|
||||
for jobtype in active_jobs:
|
||||
# Determine if removing an active job will reduce the number of failures we would catch
|
||||
# or stay the same
|
||||
remaining_failures = check_removal(failures, [jobtype])
|
||||
|
||||
if len(remaining_failures) >= number_of_failures:
|
||||
low_value_jobs.append(jobtype)
|
||||
failures = remaining_failures
|
||||
else:
|
||||
failed_revisions = []
|
||||
for revision in failures:
|
||||
if revision not in remaining_failures:
|
||||
failed_revisions.append(revision)
|
||||
|
||||
logger.info(
|
||||
"jobtype: %s is the root failure(s) of these %s revisions",
|
||||
jobtype,
|
||||
failed_revisions,
|
||||
)
|
||||
|
||||
return low_value_jobs
|
||||
|
||||
|
||||
def get_high_value_jobs(fixed_by_commit_jobs, target=100):
|
||||
"""
|
||||
fixed_by_commit_jobs:
|
||||
Revisions and jobs that have been starred that are fixed with a push or a bug
|
||||
target:
|
||||
Percentage of failures to analyze
|
||||
"""
|
||||
total = len(fixed_by_commit_jobs)
|
||||
logger.info("Processing %s revision(s)", total)
|
||||
active_jobs = job_priorities_to_jobtypes()
|
||||
|
||||
low_value_jobs = build_removals(
|
||||
active_jobs=active_jobs, failures=fixed_by_commit_jobs, target=target
|
||||
)
|
||||
|
||||
# Only return high value jobs
|
||||
for low_value_job in low_value_jobs:
|
||||
try:
|
||||
active_jobs.remove(low_value_job)
|
||||
except ValueError:
|
||||
logger.warning("%s is missing from the job list", low_value_job)
|
||||
|
||||
total = len(fixed_by_commit_jobs)
|
||||
total_detected = check_removal(fixed_by_commit_jobs, low_value_jobs)
|
||||
percent_detected = 100 * len(total_detected) / total
|
||||
logger.info(
|
||||
"We will detect %.2f%% (%s) of the %s failures",
|
||||
percent_detected,
|
||||
len(total_detected),
|
||||
total,
|
||||
)
|
||||
|
||||
return active_jobs
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
import datetime
|
||||
import logging
|
||||
|
||||
from treeherder.etl.seta import get_reference_data_names, is_job_blacklisted, valid_platform
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import SETA_LOW_VALUE_PRIORITY, SETA_PROJECTS, THE_FUTURE
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SetaError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SETAJobPriorities:
|
||||
"""
|
||||
SETA JobPriority Implementation
|
||||
"""
|
||||
|
||||
def _process(self, project, build_system, job_priorities):
|
||||
'''Return list of ref_data_name for job_priorities'''
|
||||
if not job_priorities:
|
||||
raise SetaError("Call docker-compose run backend ./manage.py initialize_seta")
|
||||
|
||||
jobs = []
|
||||
|
||||
ref_data_names_map = get_reference_data_names(project, build_system)
|
||||
|
||||
# now check the JobPriority table against the list of valid runnable
|
||||
for jp in job_priorities:
|
||||
# if this JobPriority entry is no longer supported in SETA then ignore it
|
||||
if not valid_platform(jp.platform):
|
||||
continue
|
||||
if is_job_blacklisted(jp.testtype):
|
||||
continue
|
||||
|
||||
key = jp.unique_identifier()
|
||||
if key in ref_data_names_map:
|
||||
# e.g. desktop-test-linux64-pgo/opt-reftest-13 or builder name
|
||||
jobs.append(ref_data_names_map[key])
|
||||
else:
|
||||
logger.warning(
|
||||
'Job priority key %s for (%s) not found in accepted jobs list', key, jp
|
||||
)
|
||||
|
||||
return jobs
|
||||
|
||||
def _query_job_priorities(self, priority, excluded_build_system_type):
|
||||
job_priorities = JobPriority.objects.all()
|
||||
if priority:
|
||||
job_priorities = job_priorities.filter(priority=priority)
|
||||
if excluded_build_system_type:
|
||||
job_priorities = job_priorities.exclude(buildsystem=excluded_build_system_type)
|
||||
return job_priorities
|
||||
|
||||
def _validate_request(self, build_system_type, project):
|
||||
if build_system_type not in ('buildbot', 'taskcluster', '*'):
|
||||
raise SetaError('Valid build_system_type values are buildbot or taskcluster.')
|
||||
if project not in SETA_PROJECTS:
|
||||
raise SetaError("The specified project repo '%s' is not supported by SETA." % project)
|
||||
|
||||
def seta_job_scheduling(self, project, build_system_type, priority=None):
|
||||
self._validate_request(build_system_type, project)
|
||||
if build_system_type == 'taskcluster':
|
||||
if priority is None:
|
||||
priority = SETA_LOW_VALUE_PRIORITY
|
||||
job_priorities = []
|
||||
for jp in self._query_job_priorities(
|
||||
priority=priority, excluded_build_system_type='buildbot'
|
||||
):
|
||||
if jp.has_expired() or jp.expiration_date == THE_FUTURE:
|
||||
job_priorities.append(jp)
|
||||
ref_data_names = self._process(
|
||||
project, build_system='taskcluster', job_priorities=job_priorities
|
||||
)
|
||||
else:
|
||||
excluded_build_system_type = None
|
||||
if build_system_type != '*':
|
||||
excluded_build_system_type = (
|
||||
'taskcluster' if build_system_type == 'buildbot' else 'buildbot'
|
||||
)
|
||||
job_priorities = self._query_job_priorities(priority, excluded_build_system_type)
|
||||
ref_data_names = self._process(project, build_system_type, job_priorities)
|
||||
|
||||
# We don't really need 'jobtypes' and today's date in the returning data
|
||||
# Getting rid of it will require the consumers to not expect it.
|
||||
# https://bugzilla.mozilla.org/show_bug.cgi?id=1325405
|
||||
return {'jobtypes': {str(datetime.date.today()): sorted(ref_data_names)}}
|
||||
|
||||
|
||||
# create an instance of this class, and expose `seta_job_scheduling`
|
||||
seta_job_scheduling = SETAJobPriorities().seta_job_scheduling
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
from django.core.management.base import BaseCommand
|
||||
|
||||
from treeherder.seta.analyze_failures import AnalyzeFailures
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
'Analyze jobs that failed and got tagged with fixed_by_commit '
|
||||
'and change the priority and timeout of such job.'
|
||||
)
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
dest="dry_run",
|
||||
help="This mode is for analyzing failures without " "updating the job priority table.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--ignore-failures",
|
||||
type=int,
|
||||
dest="ignore_failures",
|
||||
default=0,
|
||||
help="If a job fails less than N times we don't take that job" "into account.",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
AnalyzeFailures(**options).run()
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
import logging
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.preseed import load_preseed
|
||||
from treeherder.seta.update_job_priority import update_job_priority_table
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = 'Initialize or update SETA data; It causes no harm to run on production'
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
'--clear-job-priority-table',
|
||||
action='store_true',
|
||||
dest='clear_jp_table',
|
||||
default=False,
|
||||
help='Delete all entries in the JobPriority table.',
|
||||
)
|
||||
|
||||
def clear_job_priority_table(self):
|
||||
logger.info('Number of items in table: %d', JobPriority.objects.count())
|
||||
logger.info('Deleting all entries in the job priority table.')
|
||||
JobPriority.objects.all().delete()
|
||||
logger.info('Number of items in table: %d', JobPriority.objects.count())
|
||||
|
||||
def initialize_seta(self):
|
||||
logger.info('Updating JobPriority table.')
|
||||
logger.info('Number of items in table: %d', JobPriority.objects.count())
|
||||
update_job_priority_table()
|
||||
logger.info('Loading preseed table.')
|
||||
logger.info('Number of items in table: %d', JobPriority.objects.count())
|
||||
load_preseed()
|
||||
logger.info('Number of items in table: %d', JobPriority.objects.count())
|
||||
|
||||
def handle(self, *args, **options):
|
||||
if options['clear_jp_table']:
|
||||
self.clear_job_priority_table()
|
||||
else:
|
||||
self.initialize_seta()
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
from django.core.management.base import BaseCommand
|
||||
|
||||
from treeherder.seta.preseed import load_preseed
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = 'Update job priority table with data based on preseed.json'
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--validate",
|
||||
action="store_true",
|
||||
help="This will validate that all entries in preseed.json are valid",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
load_preseed(options.get("validate"))
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Generated by Django 1.10.5 on 2017-01-26 15:42
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
initial = True
|
||||
|
||||
dependencies = []
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='JobPriority',
|
||||
fields=[
|
||||
(
|
||||
'id',
|
||||
models.AutoField(
|
||||
auto_created=True, primary_key=True, serialize=False, verbose_name='ID'
|
||||
),
|
||||
),
|
||||
('testtype', models.CharField(max_length=128)),
|
||||
('buildsystem', models.CharField(max_length=64)),
|
||||
('buildtype', models.CharField(max_length=64)),
|
||||
('platform', models.CharField(max_length=64)),
|
||||
('priority', models.IntegerField()),
|
||||
('expiration_date', models.DateTimeField(null=True)),
|
||||
],
|
||||
),
|
||||
]
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
import logging
|
||||
|
||||
from django.db import models
|
||||
from django.utils import timezone
|
||||
|
||||
from treeherder.seta.common import unique_key
|
||||
from treeherder.seta.settings import SETA_LOW_VALUE_PRIORITY
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class JobPriorityManager(models.Manager):
|
||||
def clear_expiration_field_for_expired_jobs(self):
|
||||
'''Set the expiration date of every job that has expired.'''
|
||||
# Only select rows where there is an expiration date set
|
||||
for job in JobPriority.objects.filter(expiration_date__isnull=False):
|
||||
if job.has_expired():
|
||||
job.expiration_date = None
|
||||
job.save()
|
||||
|
||||
def adjust_jobs_priority(self, high_value_jobs, priority=1):
|
||||
"""For every job priority determine if we need to increase or decrease the job priority
|
||||
|
||||
Currently, high value jobs have a priority of 1 and a timeout of 0.
|
||||
"""
|
||||
# Only job priorities that don't have an expiration date (2 weeks for new jobs or year 2100
|
||||
# for jobs update via load_preseed) are updated
|
||||
for jp in JobPriority.objects.filter(expiration_date__isnull=True):
|
||||
if jp.unique_identifier() not in high_value_jobs:
|
||||
if jp.priority != SETA_LOW_VALUE_PRIORITY:
|
||||
logger.warning('Decreasing priority of %s', jp.unique_identifier())
|
||||
jp.priority = SETA_LOW_VALUE_PRIORITY
|
||||
jp.save(update_fields=['priority'])
|
||||
elif jp.priority != priority:
|
||||
logger.warning('Increasing priority of %s', jp.unique_identifier())
|
||||
jp.priority = priority
|
||||
jp.save(update_fields=['priority'])
|
||||
|
||||
|
||||
class JobPriority(models.Model):
|
||||
# Use custom manager
|
||||
objects = JobPriorityManager()
|
||||
|
||||
# This field is sanitized to unify name from Buildbot and TaskCluster
|
||||
testtype = models.CharField(max_length=128) # e.g. web-platform-tests-1
|
||||
buildsystem = models.CharField(max_length=64)
|
||||
buildtype = models.CharField(max_length=64) # e.g. {opt,pgo,debug}
|
||||
platform = models.CharField(max_length=64) # e.g. windows8-64
|
||||
priority = models.IntegerField() # 1 or 5
|
||||
expiration_date = models.DateTimeField(null=True)
|
||||
|
||||
# Q: Do we need indexing?
|
||||
unique_together = ('testtype', 'buildtype', 'platform')
|
||||
|
||||
def has_expired(self):
|
||||
now = timezone.now()
|
||||
if self.expiration_date:
|
||||
return self.expiration_date < now
|
||||
else:
|
||||
return True
|
||||
|
||||
def unique_identifier(self):
|
||||
return unique_key(testtype=self.testtype, buildtype=self.buildtype, platform=self.platform)
|
||||
|
||||
def __str__(self):
|
||||
return ','.join((self.buildsystem, self.testtype, self.buildtype, self.platform))
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
[
|
||||
{
|
||||
"buildtype": "asan",
|
||||
"testtype": "build-android-x86_64-asan-fuzzing/opt",
|
||||
"platform": "android-5-0-x86_64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "asan",
|
||||
"testtype": "build-linux64-asan-fuzzing/opt",
|
||||
"platform": "linux64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "opt",
|
||||
"testtype": "build-linux64-fuzzing-ccov/opt",
|
||||
"platform": "linux64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "debug",
|
||||
"testtype": "build-linux64-fuzzing/debug",
|
||||
"platform": "linux64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "asan",
|
||||
"testtype": "build-macosx64-asan-fuzzing/opt",
|
||||
"platform": "osx-cross",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "debug",
|
||||
"testtype": "build-macosx64-fuzzing/debug",
|
||||
"platform": "osx-cross",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "asan",
|
||||
"testtype": "build-win64-asan-fuzzing/opt",
|
||||
"platform": "windows2012-64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "debug",
|
||||
"testtype": "build-win64-fuzzing/debug",
|
||||
"platform": "windows2012-64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "opt",
|
||||
"testtype": "spidermonkey-sm-fuzzing-linux64/opt",
|
||||
"platform": "linux64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
},
|
||||
{
|
||||
"buildtype": "debug",
|
||||
"testtype": "build-android-x86-fuzzing/debug",
|
||||
"platform": "android-4-2-x86",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsystem": "taskcluster"
|
||||
}
|
||||
]
|
||||
|
|
@ -1,172 +0,0 @@
|
|||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
from treeherder.etl.seta import get_reference_data_names
|
||||
from treeherder.seta.common import convert_job_type_name_to_testtype
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import SETA_LOW_VALUE_PRIORITY, THE_FUTURE
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
'''preseed.json entries have fields: buildtype, testtype, platform,
|
||||
priority, expiration_date. They must match the corresponding entry in
|
||||
runnable-jobs.json.
|
||||
|
||||
buildtype should match the attribute name in the runnable jobs collection.
|
||||
|
||||
testtype should match the full task label.
|
||||
|
||||
platform should match the platform.
|
||||
|
||||
priority can be 1 to signify high value tasks or 5 to signify low
|
||||
value tasks. The default priority is 1.
|
||||
|
||||
expiration_date must be "*" to signify no expiration.
|
||||
|
||||
buildsystem should always be "taskcluster".
|
||||
|
||||
Example:
|
||||
|
||||
runnable-jobs.json:
|
||||
"build-android-x86_64-asan-fuzzing/opt": {
|
||||
"collection": {
|
||||
"asan": true
|
||||
},
|
||||
"platform": "android-5-0-x86_64",
|
||||
"symbol": "Bof"
|
||||
},
|
||||
entry in preseed.json
|
||||
{
|
||||
"buildtype": "asan",
|
||||
"testtype": "build-android-x86_64-asan-fuzzing/opt",
|
||||
"platform": "android-5-0-x86_64",
|
||||
"priority": 5,
|
||||
"expiration_date": "*",
|
||||
"buildsytem": "taskcluster"
|
||||
}
|
||||
'''
|
||||
|
||||
|
||||
def validate_preseed_entry(entry, ref_names):
|
||||
assert entry["testtype"] != "*"
|
||||
assert entry["buildtype"] != "*"
|
||||
assert entry["platform"] != "*"
|
||||
# We also support *, however, that was only useful with Buildbot
|
||||
assert entry["buildsystem"] == "taskcluster"
|
||||
# We support values different than *, however, it is not useful for preseed
|
||||
assert entry["expiration_date"] == "*"
|
||||
assert 1 <= entry["priority"] <= SETA_LOW_VALUE_PRIORITY
|
||||
|
||||
# Collect potential matches
|
||||
potential_matches = []
|
||||
for unique_identifier, ref_name in ref_names.items():
|
||||
# XXX: Now that we have fuzzy build the term testtypes is not accurate
|
||||
if ref_name == entry["testtype"]:
|
||||
potential_matches.append(unique_identifier)
|
||||
|
||||
assert len(potential_matches) > 0, Exception(
|
||||
"%s is not valid. Please check runnable_jobs.json from a Gecko decision task.",
|
||||
entry["testtype"],
|
||||
)
|
||||
|
||||
testtype = convert_job_type_name_to_testtype(entry["testtype"])
|
||||
if not testtype:
|
||||
logger.warning(
|
||||
"Preseed.json entry testtype %s is not a valid task name:", entry["testtype"]
|
||||
)
|
||||
raise Exception("preseed.json entry contains invalid testtype. Please check output above.")
|
||||
|
||||
unique_identifier = (
|
||||
testtype,
|
||||
entry["buildtype"],
|
||||
entry["platform"],
|
||||
)
|
||||
|
||||
try:
|
||||
ref_names[unique_identifier]
|
||||
except KeyError:
|
||||
logger.warning("Preseed.json entry %s matches the following:", unique_identifier)
|
||||
logger.warning(potential_matches)
|
||||
raise Exception("We failed to match your preseed.json entry. Please check output above.")
|
||||
|
||||
|
||||
def load_preseed(validate=False):
|
||||
""" Update JobPriority information from preseed.json"""
|
||||
logger.info("About to load preseed.json")
|
||||
|
||||
preseed = preseed_data()
|
||||
if validate:
|
||||
logger.info("We are going to validate the values from preseed.json")
|
||||
ref_names = get_reference_data_names()
|
||||
for job in preseed:
|
||||
if validate:
|
||||
validate_preseed_entry(job, ref_names)
|
||||
|
||||
logger.debug("Processing %s", (job["testtype"], job["buildtype"], job["platform"]))
|
||||
queryset = JobPriority.objects.all()
|
||||
|
||||
for field in ('testtype', 'buildtype', 'platform'):
|
||||
if job[field] != '*':
|
||||
# The JobPriority table does not contain the raw
|
||||
# testtype value seen in the preseed.json file. We
|
||||
# must convert the job[field] value to the appropriate
|
||||
# value before performing the query.
|
||||
field_value = (
|
||||
convert_job_type_name_to_testtype(job[field])
|
||||
if field == 'testtype'
|
||||
else job[field]
|
||||
)
|
||||
queryset = queryset.filter(**{field: field_value})
|
||||
|
||||
# Deal with the case where we have a new entry in preseed
|
||||
if not queryset:
|
||||
create_new_entry(job)
|
||||
else:
|
||||
# We can have wildcards, so loop on all returned values in data
|
||||
for jp in queryset:
|
||||
process_job_priority(jp, job)
|
||||
logger.debug("Finished")
|
||||
|
||||
|
||||
def preseed_data():
|
||||
with open(os.path.join(os.path.dirname(__file__), 'preseed.json'), 'r') as fd:
|
||||
preseed = json.load(fd)
|
||||
|
||||
return preseed
|
||||
|
||||
|
||||
def create_new_entry(job):
|
||||
if job['expiration_date'] == '*':
|
||||
job['expiration_date'] = THE_FUTURE
|
||||
|
||||
logger.info("Adding a new job priority to the database: %s", job)
|
||||
|
||||
testtype = convert_job_type_name_to_testtype(job['testtype'])
|
||||
|
||||
JobPriority.objects.create(
|
||||
testtype=testtype,
|
||||
buildtype=job['buildtype'],
|
||||
platform=job['platform'],
|
||||
priority=job['priority'],
|
||||
expiration_date=job['expiration_date'],
|
||||
buildsystem=job['buildsystem'],
|
||||
)
|
||||
|
||||
|
||||
def process_job_priority(jp, job):
|
||||
update_fields = []
|
||||
# Updating the buildtype can be dangerous as analyze_failures can set it to '*' while in here
|
||||
# we can change it back to 'taskcluster'. For now, we will assume that creating a new entry
|
||||
# will add the right value at the beginning
|
||||
if jp.__getattribute__('priority') != job['priority']:
|
||||
jp.__setattr__('priority', job['priority'])
|
||||
update_fields.append('priority')
|
||||
|
||||
if job['expiration_date'] == '*' and jp.expiration_date != THE_FUTURE:
|
||||
jp.expiration_date = THE_FUTURE
|
||||
update_fields.append('expiration_date')
|
||||
|
||||
if update_fields:
|
||||
logger.info("Updating (%s) for these fields %s", jp, ','.join(update_fields))
|
||||
jp.save(update_fields=update_fields)
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
import datetime
|
||||
|
||||
THE_FUTURE = datetime.datetime(2100, 12, 31)
|
||||
|
||||
# repos that SETA supports
|
||||
SETA_PROJECTS = [
|
||||
'autoland',
|
||||
'try',
|
||||
]
|
||||
|
||||
# for taskcluster, only jobs that start with any of these names
|
||||
# will be supported i.e. may be optimized out by SETA
|
||||
SETA_SUPPORTED_TC_JOBTYPES = [
|
||||
'test-',
|
||||
'source-test-',
|
||||
'desktop-test',
|
||||
'android-test',
|
||||
'iris-',
|
||||
'webrender-',
|
||||
'build-android-x86-fuzzing',
|
||||
'build-android-x86_64-asan-fuzzing',
|
||||
'build-linux64-asan-fuzzing-ccov',
|
||||
'build-linux64-asan-fuzzing',
|
||||
'build-linux64-fuzzing-ccov',
|
||||
'build-linux64-fuzzing',
|
||||
'build-linux64-tsan-fuzzing',
|
||||
'build-macosx64-asan-fuzzing',
|
||||
'build-macosx64-fuzzing',
|
||||
'build-win64-asan-fuzzing',
|
||||
'build-win64-fuzzing',
|
||||
'spidermonkey-sm-fuzzing-linux64',
|
||||
]
|
||||
|
||||
# platforms listed here will not be supported by SETA
|
||||
# i.e. these will never be optimized out by SETA
|
||||
SETA_UNSUPPORTED_PLATFORMS = [
|
||||
'android-4-2-armv7-api15',
|
||||
'android-4-4-armv7-api15',
|
||||
'android-5-0-armv8-api15',
|
||||
'android-5-1-armv7-api15',
|
||||
'android-6-0-armv8-api15',
|
||||
'osx-10-7', # Build
|
||||
'osx-10-9',
|
||||
'osx-10-11',
|
||||
'other',
|
||||
'taskcluster-images',
|
||||
'windows7-64', # We don't test 64-bit builds on Windows 7 test infra
|
||||
'windows8-32', # We don't test 32-bit builds on Windows 8 test infra
|
||||
'Win 6.3.9600 x86_64',
|
||||
'linux64-stylo',
|
||||
'windowsxp',
|
||||
]
|
||||
|
||||
# testtypes listed here will not be supported by SETA
|
||||
# i.e. these will never be optimized out by SETA
|
||||
SETA_UNSUPPORTED_TESTTYPES = [
|
||||
'dep',
|
||||
'nightly',
|
||||
'non-unified',
|
||||
'valgrind',
|
||||
'Opt',
|
||||
'Debug',
|
||||
'Dbg',
|
||||
'(opt)',
|
||||
'PGO Opt',
|
||||
'Valgrind Opt',
|
||||
'Artifact Opt',
|
||||
'(debug)',
|
||||
]
|
||||
|
||||
# SETA job priority values
|
||||
SETA_HIGH_VALUE_PRIORITY = 1
|
||||
SETA_LOW_VALUE_PRIORITY = 5
|
||||
|
||||
# analyze_failures retrieves jobs marked 'fixed by commit' for these repos
|
||||
SETA_FIXED_BY_COMMIT_REPOS = [
|
||||
'autoland',
|
||||
'mozilla-central',
|
||||
]
|
||||
|
||||
# analyze_failures retrieves jobs marked 'fixed by commit' for the past N days
|
||||
SETA_FIXED_BY_COMMIT_DAYS = 90
|
||||
|
||||
# when retrieving taskcluster runnable jobs, and processing
|
||||
# them, cache the resulting reference data names map for N seconds; this
|
||||
# helps reduce the number of API calls when getting job priorities
|
||||
SETA_REF_DATA_NAMES_CACHE_TIMEOUT = 3600
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
from treeherder.seta.analyze_failures import AnalyzeFailures
|
||||
from treeherder.workers.task import retryable_task
|
||||
|
||||
|
||||
@retryable_task(name='seta-analyze-failures', max_retries=3, soft_time_limit=5 * 60)
|
||||
def seta_analyze_failures():
|
||||
'''We analyze all starred test failures from the last four months which were fixed by a commit.'''
|
||||
AnalyzeFailures().run()
|
||||
|
|
@ -1,197 +0,0 @@
|
|||
'''This module is used to add new jobs to the job priority table.
|
||||
|
||||
This will query the Treeherder runnable api based on the latest task ID from
|
||||
autoland's TaskCluster index.
|
||||
|
||||
Known bug:
|
||||
* Only considering autoland makes SETA act similarly in all repositories where it is
|
||||
active. Right now, this works for integration repositories since they tend to have
|
||||
the same set of jobs. Unfortunately, this is less than ideal if we want to make this
|
||||
work for project repositories.
|
||||
'''
|
||||
import logging
|
||||
|
||||
from treeherder.etl.runnable_jobs import list_runnable_jobs
|
||||
from treeherder.etl.seta import parse_testtype, valid_platform
|
||||
from treeherder.seta.common import job_priority_index, unique_key
|
||||
from treeherder.seta.models import JobPriority
|
||||
from treeherder.seta.settings import SETA_LOW_VALUE_PRIORITY
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def update_job_priority_table():
|
||||
"""Use it to update the job priority table with data from the runnable api."""
|
||||
# XXX: We are assuming that the jobs accross 'mozilla-inbound' and 'autoland'
|
||||
# are equivalent. This could cause issues in the future
|
||||
data = query_sanitized_data(repo_name='autoland')
|
||||
if data:
|
||||
return _update_table(data)
|
||||
else:
|
||||
# XXX: Should we do this differently?
|
||||
logger.warning('We received an empty data set')
|
||||
return
|
||||
|
||||
|
||||
def _unique_key(job):
|
||||
"""Return a key to query our uniqueness mapping system.
|
||||
|
||||
This makes sure that we use a consistent key between our code and selecting jobs from the
|
||||
table.
|
||||
"""
|
||||
testtype = str(job['testtype'])
|
||||
if not testtype:
|
||||
raise Exception('Bad job {}'.format(job))
|
||||
return unique_key(
|
||||
testtype=testtype, buildtype=str(job['platform_option']), platform=str(job['platform'])
|
||||
)
|
||||
|
||||
|
||||
def _sanitize_data(runnable_jobs_data):
|
||||
"""We receive data from runnable jobs api and return the sanitized data that meets our needs.
|
||||
|
||||
This is a loop to remove duplicates (including buildsystem -> * transformations if needed)
|
||||
By doing this, it allows us to have a single database query
|
||||
|
||||
It returns sanitized_list which will contain a subset which excludes:
|
||||
* jobs that don't specify the platform
|
||||
* jobs that don't specify the testtype
|
||||
* if the job appears again, we replace build_system_type with '*'. By doing so, if a job appears
|
||||
under both 'buildbot' and 'taskcluster', its build_system_type will be '*'
|
||||
"""
|
||||
job_build_system_type = {}
|
||||
sanitized_list = []
|
||||
for job in runnable_jobs_data:
|
||||
if not valid_platform(job['platform']):
|
||||
logger.debug('Invalid platform %s', job['platform'])
|
||||
continue
|
||||
|
||||
testtype = parse_testtype(
|
||||
build_system_type=job['build_system_type'],
|
||||
job_type_name=job['job_type_name'],
|
||||
platform_option=job['platform_option'],
|
||||
ref_data_name=job['ref_data_name'],
|
||||
)
|
||||
|
||||
if not testtype:
|
||||
continue
|
||||
|
||||
# NOTE: This is *all* the data we need from the runnable API
|
||||
new_job = {
|
||||
'build_system_type': job['build_system_type'], # e.g. {buildbot,taskcluster,*}
|
||||
'platform': job['platform'], # e.g. windows8-64
|
||||
'platform_option': job['platform_option'], # e.g. {opt,debug}
|
||||
'testtype': testtype, # e.g. web-platform-tests-1
|
||||
}
|
||||
key = _unique_key(new_job)
|
||||
|
||||
# Let's build a map of all the jobs and if duplicated change the build_system_type to *
|
||||
if key not in job_build_system_type:
|
||||
job_build_system_type[key] = job['build_system_type']
|
||||
sanitized_list.append(new_job)
|
||||
elif new_job['build_system_type'] != job_build_system_type[key]:
|
||||
new_job['build_system_type'] = job_build_system_type[key]
|
||||
# This will *replace* the previous build system type with '*'
|
||||
# This guarantees that we don't have duplicates
|
||||
sanitized_list[sanitized_list.index(new_job)]['build_system_type'] = '*'
|
||||
|
||||
return sanitized_list
|
||||
|
||||
|
||||
def query_sanitized_data(repo_name='autoland'):
|
||||
"""Return sanitized jobs data based on runnable api. None if failed to obtain or no new data.
|
||||
|
||||
We need to find the latest gecko decision task ID (by querying the index [1][2]).
|
||||
|
||||
[1] firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.%s.latest.taskgraph.decision/
|
||||
[2] Index's data structure:
|
||||
{
|
||||
"namespace": "gecko.v2.autoland.latest.taskgraph.decision",
|
||||
"taskId": "Dh9ZvFk5QCSprJ877cgUmw",
|
||||
"rank": 0,
|
||||
"data": {},
|
||||
"expires": "2017-10-06T18:30:18.428Z"
|
||||
}
|
||||
"""
|
||||
runnable_jobs = list_runnable_jobs(repo_name)
|
||||
return _sanitize_data(runnable_jobs)
|
||||
|
||||
|
||||
def _initialize_values():
|
||||
logger.info('Fetch all rows from the job priority table.')
|
||||
# Get all rows of job priorities
|
||||
jp_index = job_priority_index(JobPriority.objects.all())
|
||||
return jp_index, SETA_LOW_VALUE_PRIORITY, None
|
||||
|
||||
|
||||
def _update_table(data):
|
||||
"""Add new jobs to the priority table and update the build system if required.
|
||||
data - it is a list of dictionaries that describe a job type
|
||||
|
||||
returns the number of new, failed and updated jobs
|
||||
"""
|
||||
jp_index, priority, expiration_date = _initialize_values()
|
||||
|
||||
total_jobs = len(data)
|
||||
new_jobs, failed_changes, updated_jobs = 0, 0, 0
|
||||
# Loop through sanitized jobs, add new jobs and update the build system if needed
|
||||
for job in data:
|
||||
key = _unique_key(job)
|
||||
if key in jp_index:
|
||||
# We already know about this job, we might need to update the build system
|
||||
# We're seeing the job again with another build system (e.g. buildbot vs
|
||||
# taskcluster). We need to change it to '*'
|
||||
if (
|
||||
jp_index[key]['build_system_type'] != '*'
|
||||
and jp_index[key]['build_system_type'] != job["build_system_type"]
|
||||
):
|
||||
db_job = JobPriority.objects.get(pk=jp_index[key]['pk'])
|
||||
db_job.buildsystem = '*'
|
||||
db_job.save()
|
||||
|
||||
logger.info(
|
||||
'Updated %s/%s from %s to %s',
|
||||
db_job.testtype,
|
||||
db_job.buildtype,
|
||||
job['build_system_type'],
|
||||
db_job.buildsystem,
|
||||
)
|
||||
updated_jobs += 1
|
||||
|
||||
else:
|
||||
# We have a new job from runnablejobs to add to our master list
|
||||
try:
|
||||
jobpriority = JobPriority(
|
||||
testtype=str(job["testtype"]),
|
||||
buildtype=str(job["platform_option"]),
|
||||
platform=str(job["platform"]),
|
||||
priority=priority,
|
||||
expiration_date=expiration_date,
|
||||
buildsystem=job["build_system_type"],
|
||||
)
|
||||
jobpriority.save()
|
||||
logger.debug(
|
||||
'New job was found (%s,%s,%s,%s)',
|
||||
job['testtype'],
|
||||
job['platform_option'],
|
||||
job['platform'],
|
||||
job["build_system_type"],
|
||||
)
|
||||
new_jobs += 1
|
||||
except Exception as error:
|
||||
logger.warning(str(error))
|
||||
failed_changes += 1
|
||||
|
||||
logger.info(
|
||||
'We have %s new jobs and %s updated jobs out of %s total jobs processed.',
|
||||
new_jobs,
|
||||
updated_jobs,
|
||||
total_jobs,
|
||||
)
|
||||
|
||||
if failed_changes != 0:
|
||||
logger.warning(
|
||||
'We have failed %s changes out of %s total jobs processed.', failed_changes, total_jobs
|
||||
)
|
||||
|
||||
return new_jobs, failed_changes, updated_jobs
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
from rest_framework import status, viewsets
|
||||
from rest_framework.response import Response
|
||||
|
||||
from treeherder.seta.job_priorities import SetaError, seta_job_scheduling
|
||||
|
||||
|
||||
class SetaJobPriorityViewSet(viewsets.ViewSet):
|
||||
def list(self, request, project):
|
||||
"""Routing to /api/project/{project}/seta/job-priorities/
|
||||
|
||||
This API can potentially have these consumers:
|
||||
* Buildbot
|
||||
* build_system_type=buildbot
|
||||
* priority=5
|
||||
* format=json
|
||||
* TaskCluster (Gecko decision task)
|
||||
* build_system_type=taskcluster
|
||||
* format=json
|
||||
"""
|
||||
build_system_type = request.query_params.get('build_system_type', '*')
|
||||
priority = request.query_params.get('priority')
|
||||
try:
|
||||
return Response(seta_job_scheduling(project, build_system_type, priority))
|
||||
except SetaError as e:
|
||||
return Response(str(e), status=status.HTTP_400_BAD_REQUEST)
|
||||
|
|
@ -19,7 +19,6 @@ from treeherder.webapp.api import (
|
|||
performance_data,
|
||||
push,
|
||||
refdata,
|
||||
seta,
|
||||
)
|
||||
|
||||
# router for views that are bound to a project
|
||||
|
|
@ -36,10 +35,6 @@ project_bound_router.register(
|
|||
basename='jobs',
|
||||
)
|
||||
|
||||
project_bound_router.register(
|
||||
r'seta/job-priorities', seta.SetaJobPriorityViewSet, basename='seta-job-priorities'
|
||||
)
|
||||
|
||||
project_bound_router.register(
|
||||
r'push',
|
||||
push.PushViewSet,
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче