The request body, request headers & response headers are now output with
a log level of `debug` rather than `error`, to reduce verbosity of logs.
In addition, the response body has now been combined with the log line
that outputs the URL, to make it easier to correlate lines in the log.
The type in the `_get_json()` ("Error *submitting* data") has also been
fixed, and string substitution now occurs lazily.
This test checks that the `calculate_duration` command correctly
populates the `job_duration` table both on an initial run and when new
jobs cause an existing average duration to change.
It also checks that once the task has been run, any subsequent ingestion
of jobs with matching signatures cause those jobs to have the correct
`running_eta` value when inserted into the `job` table.
Instead of storing the calculated average job durations in the
per-project job_eta tables, they are now stored in the new job_duration
table in the main treeherder database. This means we can use Django's
ORM instead of datasource.
In addition, if a job signature already exists in the table, we update
it rather than creating duplicate rows - and most importantly unlike the
previous table, the correct columns are indexed to avoid performance
problems.
This commit just changes where we store new durations, a later commit
will switch the retrieval of them from the old table to the new.
Which will replace the per-project job_eta tables.
I've not added a `modified` field, since I think it's best we wait until
we actually need it. One use-case would be for data expiration, however
I think that's best done by cross-referencing signatures on the
reference_data_signatures table, however (a) that table doesn't yet have
data expiration either, and (b) it's possible once that table is
refactored we may even want to combine it with the job_durations table
too.
At the moment tests that store jobs can get away with not having the
test project listed in the repository table. However with later commits,
the new job_duration table will have a foreign key on the repository
table, which will break those tests unless they use the test_repository
fixture, which creates the repository record for them.
Since we're not calculating ETAs (the UI does that once it knows the
start time and expected duraction), we're calculating recent average
durations instead.
We only ever use the 'running' subkey for each signature (since
bug 1181572), so there's no point in including both 'running' and
'pending' subkeys in the dict returned from get_job_eta_times().
Switches the datasource query to return the raw tuple from
cursor.fetchall() rather the dict return_type which repacks everything
unnecessarily.
Also renames the query and variables since what's being calculated is no
longer a group (it's just the average duration and not pending time too)
and 'duration' is more accurate than 'eta' (the latter is a time not an
int).
Previously the time a job spent in the pending state *and* also the time
spent in the running state were calculated and stored. However, now that
only the latter is required, the time window that the query searches
over needs to only cover the start_timestamp onwards, not the earlier
submit_timestamp.
The calculate_eta task calculates average/median/min/max & standard
deviation of recent job durations, when only the average is used.
Whilst the job_eta table will be replaced entirely in later commits, the
query that fetches the stats to populate it will still be used, so needs
cleaning up regardless.
Currently if a series does not exist on another branch (i.e. MacOSX 10.10
on b2g-inbound), Perfherder graphs won't be able to fetch the related
series. The fix for this is so simple, I'm just going to push it.
This test checks that `./manage.py makemigrations` was run, and the
resultant migrations file committed to the repo, since the last time the
Django models were updated.
Django 1.8 only supports an `exit_code` option, which unhelpfully makes
the command `sys.exit(1)` if there are *no* missing migrations, when
we're more interested in the opposite. As such, we have to confirm that
it does exit 1 (which causes a `SystemExit` exception).
On Django master they've replaced `exit_code` with `check_changes` that
inverts the check, which we should switch to once we're using a version
of Django that includes that fix.
The individual RABBITMQ_* variables are never used on their own, so it
makes more sense to switch to a URL variable that combines all of them.
Travis/Vagrant have also had BROKER URL explicitly set, since we're
generally moving away from having testing defaults set in settings.py.
Once stage/prod have BROKER_URL set, we can remove the fallback to the
old variable names, and also remove defaults entirely, making missing
settings fail fast.
Whilst Django itself handles the password property being set to `None`,
we use the Django DB configs in several places outside of Django (either
directly with MySQLdb, or via datasource which uses MySQLdb too).
MySQLdb will handle the password being the empty string, but raises
if `None` is passed, which can occur if using django-environ, due to:
https://github.com/joke2k/django-environ/issues/56
This change both works around that issue, and is also likely the right
thing to do regardless, since we shouldn't assume that the password is
even set in settings.py at all. (Django defaults the password to the
empty string, so it's perfectly acceptable to omit the password
property in the DATABASES dict entirely.)
Now that we're using django-environ's env.bool() in settings.py, the
value of TREEHERDER_DEBUG set in the Vagrant environment is correctly
picked up, so the local settings file DEBUG define is unnecessary.
* Make sorting more deterministic in cases where we have multiple
entries with the same push timestamp
* Internal PerfDatum clas renamed to just Datum
* Remove buildid parameter from Datum (not used for anything)
* Remove testrun_timestamp paramater from Datum (it's not really useful,
better to use push_timestamp
* Consolidate debug printing into one function that shows
everything
These were present in the raw SQL used prior to bug 1193836, so should
already exist on stage/prod/heroku. As such, I've added these to the
initial migration to avoid errors during migrate/having to use `--fake`.
Also remove the duplication between the two pages, by having the
submitting data section not mention requesting credentials at all, and
leave that to the common tasks page instead.
William Lachance
Tiramisu 1993
camd
Ed Morley
William Lachance
Wes Kocher