Since with the new per-user Hawk credentials, the same auth object can
be used for the whole session, so should just be passed when
instantiating TreeherderClient.
The "performance adapter class" never stored any persistent state, so
let's just refactor it into a bunch of standalone methods. Easier to
understand and reason about.
* Put talos-specific stuff in the talos data adapter
* Put generic stuff in the generic adapter, in preparation for creating
a generic perfherder data ingestion path
* Add some explanatory comments
* Use better casing for static defines
* Remove some now-unused code related to json float encoding
Since they're not specific to the Django app 'webapp'.
Whilst we're there, the local & example settings files have been
renamed. In the future I'd like to combine settings_local.example.py
with puppet/files/treeherder/local.vagrant.py, but I'll do that in
another bug.
I added a create_credentials command to help setting up the initial
development environment. The puppet setup now creates a new user and set
it as the owner of the treeherder-etl credentials.
By storing the duration as an int rather than a timedelta, we can set it
via the environment, rather than having to rely on local.py.
The variable has also been renamed to ease the transition and also make
its purpose clearer. Prod/Heroku use the default duration, and stage has
already had DATA_CYCLE_DAYS set to 45 in its environment, for parity
with the old stage local.py DATA_CYCLE_INTERVAL setting.
The hawk credentials lookup function is the glue between hawk and
the `application` django app. I wrote tests to verify its logic,
everything else is mostly configuration code.
It's unused in the UI and doesn't add any value, since we're
normally much more interested in the push_timestamp.
This can land without causing errors, since the field is DEFAULT NULL,
so can be dropped at our leisure later.
As of karma-jasmine 0.3.0, the jasmine library is not bundled with
karma-jasmine, and so jasmine-core has to be installed separately.
jasmine-jquery also has to be updated, since the old version was not
compatible. A duplicate jasmine-jquery file that was outside of the
tests vendor directory has also been removed.
This adds an autoclassify command and a detect_intermittents command.
The former is designed to take an incoming job with an error summary
and look for existing results marked as intermittent that are a close
match for the new result. At present only one matcher is implemented;
this requires an exact match in terms of test name, result and error
message. Matching is also constrained to be based on single lines; it
is anticipated that future iterations may add support for matching on
groups of lines.
The detect_intermittents command is designed to take a group of jobs
running on the same push and with the same build job (i.e. same
testsuite, same chunk, etc.) and look for new intermittents to add to
the database. This currently only looks for test failures where there
is at least one green job and one non-green job.
There is currently no UI for seeing matches or for adding new
prototypical intermittents as match candidates. There is also no
integration with bugzilla; future development should add association
of frequent intermittents with bugs.
* Stop storing mean/median/max/min for subtests as they're not always on the
same scale as the summary value
* Always use "filtered" value provided by Talos for subtest summary values
* Store value for tests as a "value" property for the series data
They were never used for anything and take up a lot of space. They
also don't fit into the new performance model we're working on. We
can always bring back the useful bits later.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
This test checks that:
* log content that falls in-between two step markers is captured in a
dummy unnamed step.
* if the final step is missing the "step finish" marker, we still save/
update the step and append any step errors to all_errors.
Bug 1060339 made check_errors always be true, since we want to parse all
logs, not just those for failing jobs. As such, we have no use for
check_errors.
Previously the tests were created after act["logurl"] was deleted, so
the resultant expected output .json file was missing the "logurl" key.
The json import was also missing - people now just have to uncomment the
test creation block, and not also add the missing import each time.
Created using |isort -p tests -rc .| and a couple of manual tweaks.
The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
* Simplify logic in talos parser (there was an optimization which didn't
save anything and just caused confusion before)
* Make it so if log parsing fails for a non-http reason, we don't try
again
urllib isn't handling the unicode found in some log lines correctly,
whereas requests does. This prevents UnicodeEncodeError exceptions when
making the request to the bugscache API to find the bug suggestions for
these log lines.
The sample notes added for the test normally have the same timestamp for
several notes, but not always. With the previous `ORDER BY`, this meant
the list of notes retrieved could vary in order depending on if the
timestamps were identical. We now additionally sort by id (descending,
to match the timestamp sort), so the returned list is deterministic.
Since bug 1140349, the objectstore endpoint has been deprecated, and
performs the same function as the jobs endpoint. Now that there are no
remaining submitters to it, let's remove it.
To avoid continually attempting to re-ingest them, thereby reducing
task runtime and database load.
In order to make this behaviour easier to test, the
pending/running/build4hr jobs process run() method now returns True if
new jobs were loaded, and False otherwise. This method was used instead
of calling the transformer mixins from the test directly, since the test
would then have had to reimplement much of the run() method anyway.
After the previous commit, the Objectstore is effectively "dead code".
So this commit removes all the dead code after anything left over in
the Objectstore has been drained and added to the DB.
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
The deployment script also had to be updated, so that we set the
prod/stage-specific environment variables before using manage.py, since
dj-database-url cannot rely on what's in the stage/prod local.py config
(which isn't a bad thing, since we're deprecating that file).
This patch upgrades the version stored in the requirements file and fixes some issues introduced by breaking changes in the new version of the library:
- Writable nested fields are not available anymore, you need an explicit create method on the serializer to write a nested field.
- ModelViewSet now requires serializer_class and queryset attributes.
- @action and @link decorators are now replaced by either @detail_route or @list_route.
- any attempt to create a ModelSerializer instance with an attribute which type is either dict or list will raise an exception.
Since we use Celery for queueing job ingestion, the objectstore is
now irrelevant. This code is the first step. This will bypass
the Objectstore and ingest jobs directly to our ``jobs`` database.
Phase 2 is to remove all the Objectstore code (in a later commit)
Phase 3 is to delete the Objectstore databases and related fields in
other tables.
It appears that on occasion we parse a log more than once, which
resulted in duplicate performance series going into the database.
Let's be resilient about this by not inserting duplicate jobs into the
database (we always attach a unique job id to every datapoint, so there's
no chance of accidentally removing entries which happen to have the same
performance numbers)
Having the ability to use different DB hosts for each project sounded
like a good idea, but in reality, we have no need for it.
This switches us to using the global read-write and read-only database
host names rather than the fields on the datasource table. As such, the
'host', 'read_only_host' and 'type' (eg 'mysql') fields can be removed.
The Django model had a unique_together on host+name, so we now need to
make 'name' (ie database name) a unique key on it's own.
In addition, this removes the 'creation_date' field, since we don't use
it anywhere, and we can just look at the commit history to see when a
repo was created. (I imagine it may have had more use if we actually had
started partitioning the databases uses the old 'dataset' count field).
In a future bug, I'll remove the redundant substitution of 'engine' for
'InnoDB' in the template schema, given that engine is now always InnoDB
in create_db().
Since otherwise we get access denied errors using run_sql on Heroku.
All other calls use datasource, so have already been set up to pass the
SSL options.
dj-database-url extracts DB host, port, username, password and database
name from the env variable 'DATABASE_URL' (unless another env variable
name is specified). If the env variable is not defined, it falls back to
the default passed to dj_database_url.config().
This means for Heroku and similar we can replace the multiple DB env
variables with just one URL for default & one for read_only.
This also effectively makes the setting of the read only DB variable
mandatory for stage/production/heroku, since DEFAULT_DATABASE_URL won't
be valid for them - so prevents us inadvertently not using the read only
DB.
Before this is deployed, we'll need to update the stage/prod puppet
configs & Heroku settings to add the new environment variable.
* Put # of runs directly beside average/geomeans in UI, and put a dotted
line underneath to make it easier to pull up tooltip
* Use a bootstrap tooltip for displaying run information (clearer)
* Use a bootstrap abbreviation to make it more clear what low/med/high
confidence actually means
Ed Morley
Tiramisu 1993
Cameron Dawson
Dustin J. Mitchell
William Lachance
Mauro Doglio
Ed Morley
Jonathan French
Maja Frydrychowicz
James Graham
William Lachance
goma
Jonathan Griffin
Kim Moir
Greg Arndt
Chris Manchester
Justin Wood
Joel Maher
Vaibhav Agrawal