Граф коммитов

113 Коммитов

Автор SHA1 Сообщение Дата
Bastien Abadie 69b9c34289 Use taskboot to deploy to PyPI (#923) 2019-09-04 12:37:58 +02:00
Pankaj Patil e9676d0c2a Rename spawn_data_pipeline.py to spawn_pipeline.py (#855) 2019-08-05 14:27:03 +02:00
Marco 8aac03002a
Use relman-* workers instead of releng-svc (#842)
Fixes #324
2019-08-03 00:40:38 +02:00
calixteman 6b34fcf850 Update mercurial to version 5.1 (#841)
Fixes #549.
2019-08-02 17:22:12 +02:00
Boris Feld 154014608c Validate batch endpoint inputs (#794) 2019-08-01 15:33:20 +02:00
cklyyung 5b0b403875 Remove linting task comment about only running on modified files (#791)
Given that the time savings are negligible.

Closes #298.
2019-07-30 14:18:30 +02:00
Marco Castelluccio a4e1399f1b Use my email as owner of CI tasks, or dependabot PRs fail 2019-07-22 11:49:17 +02:00
Boris Feld e7add98563 Update task-boot to 0.1.9 (#675) 2019-07-05 15:36:16 +02:00
Marco Castelluccio 8e23bb7f8a Add a hook to classify a Phabricator diff 2019-07-02 19:39:59 +02:00
Boris Feld 28e1daf526 Avoid putting the version in the env (#655)
Fixes #570
2019-06-28 23:55:28 +02:00
Boris Feld 54e41d1497 Use taskboot 0.1.8 (#645)
The new taskboot release solves the double build on non-tag commits and
allows the heroku deploy to be fully atomic.
2019-06-28 11:11:48 +02:00
Boris Feld 2b48e153cb Use tagged version in bugbug hooks (#640)
Fixes #634
2019-06-27 15:47:57 +02:00
Marco Castelluccio 56f224b9dc Generate microannotate repository for mozilla-central 2019-06-26 18:57:36 +02:00
Marco Castelluccio 4c20645e04 Install required json-e and pyyaml requirements for tests 2019-06-26 18:43:15 +02:00
Boris Feld 0b43d289ac Fix default Python version declaration in pre-commit config (#584)
* Fix default Python version declaration in pre-commit config

The new way of of declaring default Python version was introduced in
pre-commit 1.14.0 and warning about unknown config keys was introduced in
pre-commit 1.17.0.
2019-06-13 13:15:53 +02:00
Boris Feld c06db28442 Bump taskboot to version 1.0.7 (#583)
Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update
task-boot version to use it.

Also add missing tags and cache option when building Docker images in
data-pipeline.yml
2019-06-12 20:11:34 +02:00
Marco Castelluccio f0993a7ce9 Test installing extra dependencies too 2019-06-08 21:11:41 +02:00
Marco Castelluccio 75dce849c1 Collect coverage for setup.py too 2019-06-08 20:56:09 +02:00
Marco a63c8a12ca
Enable coverage collection during tests (#571)
* Enable coverage collection during tests

* Use codecov.io bash uploader to upload coverage report
2019-06-08 12:23:40 +02:00
Marco Castelluccio 1ab26c67d1 Be more quiet when installing packages with apt 2019-06-08 11:50:13 +02:00
Marco Castelluccio ff5eba3832 Remove unused JSON-e variables 2019-06-08 11:25:16 +02:00
Marco Castelluccio b8809c12c0 Disable detachedHead git advice to avoid useless warnings in the logs 2019-06-08 11:25:13 +02:00
Marco Castelluccio 353d21d01b Clone repository quietly 2019-06-08 11:19:01 +02:00
Marco Castelluccio 259175ae61 Install Python dependencies quietly 2019-06-08 11:17:38 +02:00
Marco Castelluccio 9de91456f6 Update to taskboot 0.1.6 2019-06-07 22:03:00 +02:00
Marco Castelluccio 0b06016793 Remove GIT_REPOSITORY and GIT_REVISION from the update hook tasks, as we now clone the repo ourselves
Otherwise task-boot fails as it tries to clone it again.
2019-06-07 17:14:26 +02:00
Marco Castelluccio 82d9c0ece0 Update to taskboot 0.1.5 2019-06-07 16:47:28 +02:00
Boris Feld 7906380e6f Bump version of taskboot to use latest version of img tool (#562)
It is necessary to support mulit-tag Docker image building
2019-06-07 12:21:09 +02:00
Boris Feld 32f56a3962 Add a script to update the hook definition with the TAG during release (#507)
Fixes #501, fixed relanding of #491.
2019-06-06 18:11:59 +02:00
Boris Feld 08e36a7d8a Build tagged Docker images (#554) 2019-06-06 18:06:16 +02:00
Marco Castelluccio ad98a3f911 Add tests for repository.hg_log
Fixes #385
2019-06-03 10:22:08 +02:00
Marco Castelluccio 0037ad35b5 Revert "Add a script to update the hook definition with the TAG during release (#491)"
This reverts commit 6ef95cfec3.
2019-05-25 00:16:41 +02:00
Marco Castelluccio e281af3b21 Revert "Fix typo for set_hook_env_var (#494)"
This reverts commit 37da4c5135.
2019-05-25 00:16:39 +02:00
Bastien Abadie 37da4c5135 Fix typo for set_hook_env_var (#494) 2019-05-24 15:37:35 +02:00
Boris Feld 6ef95cfec3 Add a script to update the hook definition with the TAG during release (#491) 2019-05-24 15:13:52 +02:00
Boris Feld 1bae5834ab Implement deployment to Heroku (#458) 2019-05-23 20:39:02 +02:00
Boris Feld 0a5e37439d Add a central place where the models are defined (#398)
* Add a central place where the models are defined

Also add some helpers to load a model.

* Add missing tensorflow dependency in extra-nn-requirements.txt
2019-05-16 15:34:38 +02:00
Boris Feld d3c3bcbece Bump version of taskboot used in taskcluster and data pipeline (#446) 2019-05-16 13:02:58 +02:00
Boris Feld f4b2b938be
Add basic check method and check script (#341)
* Add basic check method and check script

* Ensure the check of component will correctly use super result

* Add required infra to schedule model checks

* Add scheduling bits for the model checks

* Remove the filtering on classification

* Extract counting bugs to a new function in bugzilla.py

* Also checks conflated components

* Fix new hook id

* Call bugzilla with the count_only param to speed up the check

* Fix the new hook scope to match the hook id

* Fix component model check after previous refactoring

* Fix component model check method

* Use a bugzilla report for even faster component model check

* Clarify get_product_component_count docstring

We are already filtering out full component with 0 bugs

* Update conflated components mapping check

A conflated component could also be part of the conflated components mapping

* Distinguish between non-existing full components and empty full components

* Remove the filter on resolution and unnecessary url params

* Update component check method

Keep checks as separate as possible for clarity, we could merge them or makes
them faster later

* Generate dynamically the CSV report url

* Fix Docker image name the hook

* Implement component check number 5

Get the meaningful components for the last 6 months

* Handle reviews comments

* Remove extraneous print

* Removes TODO

* Use a different threshold ration when checking for new meaningful components

As we are only checking new bugs for 6 months, adjust the threshold ration to
be less sensitive to occasional burst ob bugs for q given component.

* Reduce the threshold ratio

As we check on a disjoint time window, reduce the chance of false positives

* Handle review nits

* Fix last nits
2019-05-10 12:20:23 +02:00
Boris Feld 8eb5bcab92 Make docker push depends on test tasks (#363) 2019-05-09 11:40:59 +02:00
Boris Feld f68535c055 Update the hook before triggering the route (#362) 2019-05-09 11:27:12 +02:00
Marco Castelluccio 17c3e06d94 Remove undefined dependency on rollback_test_task
Fixes #348
2019-05-07 11:53:20 +02:00
Boris Feld 6937e0e5e8 Add the rollback test in the data pipeline (#337)
Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 14:20:43 +02:00
Boris Feld f3ff723cbf Fix docker push task command (#334) 2019-05-02 23:46:01 +02:00
Boris Feld 297963e4ce Skip checking models while building the http service image, and only push it as part of the pipeline (#331)
* Add a way to skip checking models while building the http service image

* Don't push the http service on release

It isn't built with the real models on release

* Use taskboot 0.1.1
2019-05-02 23:18:51 +02:00
Boris Feld 369b44ea02 Update the index URLs in bugbug (#328)
* Update the index URLs in bugbug

* Split the http service Docker image in two

This way we can both:
- Build the first half (code + dependencies) in the usual CI.
- Build the second half at the end of the data pipeline with updated models.

Taskboot build-compose doesn't support building all services except a
specific one and it might be cumbersome to add this feature so move the second
half of the Docker image to a separate docker-compose file.
2019-05-02 17:00:32 +02:00
Bastien Abadie bd21382a3b Use mozilla/taskboot 0.1.0 (#323)
Fixes #322
2019-04-29 18:08:45 +02:00
Boris Feld 1ae2200682 Create the initial data pipeline task definition (#313)
* Create the data pipeline task definition

There is a hook (which runs every day at midnight) that will spawn the
data-pipeline using the latest docker images that were build on latest
release. The hook itself is updated on each release and is versionned in this
repository.

The hook will runs once every week and on every successful release.

Add task for building the Docker image for HTTP service.
2019-04-26 14:56:49 +02:00
Bastien Abadie da7c402089 Use Taskcluster cache when building docker image (#311) 2019-04-26 13:06:15 +02:00
Marco 6c29fc20dc
Increase maximum runtime and deadline for docker build task (#299) 2019-04-19 23:41:55 +02:00
Boris Feld 053954d70b Run pre-commit in the lint task (#297) 2019-04-19 18:01:24 +02:00
Boris Feld ded92357ee Add a new tasks for updating bugbug hook (#289)
The hook will be responsible for starting the data pipeline, but right now is
doing mostly nothing.

Use taskboot new command `build-hook` to update the hook based on the
versioned definition file.
2019-04-18 21:20:04 +02:00
Bastien Abadie 17ccdf7545 Use taskboot push-artifact (#290) 2019-04-18 19:49:12 +02:00
Marco d27d5f8b2c
Add a mock DB for tests to avoid downloading the full DB (#273)
* Rename test function in test_bug to reflect reality, and add more assertions

* Add a mock bugs DB for tests

* Don't download bugs DB anymore for running tests

* Add a test for run.py basic functionality

* Remove training test task, as the test is now a pytest
2019-04-18 14:01:25 +02:00
Shubham Garg b790ac8278 Make infra/version_check.py only run on branches and tags, not PRs (#277) 2019-04-18 12:49:00 +02:00
Bastien Abadie 7af2459c96 Use mozilla/taskboot (#287) 2019-04-17 13:24:46 +02:00
Bastien Abadie b5153abc89 Docker full build using compose (#274) 2019-04-15 17:36:30 +02:00
Bastien Abadie 32ac9eb5d6 Add Docker build task for the base image (#270) 2019-04-11 14:00:18 +02:00
Marco ce57a4f806
Revert "Add Docker build task for the base image (#265)" (#269)
This reverts commit f6d6a0867d.
2019-04-11 00:32:04 +02:00
Bastien Abadie f6d6a0867d Add Docker build task for the base image (#265) 2019-04-11 00:24:02 +02:00
Marco Castelluccio 31f371814e Deploy bugbug package to PyPI on new tags 2019-04-10 17:32:20 +02:00
Marco b8240588e1
Run linting and tests in parallel on Taskcluster (#260) 2019-04-09 22:51:54 +02:00
Marco 8155c4d350
Use Taskcluster instead of Travis (#250) 2019-04-09 18:11:47 +02:00