Граф коммитов

301 Коммитов

Автор SHA1 Сообщение Дата
Boris Feld afd67402e2 Fix copy-paste typo with the new indexing schema (#801) 2019-07-28 20:38:05 +02:00
Boris Feld a43ad03b2a Add a new indexing schema for training tasks (#795)
In order to efficiently solve #614, we need a new indexing schema
so getting all metrics following a given date is easy.
2019-07-26 18:28:04 +02:00
Marco Castelluccio 60a979be9d Store commits to ignore in a bugbug DB and generate them progressively
In the future, we will be able to get commits to ignore directly from the normal commits DB
generated by bugbug/repositor.py.
2019-07-26 18:14:57 +02:00
Marco Castelluccio 38db5e19dc Fix regressor-finder artifact paths 2019-07-26 16:09:50 +02:00
Marco Castelluccio adb39df6bb Switch to a version-control-tools version fixing an error with the 'bug' mercurial template 2019-07-25 23:56:38 +02:00
Marco Castelluccio f8a5d8f97f Fix paths of the artifacts for the regressor-finder task 2019-07-25 15:56:23 +02:00
Marco Castelluccio 93a2ac8425 Apply a patch to speed-up git hyper-blame and fix a bug due to colliding short hashes 2019-07-25 01:25:05 +02:00
Marco Castelluccio a614d34735 Move download of bugs linked to commits in the bug-retriever script
Also, make the bug-retriever task depend on the commit-retriever one, making the
download of bugs linked to commits actually work :)
2019-07-25 01:05:25 +02:00
Marco Castelluccio 5ef2a5039f Upload version files as artifacts too
Plus the tokenized bug introducing commits
2019-07-24 22:15:04 +02:00
Marco Castelluccio 22d73e3637 Apply regressor finder also on the microannotated repository with comments removed
Fixes #627
2019-07-24 22:15:04 +02:00
Marco Castelluccio 839ebf8fcf Make git repo URL a parameter, so we can find regressors using different git repositories 2019-07-24 21:01:53 +02:00
dependabot-preview[bot] e2fa9b63e6 Bump taskcluster from 14.3.1 to 15.0.0 (#774)
Bump [taskcluster](https://github.com/taskcluster/taskcluster) from 14.3.1 to 15.0.0.
- [Release notes](https://github.com/taskcluster/taskcluster/releases)
- [Changelog](https://github.com/taskcluster/taskcluster/blob/master/CHANGELOG.md)
- [Commits](https://github.com/taskcluster/taskcluster/compare/v14.3.1...v15.0.0)
2019-07-24 11:14:39 +02:00
Marco Castelluccio 63c967f9a0 Move regressor-finder task after the microannotate ones 2019-07-24 10:59:45 +02:00
Marco Castelluccio 61ab656795 Add regressor-finder to run-again's dependencies 2019-07-24 10:56:43 +02:00
Marco Castelluccio 5b8ac293b9 Use releng-svc-compute for the regressor finder script 2019-07-24 01:46:33 +02:00
Marco Castelluccio d245c9e998 Use rust-code-analysis v0.0.2 2019-07-23 13:37:27 +02:00
Marco Castelluccio 1e35bc3605 Disable detached head warning 2019-07-23 12:26:38 +02:00
Marco Castelluccio 539dbc223f Remove some unused directories of version-control-tools and depot_tools 2019-07-23 12:26:12 +02:00
Marco Castelluccio f28fc651d0 Install rust-code-analysis in the commit retrieval image 2019-07-23 12:16:13 +02:00
Marco Castelluccio debf7d4e02 Add all microannotate-* tasks as dependencies of run-again 2019-07-23 10:51:21 +02:00
Marco Castelluccio a03e9e8df1 Fix mirror repository URL 2019-07-23 10:50:41 +02:00
Marco Castelluccio 7bd391e307 Generate mirror repositories with comments removed and both tokenized and with comments removed 2019-07-23 02:14:22 +02:00
Marco Castelluccio ab048e0a6b Support generating mirror repositories with comments removed 2019-07-23 02:14:22 +02:00
Marco Castelluccio fbaef0661d Store regressor finder results in bugbug DBs and make it run only on commits which haven't been analyzed yet 2019-07-23 02:14:22 +02:00
Marco Castelluccio 7608dd56cf Make git-hyper-blame run with Python 2 2019-07-22 23:20:01 +02:00
Marco Castelluccio 453bd8125f Checkout to a specific revision of depot_tools, to make everything reproducible 2019-07-22 23:20:01 +02:00
Marco Castelluccio b5d67d156a Add depot_tools path at the end of PATH 2019-07-22 23:20:01 +02:00
Marco 77ec8b529d
Add a WIP script to find bug-introducing commits (#748)
* Install depot_tools in the commit retrieval image

* Add a WIP script to find bug-introducing commits

* Add a task which runs the bug-introducing commits finder script
2019-07-22 14:41:34 +02:00
dependabot-preview[bot] 660935c0db Bump taskcluster from 7.0.1 to 14.3.1 (#763)
Bump [taskcluster](https://github.com/taskcluster/taskcluster) from 7.0.1 to 14.3.1.
- [Release notes](https://github.com/taskcluster/taskcluster/releases)
- [Changelog](https://github.com/taskcluster/taskcluster/blob/master/CHANGELOG.md)
- [Commits](https://github.com/taskcluster/taskcluster/commits/v14.3.1)
2019-07-22 12:28:28 +02:00
Marco Castelluccio 830d88b1eb Disable the cache when installing bugbug 2019-07-16 20:48:48 +02:00
Marco Castelluccio e858728229 Remove version-control-tools Mercurial directory, as we don't need to perform any mercurial operation on it 2019-07-16 20:48:48 +02:00
Marco Castelluccio 8de0987369 Remove secrets:get:* scopes from hook, as they are implied by the roles 2019-07-15 22:31:30 +02:00
Marco Castelluccio 868e65fe32 Add scope to trigger bugbug-annotate hook to the 'run-again' task 2019-07-15 22:05:43 +02:00
Marco Castelluccio 9b0e9e8c90 Retrigger when not done, not when done 2019-07-15 22:03:39 +02:00
Marco Castelluccio 7fbfc41325 Consider tasks which don't have a 'done' artifact as fully done 2019-07-15 22:03:39 +02:00
Marco Castelluccio 085d157af4 Fix 'done' artifact path 2019-07-15 22:01:52 +02:00
Marco Castelluccio 7abab83654 Add taskclusterProxy feature for the 'run-again' task 2019-07-15 21:41:50 +02:00
Marco Castelluccio 66367584cd Revert "Enable feature importance calculation for the defect/enhancement/task model"
This reverts commit d9cdcdc238.

It's running out of memory on releng-svc-compute workers (c5.4xlarge), so we need to temporarily disable it.
2019-07-15 15:49:28 +02:00
Marco f877420959
Retrigger microannotate hook if the generation process is not fully done (#700)
* Generate an artifact specifying if the microannotate generation is fully done

* Retrigger microannotate hook if the generation process is not fully done

Fixes #652

* Update to microannotate 0.0.2
2019-07-15 14:01:56 +02:00
Anurag Aggarwal 656d6e844b Remove bugs_retrieval image and use the base image instead in its place (#691)
* Fixes #633
2019-07-12 14:17:41 +02:00
Marco Castelluccio d9cdcdc238 Enable feature importance calculation for the defect/enhancement/task model 2019-07-11 20:44:07 +02:00
Marco d426d5e66e
Revert "Revert "Remove pip from the commit retrieval image (#676)" (#686)" (#687)
This reverts commit e84c248e19.
2019-07-11 02:02:04 +02:00
Marco e84c248e19
Revert "Remove pip from the commit retrieval image (#676)" (#686)
This reverts commit e7ed31940c.
2019-07-10 16:52:04 +02:00
Marco Castelluccio 9a53baf7ec Upload importances.json files as an artifact 2019-07-10 16:39:48 +02:00
Marco Castelluccio 17b027c767 Enable feature importance calculation at training time for the regressor model 2019-07-10 16:25:38 +02:00
Davide Monfrecola e7ed31940c Remove pip from the commit retrieval image (#676)
Fixes #647
2019-07-09 18:32:57 +02:00
Marco Castelluccio 261bff6ab0 Rename public/probs,json artifact to public/probs.json 2019-07-09 14:41:03 +02:00
Boris Feld e7add98563 Update task-boot to 0.1.9 (#675) 2019-07-05 15:36:16 +02:00
Marco Castelluccio d08e062c68 Improve triggerSchema of the hooks and test it 2019-07-03 17:26:01 +02:00
Marco Castelluccio aac6326ba5 Index classify-patch task in the Taskcluster index by diff ID
Fixes #668
2019-07-03 15:46:35 +02:00
Marco Castelluccio 28cb2b291e Insert the Phabricator diff ID in the command directly rather than in the env 2019-07-03 15:46:21 +02:00
Marco Castelluccio 3315532904 Use mozilla/bugbug-commit-retrieval image for the classify-patch hook
Since we need to use Mercurial
2019-07-03 10:40:02 +02:00
Marco Castelluccio 9de3aa70f2 Add TC_SECRET_ID to the env of the classify-patch hook 2019-07-03 10:39:37 +02:00
Marco Castelluccio 7e6cf9cf2e Set version of other bugbug images too 2019-07-02 21:04:23 +02:00
Marco Castelluccio cb5f54f89f Use bash to resolve env variable 2019-07-02 21:04:23 +02:00
Marco Castelluccio 71ebbae38a Handle cases where the hook definition already contains something in the env
Also refactor the file to make it more easily testable
2019-07-02 20:52:57 +02:00
Marco Castelluccio 8e23bb7f8a Add a hook to classify a Phabricator diff 2019-07-02 19:39:59 +02:00
Marco Castelluccio 6ce18762de 'payload.command' should be an array 2019-07-02 13:26:46 +02:00
Marco Castelluccio d12a25f644 Upload feature visualization image as an artifact of the training tasks 2019-07-01 13:10:39 +02:00
Boris Feld 7459f79317 Use the base image for training models (#656)
Fixes #350
2019-06-29 00:01:51 +02:00
Boris Feld d24993d0ac Remove dependency on rollbacktest in docker build. (#653)
Fixes #651
2019-06-28 15:32:39 +02:00
Boris Feld 2977a8af89 Start tracking the decision task of the data pipeline (#649)
This way we can automate the stop of a running data pipeline during the
rollback process.
2019-06-28 15:32:04 +02:00
Boris Feld 54e41d1497 Use taskboot 0.1.8 (#645)
The new taskboot release solves the double build on non-tag commits and
allows the heroku deploy to be fully atomic.
2019-06-28 11:11:48 +02:00
Boris Feld 2b48e153cb Use tagged version in bugbug hooks (#640)
Fixes #634
2019-06-27 15:47:57 +02:00
Marco Castelluccio eb8f847d43 Install 'git' package in the commit_retrieval image
Even though it's only required for the annotate pipeline.
2019-06-27 01:25:58 +02:00
Marco Castelluccio f65bf19ab8 Disable pip cache when installing Mercurial 2019-06-27 01:25:22 +02:00
Marco Castelluccio afac5cbdb6 Remove apt lists after installing Python 2 2019-06-27 01:24:59 +02:00
Marco Castelluccio ad67d76992 Use bugbug-commit-retrieval image for the microannotate-generate task, as need Mercurial to be installed 2019-06-27 01:14:32 +02:00
Marco Castelluccio 4a2759e9b0 Add TC_SECRET_ID env variable to microannotate-generate task 2019-06-27 01:13:51 +02:00
Marco Castelluccio 8fd3be99ce Command must be split in two lines 2019-06-26 22:28:40 +02:00
Marco Castelluccio 252e29398d command should be an array 2019-06-26 21:01:31 +02:00
Marco Castelluccio 4b56be37a9 Add missing secrets scope to annotate hook 2019-06-26 19:45:44 +02:00
Marco Castelluccio b01eeafe44 Add annotate-pipeline.yml file to the spawn_pipeline Docker image 2019-06-26 19:45:22 +02:00
Marco Castelluccio ea7a009450 Fix schedule value 2019-06-26 19:40:19 +02:00
Marco Castelluccio 56f224b9dc Generate microannotate repository for mozilla-central 2019-06-26 18:57:36 +02:00
x249wang ab28e8ace2 Use zstandard instead of xz (#524)
Fixes #461.
2019-06-24 13:16:44 +02:00
Boris Feld 9834053a36 Start tracking training metrics as Taskcluster artifacts (#604)
Fixes #342
2019-06-22 14:18:08 -07:00
Boris Feld 27f9104fb5 Make sure the Docker build task uses the tagged code (#610)
If not, new master code might get released and conflict with the code in the
bugbug images.

 Fixes #609
2019-06-21 08:20:08 -07:00
Boris Feld 417f2aae69 Use a single spawn image for both hooks (#594)
This removes the need to keep two different Docker images for spawning the
data pipeline and the check pipeline.

Fixes #365
2019-06-14 18:54:21 +02:00
Boris Feld c06db28442 Bump taskboot to version 1.0.7 (#583)
Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update
task-boot version to use it.

Also add missing tags and cache option when building Docker images in
data-pipeline.yml
2019-06-12 20:11:34 +02:00
Marco Castelluccio 89b37b96ae Upload version file too in the bugs retrieval task 2019-06-09 00:13:20 +02:00
Marco Castelluccio 353d21d01b Clone repository quietly 2019-06-08 11:19:01 +02:00
Marco Castelluccio 4a991ac6ef Fix download of bugs DB in the rollback test 2019-06-08 11:17:15 +02:00
Marco Castelluccio 76b82d0427 Load yaml safely
This is mostly to avoid warnings, as the yaml files we load are fully controlled by us.
2019-06-07 23:34:14 +02:00
Marco Castelluccio 9de91456f6 Update to taskboot 0.1.6 2019-06-07 22:03:00 +02:00
Boris Feld a8faa48d8a Support classifying batches of bugs with a background worker (#321) 2019-06-07 21:22:14 +02:00
Marco Castelluccio 82d9c0ece0 Update to taskboot 0.1.5 2019-06-07 16:47:28 +02:00
Boris Feld 2e05e57be2 Build docker images data pipeline tag (#566)
* Build the HTTP Docker image with the right tag

* Ensure the builded docker image has the right parent image
2019-06-07 16:46:05 +02:00
Boris Feld 2988700028 Use tagged index urls for pushing artifacts (#561)
* Use tagged index urls for pushing artifacts

Also replace previous code that updated Docker image tag to use JSON-e
templating instead.
2019-06-07 12:52:29 +02:00
Boris Feld 7906380e6f Bump version of taskboot to use latest version of img tool (#562)
It is necessary to support mulit-tag Docker image building
2019-06-07 12:21:09 +02:00
Boris Feld e0accae208 Move string formatting to f-string in spawn_data_pipeline (#559) 2019-06-07 11:04:33 +02:00
pyup.io bot c590278bff Update pyyaml from 5.1 to 5.1.1 (#560) 2019-06-07 10:56:26 +02:00
Boris Feld 5a31c99ac9 Add support for specific Docker tag in spawn_data_pipeline.py (#553)
* Revert "Revert "Add support for specific Docker tag in spawn_data_pipeline.py (#489)" (#499)"

This reverts commit 249ed40eb6.

* Ignore task with a tagged docker image

* Restrict Docker tag update to bugbug related images
2019-06-06 19:14:27 +02:00
Boris Feld 32f56a3962 Add a script to update the hook definition with the TAG during release (#507)
Fixes #501, fixed relanding of #491.
2019-06-06 18:11:59 +02:00
Marco Castelluccio f5951ad63a Support retrieving some label files at runtime, and do it for the regressor labels 2019-06-05 00:37:26 +02:00
Marco Castelluccio 7790f5e3d5 Use raw CSV file, not GitHub's HTML page 2019-06-04 13:08:24 +02:00
Marco Castelluccio d57177f1e4 Fix destination path of the regressor.csv label file 2019-06-04 13:07:59 +02:00
Marco Castelluccio 44e26ff0e8 Add a training task for the Regressor model 2019-06-03 22:15:18 +02:00
Marco Castelluccio 2804436357 Download regressor labels from marco-c/mozilla-central-regressors repository in the train_regressor Docker image 2019-06-03 22:14:47 +02:00
Marco Castelluccio 72ddfea2e3 Add a Docker image for the task to train the Regressor model 2019-06-03 21:46:35 +02:00
Marco Castelluccio 4ce438a35a Fix typo in artifact name for the commits retrieval task 2019-06-03 21:37:39 +02:00
Marco d8b84ca798
Support retrieving commits in steps (#536)
* Support retrieving commits in steps

* Store component mapping ETag to actually avoid downloading it again when not needed

* Store a version file alongside the DBs

* Export the commits DB version file and the experiences values as artifacts of the commit-retriever task
2019-06-03 19:29:08 +02:00
Marco Castelluccio e62dd6f37d Make rollback-test task verbose 2019-06-03 11:06:32 +02:00
Ayush Shridhar 9d71677667 Add a training task for the Duplicate model (#525) 2019-05-31 17:05:58 +02:00
Marco Castelluccio bd3e4c7900 Increase the maximum runtime for the commits retrieval task 2019-05-30 13:27:23 +02:00
Marco Castelluccio 42d2ff2db8 Add a training task for the Backout model 2019-05-30 13:27:06 +02:00
Marco Castelluccio cd0405d223 Add a Docker image for the Backout model 2019-05-30 13:24:55 +02:00
Boris Feld 6ee9fb57f0 Fix Docker build by downloading the models inside the image. Fix #504 (#516)
The data pipeline failed before because it tried downloading the model from
outside the Docker image and didn't had bugbug installed.

The clean way of solving this would be to build a base http service image on
release and build another one where we simply download the models but let's
fix it this way for now.
2019-05-29 20:43:58 +02:00
Marco Castelluccio 0037ad35b5 Revert "Add a script to update the hook definition with the TAG during release (#491)"
This reverts commit 6ef95cfec3.
2019-05-25 00:16:41 +02:00
Marco 249ed40eb6
Revert "Add support for specific Docker tag in spawn_data_pipeline.py (#489)" (#499)
This reverts commit 6e7c815f13.
2019-05-24 17:30:24 +02:00
Boris Feld 6ef95cfec3 Add a script to update the hook definition with the TAG during release (#491) 2019-05-24 15:13:52 +02:00
Boris Feld 6e7c815f13 Add support for specific Docker tag in spawn_data_pipeline.py (#489) 2019-05-24 12:47:20 +02:00
Boris Feld 1bae5834ab Implement deployment to Heroku (#458) 2019-05-23 20:39:02 +02:00
Ayush Shridhar b41170baa5 Add training task for the StepsToReproduce model (#441) 2019-05-22 21:43:11 +02:00
Ayush Shridhar 91bf939fb7 Add training task for the RegressionRange model (#466) 2019-05-22 18:58:47 +02:00
Marco 025e3f4da2
Fix command to train defect/enhancement/task model (#476)
* Fix command to train defect/enhancement/task model

Fixes #475

* Add more logging in the trainer script, and assert the model is generated
2019-05-21 14:46:57 +02:00
Marco 0093f8c10f
Fix SHOULD_DOWNLOAD_MODEL environment variable name (#469)
Fixes #468
2019-05-20 15:24:33 +02:00
Boris Feld dd00d7b9ec Add the support for downloading the model before checking it (#452)
Also put the right configuration in the check pipeline
2019-05-17 11:45:42 +02:00
pyup.io bot f8914dd59d Update requests to 2.22.0 (#448) 2019-05-16 17:31:46 +02:00
Marco 8a5795417a
Add a pre-commit hook using codespell (#411)
* Add a pre-commit hook using codespell

Fixes #410

* Fix some spelling mistakes
2019-05-16 17:24:18 +02:00
Boris Feld d3c3bcbece Bump version of taskboot used in taskcluster and data pipeline (#446) 2019-05-16 13:02:58 +02:00
Marco Castelluccio 52f00714e9 No need for big instance types for the check component model task 2019-05-15 12:03:12 +02:00
Marco Castelluccio 3719540351 Change role for the checks hook 2019-05-15 11:54:02 +02:00
Marco Castelluccio ff9ea35ed0 Reduce deadlines to maximum of 5 days
Taskcluster only allows up to 5 days
2019-05-14 20:39:00 +02:00
Marco Castelluccio 350ddbc1e9 Remove bad trailing comma in infra/taskcluster-hook-check-models-start.json
Fixes #413
2019-05-14 20:30:56 +02:00
Marco 9223954520
Remove training tasks' unneeded dependencies on commit retrieval task (#407)
Fixes #390
2019-05-14 15:22:44 +02:00
Marco c4bd01278e
Add 'expires' to all tasks to avoid them expiring in a too long time (#393)
Fixes #391.
2019-05-12 21:46:58 +02:00
Marco e3230ca999
Increase deadline of data pipeline tasks (#389)
Fixes #388.
2019-05-10 16:12:46 +02:00
Boris Feld f4b2b938be
Add basic check method and check script (#341)
* Add basic check method and check script

* Ensure the check of component will correctly use super result

* Add required infra to schedule model checks

* Add scheduling bits for the model checks

* Remove the filtering on classification

* Extract counting bugs to a new function in bugzilla.py

* Also checks conflated components

* Fix new hook id

* Call bugzilla with the count_only param to speed up the check

* Fix the new hook scope to match the hook id

* Fix component model check after previous refactoring

* Fix component model check method

* Use a bugzilla report for even faster component model check

* Clarify get_product_component_count docstring

We are already filtering out full component with 0 bugs

* Update conflated components mapping check

A conflated component could also be part of the conflated components mapping

* Distinguish between non-existing full components and empty full components

* Remove the filter on resolution and unnecessary url params

* Update component check method

Keep checks as separate as possible for clarity, we could merge them or makes
them faster later

* Generate dynamically the CSV report url

* Fix Docker image name the hook

* Implement component check number 5

Get the meaningful components for the last 6 months

* Handle reviews comments

* Remove extraneous print

* Removes TODO

* Use a different threshold ration when checking for new meaningful components

As we are only checking new bugs for 6 months, adjust the threshold ration to
be less sensitive to occasional burst ob bugs for q given component.

* Reduce the threshold ratio

As we check on a disjoint time window, reduce the chance of false positives

* Handle review nits

* Fix last nits
2019-05-10 12:20:23 +02:00
Marco 6f09488573
Rename mozilla/bugbug-train-defect image to mozilla/bugbug-train-defectenhancementtask (#375)
Fixes #364.
2019-05-09 23:36:38 +02:00
Boris Feld 4954fe38d4 Change scheduling of the data pipeline to run once every two weeks (#361)
Fixes #356.
2019-05-09 11:44:53 +02:00
Marco Castelluccio c3f55e682a Rename train-defect to train-defectenhancementtask 2019-05-07 13:16:22 +02:00
Marco Castelluccio 2eaf90be20 Add a cache to the commit retrieval task
Fixes #347
2019-05-07 11:38:02 +02:00
Boris Feld 6937e0e5e8 Add the rollback test in the data pipeline (#337)
Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 14:20:43 +02:00
Marco 9995b8c236
Make training code more generic to make it possible to train on other kinds of objects (e.g. commits) (#335)
* Move feature cleanup functions in a separate module

As they can be shared for different objectives, e.g. both training on bugs and on commits.

* Make Model more generic to make it possible to train on different objects

Introduce BugModel and CommitModel, as base classes for models training on bugs and on commits.

Update all models to use BugModel and to use the new feature_cleanup module.

Fixes #306.

* Update ID and description of the defect/enhancement/task Taskcluster task definition

* Add a module to extract features from commit data

* Add an example model training on commits to predict commits which will be backed out

* Update defect model name, and add possibility to train backout model
2019-05-03 11:57:48 +02:00
Boris Feld 297963e4ce Skip checking models while building the http service image, and only push it as part of the pipeline (#331)
* Add a way to skip checking models while building the http service image

* Don't push the http service on release

It isn't built with the real models on release

* Use taskboot 0.1.1
2019-05-02 23:18:51 +02:00
Boris Feld 369b44ea02 Update the index URLs in bugbug (#328)
* Update the index URLs in bugbug

* Split the http service Docker image in two

This way we can both:
- Build the first half (code + dependencies) in the usual CI.
- Build the second half at the end of the data pipeline with updated models.

Taskboot build-compose doesn't support building all services except a
specific one and it might be cumbersome to add this feature so move the second
half of the Docker image to a separate docker-compose file.
2019-05-02 17:00:32 +02:00
Boris Feld 8c633b6da1 Remove the experimental Dockerfile syntax (#330)
This way we can still build locally with docker-compose. We don't use the experimental syntax anymore.
2019-05-02 16:01:48 +02:00
Boris Feld 6e7ca892cd Introduce a new Docker image for data-pipeline spawning (#320) 2019-05-02 14:36:50 +02:00
Marco f4b096519f
Don't use pip cache directory, and remove downloaded apt lists (#314) 2019-04-26 16:14:52 +02:00
Boris Feld 1ae2200682 Create the initial data pipeline task definition (#313)
* Create the data pipeline task definition

There is a hook (which runs every day at midnight) that will spawn the
data-pipeline using the latest docker images that were build on latest
release. The hook itself is updated on each release and is versionned in this
repository.

The hook will runs once every week and on every successful release.

Add task for building the Docker image for HTTP service.
2019-04-26 14:56:49 +02:00
Bastien Abadie 87a5674a9c Setup bugbug with bind mount 2019-04-25 13:08:26 +02:00
Bastien Abadie fe01e45dcd Switch to slim image 2019-04-25 11:15:34 +02:00
Marco Castelluccio e199972050 Actually use f-string in version check script 2019-04-19 19:13:28 +02:00
Boris Feld 053954d70b Run pre-commit in the lint task (#297) 2019-04-19 18:01:24 +02:00
Boris Feld ded92357ee Add a new tasks for updating bugbug hook (#289)
The hook will be responsible for starting the data pipeline, but right now is
doing mostly nothing.

Use taskboot new command `build-hook` to update the hook based on the
versioned definition file.
2019-04-18 21:20:04 +02:00
Bastien Abadie b5153abc89 Docker full build using compose (#274) 2019-04-15 17:36:30 +02:00
Boris Feld d899ebc068 Add missing Dockerfiles for the training steps (#276) 2019-04-12 12:31:13 +02:00
Marco Castelluccio d0f98674b8 Be more verbose when the version check command fails 2019-04-11 00:40:11 +02:00
Marco Castelluccio 31f371814e Deploy bugbug package to PyPI on new tags 2019-04-10 17:32:20 +02:00
Boris Feld b651744b18 Import retriever services and add Docker image definition (#251)
* Import Retriever class from release-services repository

This basically import the `retriever.py` file from the `release-services`
repository at hash 77cdddd. I removed imports and reference to cli-common
helpers that will likely needs to be reimplemented, like the raven support.

The next commit will defines some Dockerfiles that will use the imported file.

* Add docker image definition

Build three Docker image, one is for bugbug itself. It is just installing
bugbug and its dependencies.

One is for retrieving information from the mozilla-central Mercurial
repository, it depends on the first one and install the right Mercurial
version.

The last one is for retrieving information from the Bugzilla instance, it
depends in the first one and needs a valid Bugzilla token.

* Separate the two tasks into separate script files

They share almost no code at all so they don't need to be in the same file

* Apply Black on the scripts to makes Flake8 happy
2019-04-09 16:30:09 +02:00