Boris Feld
afd67402e2
Fix copy-paste typo with the new indexing schema ( #801 )
2019-07-28 20:38:05 +02:00
Boris Feld
a43ad03b2a
Add a new indexing schema for training tasks ( #795 )
...
In order to efficiently solve #614 , we need a new indexing schema
so getting all metrics following a given date is easy.
2019-07-26 18:28:04 +02:00
Marco Castelluccio
60a979be9d
Store commits to ignore in a bugbug DB and generate them progressively
...
In the future, we will be able to get commits to ignore directly from the normal commits DB
generated by bugbug/repositor.py.
2019-07-26 18:14:57 +02:00
Marco Castelluccio
38db5e19dc
Fix regressor-finder artifact paths
2019-07-26 16:09:50 +02:00
Marco Castelluccio
adb39df6bb
Switch to a version-control-tools version fixing an error with the 'bug' mercurial template
2019-07-25 23:56:38 +02:00
Marco Castelluccio
f8a5d8f97f
Fix paths of the artifacts for the regressor-finder task
2019-07-25 15:56:23 +02:00
Marco Castelluccio
93a2ac8425
Apply a patch to speed-up git hyper-blame and fix a bug due to colliding short hashes
2019-07-25 01:25:05 +02:00
Marco Castelluccio
a614d34735
Move download of bugs linked to commits in the bug-retriever script
...
Also, make the bug-retriever task depend on the commit-retriever one, making the
download of bugs linked to commits actually work :)
2019-07-25 01:05:25 +02:00
Marco Castelluccio
5ef2a5039f
Upload version files as artifacts too
...
Plus the tokenized bug introducing commits
2019-07-24 22:15:04 +02:00
Marco Castelluccio
22d73e3637
Apply regressor finder also on the microannotated repository with comments removed
...
Fixes #627
2019-07-24 22:15:04 +02:00
Marco Castelluccio
839ebf8fcf
Make git repo URL a parameter, so we can find regressors using different git repositories
2019-07-24 21:01:53 +02:00
dependabot-preview[bot]
e2fa9b63e6
Bump taskcluster from 14.3.1 to 15.0.0 ( #774 )
...
Bump [taskcluster](https://github.com/taskcluster/taskcluster ) from 14.3.1 to 15.0.0.
- [Release notes](https://github.com/taskcluster/taskcluster/releases )
- [Changelog](https://github.com/taskcluster/taskcluster/blob/master/CHANGELOG.md )
- [Commits](https://github.com/taskcluster/taskcluster/compare/v14.3.1...v15.0.0 )
2019-07-24 11:14:39 +02:00
Marco Castelluccio
63c967f9a0
Move regressor-finder task after the microannotate ones
2019-07-24 10:59:45 +02:00
Marco Castelluccio
61ab656795
Add regressor-finder to run-again's dependencies
2019-07-24 10:56:43 +02:00
Marco Castelluccio
5b8ac293b9
Use releng-svc-compute for the regressor finder script
2019-07-24 01:46:33 +02:00
Marco Castelluccio
d245c9e998
Use rust-code-analysis v0.0.2
2019-07-23 13:37:27 +02:00
Marco Castelluccio
1e35bc3605
Disable detached head warning
2019-07-23 12:26:38 +02:00
Marco Castelluccio
539dbc223f
Remove some unused directories of version-control-tools and depot_tools
2019-07-23 12:26:12 +02:00
Marco Castelluccio
f28fc651d0
Install rust-code-analysis in the commit retrieval image
2019-07-23 12:16:13 +02:00
Marco Castelluccio
debf7d4e02
Add all microannotate-* tasks as dependencies of run-again
2019-07-23 10:51:21 +02:00
Marco Castelluccio
a03e9e8df1
Fix mirror repository URL
2019-07-23 10:50:41 +02:00
Marco Castelluccio
7bd391e307
Generate mirror repositories with comments removed and both tokenized and with comments removed
2019-07-23 02:14:22 +02:00
Marco Castelluccio
ab048e0a6b
Support generating mirror repositories with comments removed
2019-07-23 02:14:22 +02:00
Marco Castelluccio
fbaef0661d
Store regressor finder results in bugbug DBs and make it run only on commits which haven't been analyzed yet
2019-07-23 02:14:22 +02:00
Marco Castelluccio
7608dd56cf
Make git-hyper-blame run with Python 2
2019-07-22 23:20:01 +02:00
Marco Castelluccio
453bd8125f
Checkout to a specific revision of depot_tools, to make everything reproducible
2019-07-22 23:20:01 +02:00
Marco Castelluccio
b5d67d156a
Add depot_tools path at the end of PATH
2019-07-22 23:20:01 +02:00
Marco
77ec8b529d
Add a WIP script to find bug-introducing commits ( #748 )
...
* Install depot_tools in the commit retrieval image
* Add a WIP script to find bug-introducing commits
* Add a task which runs the bug-introducing commits finder script
2019-07-22 14:41:34 +02:00
dependabot-preview[bot]
660935c0db
Bump taskcluster from 7.0.1 to 14.3.1 ( #763 )
...
Bump [taskcluster](https://github.com/taskcluster/taskcluster ) from 7.0.1 to 14.3.1.
- [Release notes](https://github.com/taskcluster/taskcluster/releases )
- [Changelog](https://github.com/taskcluster/taskcluster/blob/master/CHANGELOG.md )
- [Commits](https://github.com/taskcluster/taskcluster/commits/v14.3.1 )
2019-07-22 12:28:28 +02:00
Marco Castelluccio
830d88b1eb
Disable the cache when installing bugbug
2019-07-16 20:48:48 +02:00
Marco Castelluccio
e858728229
Remove version-control-tools Mercurial directory, as we don't need to perform any mercurial operation on it
2019-07-16 20:48:48 +02:00
Marco Castelluccio
8de0987369
Remove secrets:get:* scopes from hook, as they are implied by the roles
2019-07-15 22:31:30 +02:00
Marco Castelluccio
868e65fe32
Add scope to trigger bugbug-annotate hook to the 'run-again' task
2019-07-15 22:05:43 +02:00
Marco Castelluccio
9b0e9e8c90
Retrigger when not done, not when done
2019-07-15 22:03:39 +02:00
Marco Castelluccio
7fbfc41325
Consider tasks which don't have a 'done' artifact as fully done
2019-07-15 22:03:39 +02:00
Marco Castelluccio
085d157af4
Fix 'done' artifact path
2019-07-15 22:01:52 +02:00
Marco Castelluccio
7abab83654
Add taskclusterProxy feature for the 'run-again' task
2019-07-15 21:41:50 +02:00
Marco Castelluccio
66367584cd
Revert "Enable feature importance calculation for the defect/enhancement/task model"
...
This reverts commit d9cdcdc238
.
It's running out of memory on releng-svc-compute workers (c5.4xlarge), so we need to temporarily disable it.
2019-07-15 15:49:28 +02:00
Marco
f877420959
Retrigger microannotate hook if the generation process is not fully done ( #700 )
...
* Generate an artifact specifying if the microannotate generation is fully done
* Retrigger microannotate hook if the generation process is not fully done
Fixes #652
* Update to microannotate 0.0.2
2019-07-15 14:01:56 +02:00
Anurag Aggarwal
656d6e844b
Remove bugs_retrieval image and use the base image instead in its place ( #691 )
...
* Fixes #633
2019-07-12 14:17:41 +02:00
Marco Castelluccio
d9cdcdc238
Enable feature importance calculation for the defect/enhancement/task model
2019-07-11 20:44:07 +02:00
Marco
d426d5e66e
Revert "Revert "Remove pip from the commit retrieval image ( #676 )" ( #686 )" ( #687 )
...
This reverts commit e84c248e19
.
2019-07-11 02:02:04 +02:00
Marco
e84c248e19
Revert "Remove pip from the commit retrieval image ( #676 )" ( #686 )
...
This reverts commit e7ed31940c
.
2019-07-10 16:52:04 +02:00
Marco Castelluccio
9a53baf7ec
Upload importances.json files as an artifact
2019-07-10 16:39:48 +02:00
Marco Castelluccio
17b027c767
Enable feature importance calculation at training time for the regressor model
2019-07-10 16:25:38 +02:00
Davide Monfrecola
e7ed31940c
Remove pip from the commit retrieval image ( #676 )
...
Fixes #647
2019-07-09 18:32:57 +02:00
Marco Castelluccio
261bff6ab0
Rename public/probs,json artifact to public/probs.json
2019-07-09 14:41:03 +02:00
Boris Feld
e7add98563
Update task-boot to 0.1.9 ( #675 )
2019-07-05 15:36:16 +02:00
Marco Castelluccio
d08e062c68
Improve triggerSchema of the hooks and test it
2019-07-03 17:26:01 +02:00
Marco Castelluccio
aac6326ba5
Index classify-patch task in the Taskcluster index by diff ID
...
Fixes #668
2019-07-03 15:46:35 +02:00
Marco Castelluccio
28cb2b291e
Insert the Phabricator diff ID in the command directly rather than in the env
2019-07-03 15:46:21 +02:00
Marco Castelluccio
3315532904
Use mozilla/bugbug-commit-retrieval image for the classify-patch hook
...
Since we need to use Mercurial
2019-07-03 10:40:02 +02:00
Marco Castelluccio
9de3aa70f2
Add TC_SECRET_ID to the env of the classify-patch hook
2019-07-03 10:39:37 +02:00
Marco Castelluccio
7e6cf9cf2e
Set version of other bugbug images too
2019-07-02 21:04:23 +02:00
Marco Castelluccio
cb5f54f89f
Use bash to resolve env variable
2019-07-02 21:04:23 +02:00
Marco Castelluccio
71ebbae38a
Handle cases where the hook definition already contains something in the env
...
Also refactor the file to make it more easily testable
2019-07-02 20:52:57 +02:00
Marco Castelluccio
8e23bb7f8a
Add a hook to classify a Phabricator diff
2019-07-02 19:39:59 +02:00
Marco Castelluccio
6ce18762de
'payload.command' should be an array
2019-07-02 13:26:46 +02:00
Marco Castelluccio
d12a25f644
Upload feature visualization image as an artifact of the training tasks
2019-07-01 13:10:39 +02:00
Boris Feld
7459f79317
Use the base image for training models ( #656 )
...
Fixes #350
2019-06-29 00:01:51 +02:00
Boris Feld
d24993d0ac
Remove dependency on rollbacktest in docker build. ( #653 )
...
Fixes #651
2019-06-28 15:32:39 +02:00
Boris Feld
2977a8af89
Start tracking the decision task of the data pipeline ( #649 )
...
This way we can automate the stop of a running data pipeline during the
rollback process.
2019-06-28 15:32:04 +02:00
Boris Feld
54e41d1497
Use taskboot 0.1.8 ( #645 )
...
The new taskboot release solves the double build on non-tag commits and
allows the heroku deploy to be fully atomic.
2019-06-28 11:11:48 +02:00
Boris Feld
2b48e153cb
Use tagged version in bugbug hooks ( #640 )
...
Fixes #634
2019-06-27 15:47:57 +02:00
Marco Castelluccio
eb8f847d43
Install 'git' package in the commit_retrieval image
...
Even though it's only required for the annotate pipeline.
2019-06-27 01:25:58 +02:00
Marco Castelluccio
f65bf19ab8
Disable pip cache when installing Mercurial
2019-06-27 01:25:22 +02:00
Marco Castelluccio
afac5cbdb6
Remove apt lists after installing Python 2
2019-06-27 01:24:59 +02:00
Marco Castelluccio
ad67d76992
Use bugbug-commit-retrieval image for the microannotate-generate task, as need Mercurial to be installed
2019-06-27 01:14:32 +02:00
Marco Castelluccio
4a2759e9b0
Add TC_SECRET_ID env variable to microannotate-generate task
2019-06-27 01:13:51 +02:00
Marco Castelluccio
8fd3be99ce
Command must be split in two lines
2019-06-26 22:28:40 +02:00
Marco Castelluccio
252e29398d
command should be an array
2019-06-26 21:01:31 +02:00
Marco Castelluccio
4b56be37a9
Add missing secrets scope to annotate hook
2019-06-26 19:45:44 +02:00
Marco Castelluccio
b01eeafe44
Add annotate-pipeline.yml file to the spawn_pipeline Docker image
2019-06-26 19:45:22 +02:00
Marco Castelluccio
ea7a009450
Fix schedule value
2019-06-26 19:40:19 +02:00
Marco Castelluccio
56f224b9dc
Generate microannotate repository for mozilla-central
2019-06-26 18:57:36 +02:00
x249wang
ab28e8ace2
Use zstandard instead of xz ( #524 )
...
Fixes #461 .
2019-06-24 13:16:44 +02:00
Boris Feld
9834053a36
Start tracking training metrics as Taskcluster artifacts ( #604 )
...
Fixes #342
2019-06-22 14:18:08 -07:00
Boris Feld
27f9104fb5
Make sure the Docker build task uses the tagged code ( #610 )
...
If not, new master code might get released and conflict with the code in the
bugbug images.
Fixes #609
2019-06-21 08:20:08 -07:00
Boris Feld
417f2aae69
Use a single spawn image for both hooks ( #594 )
...
This removes the need to keep two different Docker images for spawning the
data pipeline and the check pipeline.
Fixes #365
2019-06-14 18:54:21 +02:00
Boris Feld
c06db28442
Bump taskboot to version 1.0.7 ( #583 )
...
Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update
task-boot version to use it.
Also add missing tags and cache option when building Docker images in
data-pipeline.yml
2019-06-12 20:11:34 +02:00
Marco Castelluccio
89b37b96ae
Upload version file too in the bugs retrieval task
2019-06-09 00:13:20 +02:00
Marco Castelluccio
353d21d01b
Clone repository quietly
2019-06-08 11:19:01 +02:00
Marco Castelluccio
4a991ac6ef
Fix download of bugs DB in the rollback test
2019-06-08 11:17:15 +02:00
Marco Castelluccio
76b82d0427
Load yaml safely
...
This is mostly to avoid warnings, as the yaml files we load are fully controlled by us.
2019-06-07 23:34:14 +02:00
Marco Castelluccio
9de91456f6
Update to taskboot 0.1.6
2019-06-07 22:03:00 +02:00
Boris Feld
a8faa48d8a
Support classifying batches of bugs with a background worker ( #321 )
2019-06-07 21:22:14 +02:00
Marco Castelluccio
82d9c0ece0
Update to taskboot 0.1.5
2019-06-07 16:47:28 +02:00
Boris Feld
2e05e57be2
Build docker images data pipeline tag ( #566 )
...
* Build the HTTP Docker image with the right tag
* Ensure the builded docker image has the right parent image
2019-06-07 16:46:05 +02:00
Boris Feld
2988700028
Use tagged index urls for pushing artifacts ( #561 )
...
* Use tagged index urls for pushing artifacts
Also replace previous code that updated Docker image tag to use JSON-e
templating instead.
2019-06-07 12:52:29 +02:00
Boris Feld
7906380e6f
Bump version of taskboot to use latest version of img tool ( #562 )
...
It is necessary to support mulit-tag Docker image building
2019-06-07 12:21:09 +02:00
Boris Feld
e0accae208
Move string formatting to f-string in spawn_data_pipeline ( #559 )
2019-06-07 11:04:33 +02:00
pyup.io bot
c590278bff
Update pyyaml from 5.1 to 5.1.1 ( #560 )
2019-06-07 10:56:26 +02:00
Boris Feld
5a31c99ac9
Add support for specific Docker tag in spawn_data_pipeline.py ( #553 )
...
* Revert "Revert "Add support for specific Docker tag in spawn_data_pipeline.py (#489 )" (#499 )"
This reverts commit 249ed40eb6
.
* Ignore task with a tagged docker image
* Restrict Docker tag update to bugbug related images
2019-06-06 19:14:27 +02:00
Boris Feld
32f56a3962
Add a script to update the hook definition with the TAG during release ( #507 )
...
Fixes #501 , fixed relanding of #491 .
2019-06-06 18:11:59 +02:00
Marco Castelluccio
f5951ad63a
Support retrieving some label files at runtime, and do it for the regressor labels
2019-06-05 00:37:26 +02:00
Marco Castelluccio
7790f5e3d5
Use raw CSV file, not GitHub's HTML page
2019-06-04 13:08:24 +02:00
Marco Castelluccio
d57177f1e4
Fix destination path of the regressor.csv label file
2019-06-04 13:07:59 +02:00
Marco Castelluccio
44e26ff0e8
Add a training task for the Regressor model
2019-06-03 22:15:18 +02:00
Marco Castelluccio
2804436357
Download regressor labels from marco-c/mozilla-central-regressors repository in the train_regressor Docker image
2019-06-03 22:14:47 +02:00
Marco Castelluccio
72ddfea2e3
Add a Docker image for the task to train the Regressor model
2019-06-03 21:46:35 +02:00
Marco Castelluccio
4ce438a35a
Fix typo in artifact name for the commits retrieval task
2019-06-03 21:37:39 +02:00
Marco
d8b84ca798
Support retrieving commits in steps ( #536 )
...
* Support retrieving commits in steps
* Store component mapping ETag to actually avoid downloading it again when not needed
* Store a version file alongside the DBs
* Export the commits DB version file and the experiences values as artifacts of the commit-retriever task
2019-06-03 19:29:08 +02:00
Marco Castelluccio
e62dd6f37d
Make rollback-test task verbose
2019-06-03 11:06:32 +02:00
Ayush Shridhar
9d71677667
Add a training task for the Duplicate model ( #525 )
2019-05-31 17:05:58 +02:00
Marco Castelluccio
bd3e4c7900
Increase the maximum runtime for the commits retrieval task
2019-05-30 13:27:23 +02:00
Marco Castelluccio
42d2ff2db8
Add a training task for the Backout model
2019-05-30 13:27:06 +02:00
Marco Castelluccio
cd0405d223
Add a Docker image for the Backout model
2019-05-30 13:24:55 +02:00
Boris Feld
6ee9fb57f0
Fix Docker build by downloading the models inside the image. Fix #504 ( #516 )
...
The data pipeline failed before because it tried downloading the model from
outside the Docker image and didn't had bugbug installed.
The clean way of solving this would be to build a base http service image on
release and build another one where we simply download the models but let's
fix it this way for now.
2019-05-29 20:43:58 +02:00
Marco Castelluccio
0037ad35b5
Revert "Add a script to update the hook definition with the TAG during release ( #491 )"
...
This reverts commit 6ef95cfec3
.
2019-05-25 00:16:41 +02:00
Marco
249ed40eb6
Revert "Add support for specific Docker tag in spawn_data_pipeline.py ( #489 )" ( #499 )
...
This reverts commit 6e7c815f13
.
2019-05-24 17:30:24 +02:00
Boris Feld
6ef95cfec3
Add a script to update the hook definition with the TAG during release ( #491 )
2019-05-24 15:13:52 +02:00
Boris Feld
6e7c815f13
Add support for specific Docker tag in spawn_data_pipeline.py ( #489 )
2019-05-24 12:47:20 +02:00
Boris Feld
1bae5834ab
Implement deployment to Heroku ( #458 )
2019-05-23 20:39:02 +02:00
Ayush Shridhar
b41170baa5
Add training task for the StepsToReproduce model ( #441 )
2019-05-22 21:43:11 +02:00
Ayush Shridhar
91bf939fb7
Add training task for the RegressionRange model ( #466 )
2019-05-22 18:58:47 +02:00
Marco
025e3f4da2
Fix command to train defect/enhancement/task model ( #476 )
...
* Fix command to train defect/enhancement/task model
Fixes #475
* Add more logging in the trainer script, and assert the model is generated
2019-05-21 14:46:57 +02:00
Marco
0093f8c10f
Fix SHOULD_DOWNLOAD_MODEL environment variable name ( #469 )
...
Fixes #468
2019-05-20 15:24:33 +02:00
Boris Feld
dd00d7b9ec
Add the support for downloading the model before checking it ( #452 )
...
Also put the right configuration in the check pipeline
2019-05-17 11:45:42 +02:00
pyup.io bot
f8914dd59d
Update requests to 2.22.0 ( #448 )
2019-05-16 17:31:46 +02:00
Marco
8a5795417a
Add a pre-commit hook using codespell ( #411 )
...
* Add a pre-commit hook using codespell
Fixes #410
* Fix some spelling mistakes
2019-05-16 17:24:18 +02:00
Boris Feld
d3c3bcbece
Bump version of taskboot used in taskcluster and data pipeline ( #446 )
2019-05-16 13:02:58 +02:00
Marco Castelluccio
52f00714e9
No need for big instance types for the check component model task
2019-05-15 12:03:12 +02:00
Marco Castelluccio
3719540351
Change role for the checks hook
2019-05-15 11:54:02 +02:00
Marco Castelluccio
ff9ea35ed0
Reduce deadlines to maximum of 5 days
...
Taskcluster only allows up to 5 days
2019-05-14 20:39:00 +02:00
Marco Castelluccio
350ddbc1e9
Remove bad trailing comma in infra/taskcluster-hook-check-models-start.json
...
Fixes #413
2019-05-14 20:30:56 +02:00
Marco
9223954520
Remove training tasks' unneeded dependencies on commit retrieval task ( #407 )
...
Fixes #390
2019-05-14 15:22:44 +02:00
Marco
c4bd01278e
Add 'expires' to all tasks to avoid them expiring in a too long time ( #393 )
...
Fixes #391 .
2019-05-12 21:46:58 +02:00
Marco
e3230ca999
Increase deadline of data pipeline tasks ( #389 )
...
Fixes #388 .
2019-05-10 16:12:46 +02:00
Boris Feld
f4b2b938be
Add basic check method and check script ( #341 )
...
* Add basic check method and check script
* Ensure the check of component will correctly use super result
* Add required infra to schedule model checks
* Add scheduling bits for the model checks
* Remove the filtering on classification
* Extract counting bugs to a new function in bugzilla.py
* Also checks conflated components
* Fix new hook id
* Call bugzilla with the count_only param to speed up the check
* Fix the new hook scope to match the hook id
* Fix component model check after previous refactoring
* Fix component model check method
* Use a bugzilla report for even faster component model check
* Clarify get_product_component_count docstring
We are already filtering out full component with 0 bugs
* Update conflated components mapping check
A conflated component could also be part of the conflated components mapping
* Distinguish between non-existing full components and empty full components
* Remove the filter on resolution and unnecessary url params
* Update component check method
Keep checks as separate as possible for clarity, we could merge them or makes
them faster later
* Generate dynamically the CSV report url
* Fix Docker image name the hook
* Implement component check number 5
Get the meaningful components for the last 6 months
* Handle reviews comments
* Remove extraneous print
* Removes TODO
* Use a different threshold ration when checking for new meaningful components
As we are only checking new bugs for 6 months, adjust the threshold ration to
be less sensitive to occasional burst ob bugs for q given component.
* Reduce the threshold ratio
As we check on a disjoint time window, reduce the chance of false positives
* Handle review nits
* Fix last nits
2019-05-10 12:20:23 +02:00
Marco
6f09488573
Rename mozilla/bugbug-train-defect image to mozilla/bugbug-train-defectenhancementtask ( #375 )
...
Fixes #364 .
2019-05-09 23:36:38 +02:00
Boris Feld
4954fe38d4
Change scheduling of the data pipeline to run once every two weeks ( #361 )
...
Fixes #356 .
2019-05-09 11:44:53 +02:00
Marco Castelluccio
c3f55e682a
Rename train-defect to train-defectenhancementtask
2019-05-07 13:16:22 +02:00
Marco Castelluccio
2eaf90be20
Add a cache to the commit retrieval task
...
Fixes #347
2019-05-07 11:38:02 +02:00
Boris Feld
6937e0e5e8
Add the rollback test in the data pipeline ( #337 )
...
Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 14:20:43 +02:00
Marco
9995b8c236
Make training code more generic to make it possible to train on other kinds of objects (e.g. commits) ( #335 )
...
* Move feature cleanup functions in a separate module
As they can be shared for different objectives, e.g. both training on bugs and on commits.
* Make Model more generic to make it possible to train on different objects
Introduce BugModel and CommitModel, as base classes for models training on bugs and on commits.
Update all models to use BugModel and to use the new feature_cleanup module.
Fixes #306 .
* Update ID and description of the defect/enhancement/task Taskcluster task definition
* Add a module to extract features from commit data
* Add an example model training on commits to predict commits which will be backed out
* Update defect model name, and add possibility to train backout model
2019-05-03 11:57:48 +02:00
Boris Feld
297963e4ce
Skip checking models while building the http service image, and only push it as part of the pipeline ( #331 )
...
* Add a way to skip checking models while building the http service image
* Don't push the http service on release
It isn't built with the real models on release
* Use taskboot 0.1.1
2019-05-02 23:18:51 +02:00
Boris Feld
369b44ea02
Update the index URLs in bugbug ( #328 )
...
* Update the index URLs in bugbug
* Split the http service Docker image in two
This way we can both:
- Build the first half (code + dependencies) in the usual CI.
- Build the second half at the end of the data pipeline with updated models.
Taskboot build-compose doesn't support building all services except a
specific one and it might be cumbersome to add this feature so move the second
half of the Docker image to a separate docker-compose file.
2019-05-02 17:00:32 +02:00
Boris Feld
8c633b6da1
Remove the experimental Dockerfile syntax ( #330 )
...
This way we can still build locally with docker-compose. We don't use the experimental syntax anymore.
2019-05-02 16:01:48 +02:00
Boris Feld
6e7ca892cd
Introduce a new Docker image for data-pipeline spawning ( #320 )
2019-05-02 14:36:50 +02:00
Marco
f4b096519f
Don't use pip cache directory, and remove downloaded apt lists ( #314 )
2019-04-26 16:14:52 +02:00
Boris Feld
1ae2200682
Create the initial data pipeline task definition ( #313 )
...
* Create the data pipeline task definition
There is a hook (which runs every day at midnight) that will spawn the
data-pipeline using the latest docker images that were build on latest
release. The hook itself is updated on each release and is versionned in this
repository.
The hook will runs once every week and on every successful release.
Add task for building the Docker image for HTTP service.
2019-04-26 14:56:49 +02:00
Bastien Abadie
87a5674a9c
Setup bugbug with bind mount
2019-04-25 13:08:26 +02:00
Bastien Abadie
fe01e45dcd
Switch to slim image
2019-04-25 11:15:34 +02:00
Marco Castelluccio
e199972050
Actually use f-string in version check script
2019-04-19 19:13:28 +02:00
Boris Feld
053954d70b
Run pre-commit in the lint task ( #297 )
2019-04-19 18:01:24 +02:00
Boris Feld
ded92357ee
Add a new tasks for updating bugbug hook ( #289 )
...
The hook will be responsible for starting the data pipeline, but right now is
doing mostly nothing.
Use taskboot new command `build-hook` to update the hook based on the
versioned definition file.
2019-04-18 21:20:04 +02:00
Bastien Abadie
b5153abc89
Docker full build using compose ( #274 )
2019-04-15 17:36:30 +02:00
Boris Feld
d899ebc068
Add missing Dockerfiles for the training steps ( #276 )
2019-04-12 12:31:13 +02:00
Marco Castelluccio
d0f98674b8
Be more verbose when the version check command fails
2019-04-11 00:40:11 +02:00
Marco Castelluccio
31f371814e
Deploy bugbug package to PyPI on new tags
2019-04-10 17:32:20 +02:00
Boris Feld
b651744b18
Import retriever services and add Docker image definition ( #251 )
...
* Import Retriever class from release-services repository
This basically import the `retriever.py` file from the `release-services`
repository at hash 77cdddd. I removed imports and reference to cli-common
helpers that will likely needs to be reimplemented, like the raven support.
The next commit will defines some Dockerfiles that will use the imported file.
* Add docker image definition
Build three Docker image, one is for bugbug itself. It is just installing
bugbug and its dependencies.
One is for retrieving information from the mozilla-central Mercurial
repository, it depends on the first one and install the right Mercurial
version.
The last one is for retrieving information from the Bugzilla instance, it
depends in the first one and needs a valid Bugzilla token.
* Separate the two tasks into separate script files
They share almost no code at all so they don't need to be in the same file
* Apply Black on the scripts to makes Flake8 happy
2019-04-09 16:30:09 +02:00