Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update
task-boot version to use it.
Also add missing tags and cache option when building Docker images in
data-pipeline.yml
* Support retrieving commits in steps
* Store component mapping ETag to actually avoid downloading it again when not needed
* Store a version file alongside the DBs
* Export the commits DB version file and the experiences values as artifacts of the commit-retriever task
The data pipeline failed before because it tried downloading the model from
outside the Docker image and didn't had bugbug installed.
The clean way of solving this would be to build a base http service image on
release and build another one where we simply download the models but let's
fix it this way for now.
* Move feature cleanup functions in a separate module
As they can be shared for different objectives, e.g. both training on bugs and on commits.
* Make Model more generic to make it possible to train on different objects
Introduce BugModel and CommitModel, as base classes for models training on bugs and on commits.
Update all models to use BugModel and to use the new feature_cleanup module.
Fixes#306.
* Update ID and description of the defect/enhancement/task Taskcluster task definition
* Add a module to extract features from commit data
* Add an example model training on commits to predict commits which will be backed out
* Update defect model name, and add possibility to train backout model
* Add a way to skip checking models while building the http service image
* Don't push the http service on release
It isn't built with the real models on release
* Use taskboot 0.1.1
* Update the index URLs in bugbug
* Split the http service Docker image in two
This way we can both:
- Build the first half (code + dependencies) in the usual CI.
- Build the second half at the end of the data pipeline with updated models.
Taskboot build-compose doesn't support building all services except a
specific one and it might be cumbersome to add this feature so move the second
half of the Docker image to a separate docker-compose file.
Boris Feld
Marco Castelluccio
Ayush Shridhar