* Add a central place where the models are defined
Also add some helpers to load a model.
* Add missing tensorflow dependency in extra-nn-requirements.txt
* Add basic check method and check script
* Ensure the check of component will correctly use super result
* Add required infra to schedule model checks
* Add scheduling bits for the model checks
* Remove the filtering on classification
* Extract counting bugs to a new function in bugzilla.py
* Also checks conflated components
* Fix new hook id
* Call bugzilla with the count_only param to speed up the check
* Fix the new hook scope to match the hook id
* Fix component model check after previous refactoring
* Fix component model check method
* Use a bugzilla report for even faster component model check
* Clarify get_product_component_count docstring
We are already filtering out full component with 0 bugs
* Update conflated components mapping check
A conflated component could also be part of the conflated components mapping
* Distinguish between non-existing full components and empty full components
* Remove the filter on resolution and unnecessary url params
* Update component check method
Keep checks as separate as possible for clarity, we could merge them or makes
them faster later
* Generate dynamically the CSV report url
* Fix Docker image name the hook
* Implement component check number 5
Get the meaningful components for the last 6 months
* Handle reviews comments
* Remove extraneous print
* Removes TODO
* Use a different threshold ration when checking for new meaningful components
As we are only checking new bugs for 6 months, adjust the threshold ration to
be less sensitive to occasional burst ob bugs for q given component.
* Reduce the threshold ratio
As we check on a disjoint time window, reduce the chance of false positives
* Handle review nits
* Fix last nits
* Add a way to skip checking models while building the http service image
* Don't push the http service on release
It isn't built with the real models on release
* Use taskboot 0.1.1
* Update the index URLs in bugbug
* Split the http service Docker image in two
This way we can both:
- Build the first half (code + dependencies) in the usual CI.
- Build the second half at the end of the data pipeline with updated models.
Taskboot build-compose doesn't support building all services except a
specific one and it might be cumbersome to add this feature so move the second
half of the Docker image to a separate docker-compose file.
* Create the data pipeline task definition
There is a hook (which runs every day at midnight) that will spawn the
data-pipeline using the latest docker images that were build on latest
release. The hook itself is updated on each release and is versionned in this
repository.
The hook will runs once every week and on every successful release.
Add task for building the Docker image for HTTP service.
The hook will be responsible for starting the data pipeline, but right now is
doing mostly nothing.
Use taskboot new command `build-hook` to update the hook based on the
versioned definition file.
* Rename test function in test_bug to reflect reality, and add more assertions
* Add a mock bugs DB for tests
* Don't download bugs DB anymore for running tests
* Add a test for run.py basic functionality
* Remove training test task, as the test is now a pytest
Marco Castelluccio
Bastien Abadie
Boris Feld
Shubham Garg