mozilla
/
bugbug
зеркало из https://github.com/mozilla/bugbug.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
bugbug/infra/data-pipeline.yml

456 строки
15 KiB
YAML
Исходник Обычный вид История

Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
version: 1
tasks:
- ID: commit-retrieval
created: {$fromNow: ''}
Increase deadline of data pipeline tasks (#389) Fixes #388.
2019-05-10 17:12:46 +03:00
deadline: {$fromNow: '72 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
payload:
Increase the maximum runtime for the commits retrieval task
2019-05-30 14:27:23 +03:00
maxRunTime: 86400
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-commit-retrieval:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/commits.json.xz:
path: /data/commits.json.xz
type: file
Support retrieving commits in steps (#536) * Support retrieving commits in steps * Store component mapping ETag to actually avoid downloading it again when not needed * Store a version file alongside the DBs * Export the commits DB version file and the experiences values as artifacts of the commit-retriever task
2019-06-03 20:29:08 +03:00
public/commits.json.version:
path: /data/commits.json.version
type: file
Fix typo in artifact name for the commits retrieval task
2019-06-03 22:37:39 +03:00
public/commit_experiences.pickle.xz:
path: /data/commit_experiences.pickle.xz
Support retrieving commits in steps (#536) * Support retrieving commits in steps * Store component mapping ETag to actually avoid downloading it again when not needed * Store a version file alongside the DBs * Export the commits DB version file and the experiences values as artifacts of the commit-retriever task
2019-06-03 20:29:08 +03:00
type: file
Add a cache to the commit retrieval task Fixes #347
2019-05-07 12:28:55 +03:00
cache:
bugbug-mercurial-repository: /cache
scopes:
- "docker-worker:cache:bugbug-mercurial-repository"
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.data_commits.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.data_commits.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
name: bugbug commit retrieval
description: bugbug commit retrieval
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
- ID: bugs-retrieval
created: {$fromNow: ''}
Increase deadline of data pipeline tasks (#389) Fixes #388.
2019-05-10 17:12:46 +03:00
deadline: {$fromNow: '72 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc
payload:
env:
TC_SECRET_ID: project/relman/bugbug/production
maxRunTime: 86400
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-bugs-retrieval:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/bugs.json.xz:
path: /data/bugs.json.xz
type: file
Upload version file too in the bugs retrieval task
2019-06-09 01:13:20 +03:00
public/bugs.json.version:
path: /data/bugs.json.version
type: file
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
features:
taskclusterProxy:
true
scopes:
- "secrets:get:project/relman/bugbug/production"
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.data_bugs.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.data_bugs.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
name: bugbug bugs retrieval
description: bugbug bugs retrieval
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add the rollback test in the data pipeline (#337) Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 15:20:43 +03:00
- ID: rollback-test-task
created: {$fromNow: ''}
Increase deadline of data pipeline tasks (#389) Fixes #388.
2019-05-10 17:12:46 +03:00
deadline: {$fromNow: '73 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Add the rollback test in the data pipeline (#337) Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 15:20:43 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc
dependencies:
- bugs-retrieval
payload:
maxRunTime: 3600
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-base:${version}
Add the rollback test in the data pipeline (#337) Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 15:20:43 +03:00
command:
- "/bin/bash"
- "-lcx"
Fix download of bugs DB in the rollback test
2019-06-08 12:17:15 +03:00
- "python -c 'from bugbug import bugzilla, db; db.download(bugzilla.BUGS_DB)' &&
Make rollback-test task verbose
2019-06-03 12:06:20 +03:00
python -m bugbug.bug_snapshot --verbose"
Add the rollback test in the data pipeline (#337) Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 15:20:43 +03:00
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
metadata:
name: bugbug rollback test
description: bugbug rollback test
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add a training task for the Backout model
2019-05-30 14:27:06 +03:00
- ID: train-backout
created: {$fromNow: ''}
deadline: {$fromNow: '118 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- commit-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-backout:${version}
Add a training task for the Backout model
2019-05-30 14:27:06 +03:00
artifacts:
public/backoutmodel.xz:
path: /backoutmodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_backout.${version}
Add a training task for the Backout model
2019-05-30 14:27:06 +03:00
- index.project.relman.bugbug.train_backout.latest
metadata:
name: bugbug train backout model
description: bugbug train backout model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
- ID: train-component
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '118 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-component:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/componentmodel.xz:
path: /componentmodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_component.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.train_component.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
name: bugbug train component model
description: bugbug train component model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Make training code more generic to make it possible to train on other kinds of objects (e.g. commits) (#335) * Move feature cleanup functions in a separate module As they can be shared for different objectives, e.g. both training on bugs and on commits. * Make Model more generic to make it possible to train on different objects Introduce BugModel and CommitModel, as base classes for models training on bugs and on commits. Update all models to use BugModel and to use the new feature_cleanup module. Fixes #306. * Update ID and description of the defect/enhancement/task Taskcluster task definition * Add a module to extract features from commit data * Add an example model training on commits to predict commits which will be backed out * Update defect model name, and add possibility to train backout model
2019-05-03 12:57:48 +03:00
- ID: train-defectenhancementtask
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '118 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-defectenhancementtask:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/defectenhancementtaskmodel.xz:
path: /defectenhancementtaskmodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_defectenhancementtask.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.train_defectenhancementtask.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
Make training code more generic to make it possible to train on other kinds of objects (e.g. commits) (#335) * Move feature cleanup functions in a separate module As they can be shared for different objectives, e.g. both training on bugs and on commits. * Make Model more generic to make it possible to train on different objects Introduce BugModel and CommitModel, as base classes for models training on bugs and on commits. Update all models to use BugModel and to use the new feature_cleanup module. Fixes #306. * Update ID and description of the defect/enhancement/task Taskcluster task definition * Add a module to extract features from commit data * Add an example model training on commits to predict commits which will be backed out * Update defect model name, and add possibility to train backout model
2019-05-03 12:57:48 +03:00
name: bugbug train defect/enhancement/task model
description: bugbug train defect/enhancement/task model
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
- ID: train-regression
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '118 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-regression:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/regressionmodel.xz:
path: /regressionmodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_regression.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.train_regression.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
name: bugbug train regression model
description: bugbug train regression model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add a training task for the Regressor model
2019-06-03 23:15:18 +03:00
- ID: train-regressor
created: {$fromNow: ''}
deadline: {$fromNow: '118 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- commit-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-regressor:${version}
Add a training task for the Regressor model
2019-06-03 23:15:18 +03:00
artifacts:
public/regressormodel.xz:
path: /regressormodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_regressor.${version}
Add a training task for the Regressor model
2019-06-03 23:15:18 +03:00
- index.project.relman.bugbug.train_regressor.latest
metadata:
name: bugbug train regressor model
description: bugbug train regressor model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
- ID: train-tracking
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '118 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-tracking:${version}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/trackingmodel.xz:
path: /trackingmodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_tracking.${version}
Update the index URLs in bugbug (#328) * Update the index URLs in bugbug * Split the http service Docker image in two This way we can both: - Build the first half (code + dependencies) in the usual CI. - Build the second half at the end of the data pipeline with updated models. Taskboot build-compose doesn't support building all services except a specific one and it might be cumbersome to add this feature so move the second half of the Docker image to a separate docker-compose file.
2019-05-02 18:00:32 +03:00
- index.project.relman.bugbug.train_tracking.latest
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
metadata:
name: bugbug train tracking model
description: bugbug train tracking model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add training task for the RegressionRange model (#466)
2019-05-22 19:58:47 +03:00
- ID: train-regressionrange
created: {$fromNow: ''}
deadline: {$fromNow: '118 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-regressionrange:${version}
Add training task for the RegressionRange model (#466)
2019-05-22 19:58:47 +03:00
artifacts:
public/regressionrangemodel.xz:
path: /regressionrangemodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_regressionrange.${version}
Add training task for the RegressionRange model (#466)
2019-05-22 19:58:47 +03:00
- index.project.relman.bugbug.train_regressionrange.latest
metadata:
name: bugbug train regressionrange model
description: bugbug train regressionrange model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add training task for the StepsToReproduce model (#441)
2019-05-22 22:43:11 +03:00
- ID: train-stepstoreproduce
created: {$fromNow: ''}
deadline: {$fromNow: '118 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-stepstoreproduce:${version}
Add training task for the StepsToReproduce model (#441)
2019-05-22 22:43:11 +03:00
artifacts:
public/stepstoreproducemodel.xz:
path: /stepstoreproducemodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_stepstoreproduce.${version}
Add training task for the StepsToReproduce model (#441)
2019-05-22 22:43:11 +03:00
- index.project.relman.bugbug.train_stepstoreproduce.latest
metadata:
name: bugbug train stepstoreproduce model
description: bugbug train stepstoreproduce model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Add a training task for the Duplicate model (#525)
2019-05-31 18:05:58 +03:00
- ID: train-duplicate
created: {$fromNow: ''}
deadline: {$fromNow: '118 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: releng-svc-compute
dependencies:
- bugs-retrieval
payload:
maxRunTime: 25200
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
image: mozilla/bugbug-train-duplicate:${version}
Add a training task for the Duplicate model (#525)
2019-05-31 18:05:58 +03:00
artifacts:
public/duplicatemodel.xz:
path: /duplicatemodel.xz
type: file
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
Use tagged index urls for pushing artifacts (#561) * Use tagged index urls for pushing artifacts Also replace previous code that updated Docker image tag to use JSON-e templating instead.
2019-06-07 13:52:29 +03:00
- index.project.relman.bugbug.train_duplicate.${version}
Add a training task for the Duplicate model (#525)
2019-05-31 18:05:58 +03:00
- index.project.relman.bugbug.train_duplicate.latest
metadata:
name: bugbug train duplicate model
description: bugbug train duplicate model
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
- ID: docker-build
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '119 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: releng-svc
dependencies:
Add the rollback test in the data pipeline (#337) Add the rollback test in the data pipeline and move the bug snapshot test to a pytest test
2019-05-03 15:20:43 +03:00
- rollback-test-task
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
- train-tracking
- train-regression
Rename train-defect to train-defectenhancementtask
2019-05-07 14:15:01 +03:00
- train-defectenhancementtask
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
- train-component
payload:
capabilities:
privileged: true
maxRunTime: 3600
Bump taskboot to version 1.0.7 (#583) Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update task-boot version to use it. Also add missing tags and cache option when building Docker images in data-pipeline.yml
2019-06-12 21:11:34 +03:00
image: mozilla/taskboot:0.1.7
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
command:
- "/bin/sh"
- "-lcxe"
Clone repository quietly
2019-06-08 12:19:01 +03:00
- "git clone --quiet https://github.com/mozilla/bugbug /code &&
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
cd /code &&
Make sure the Docker build task uses the tagged code (#610) If not, new master code might get released and conflict with the code in the bugbug images. Fixes #609
2019-06-21 18:20:09 +03:00
git checkout ${version} &&
Bump taskboot to version 1.0.7 (#583) Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update task-boot version to use it. Also add missing tags and cache option when building Docker images in data-pipeline.yml
2019-06-12 21:11:34 +03:00
taskboot --cache /cache --target /code build-compose --registry=registry.hub.docker.com --write /images --service bugbug-http-service --tag ${version} --tag latest --build-arg BUGBUG_VERSION=${version} &&
taskboot --cache /cache --target /code build-compose --registry=registry.hub.docker.com --write /images --service bugbug-http-service-bg-worker --tag ${version} --tag latest --build-arg BUGBUG_VERSION=${version}"
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
artifacts:
public/bugbug:
expires: {$fromNow: '2 weeks'}
path: /images
type: directory
scopes:
- docker-worker:capability:privileged
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
metadata:
name: bugbug docker http build
description: bugbug docker http build
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
- ID: docker-push
dependencies:
- docker-build
scopes:
- secrets:get:project/relman/bugbug/deploy
created: {$fromNow: ''}
Reduce deadlines to maximum of 5 days Taskcluster only allows up to 5 days
2019-05-14 21:38:56 +03:00
deadline: {$fromNow: '120 hours'}
Add 'expires' to all tasks to avoid them expiring in a too long time (#393) Fixes #391.
2019-05-12 22:46:58 +03:00
expires: {$fromNow: '1 month'}
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
provisionerId: aws-provisioner-v1
workerType: github-worker
payload:
features:
taskclusterProxy:
true
maxRunTime: 3600
Bump taskboot to version 1.0.7 (#583) Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update task-boot version to use it. Also add missing tags and cache option when building Docker images in data-pipeline.yml
2019-06-12 21:11:34 +03:00
image: mozilla/taskboot:0.1.7
Create the initial data pipeline task definition (#313) * Create the data pipeline task definition There is a hook (which runs every day at midnight) that will spawn the data-pipeline using the latest docker images that were build on latest release. The hook itself is updated on each release and is versionned in this repository. The hook will runs once every week and on every successful release. Add task for building the Docker image for HTTP service.
2019-04-26 15:56:49 +03:00
env:
TASKCLUSTER_SECRET: project/relman/bugbug/deploy
command:
- taskboot
- push-artifact
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
metadata:
name: bugbug docker http push
description: bugbug docker http push
owner: release-mgmt-analysis@mozilla.com
Use mozilla/taskboot 0.1.0 (#323) Fixes #322
2019-04-29 19:08:45 +03:00
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml
Implement deployment to Heroku (#458)
2019-05-23 21:39:02 +03:00
- ID: http-deploy
dependencies:
- docker-build
scopes:
- secrets:get:project/relman/bugbug/deploy
created: {$fromNow: ''}
deadline: {$fromNow: '120 hours'}
expires: {$fromNow: '1 month'}
provisionerId: aws-provisioner-v1
workerType: github-worker
payload:
features:
taskclusterProxy:
true
maxRunTime: 3600
Bump taskboot to version 1.0.7 (#583) Now that https://github.com/mozilla/task-boot/issues/39 is fixed, let's update task-boot version to use it. Also add missing tags and cache option when building Docker images in data-pipeline.yml
2019-06-12 21:11:34 +03:00
image: mozilla/taskboot:0.1.7
Implement deployment to Heroku (#458)
2019-05-23 21:39:02 +03:00
env:
TASKCLUSTER_SECRET: project/relman/bugbug/deploy
command:
Support classifying batches of bugs with a background worker (#321)
2019-06-07 22:22:14 +03:00
- "/bin/sh"
- "-lcxe"
- "taskboot deploy-heroku --artifact-filter public/bugbug/bugbug-http-service.tar --heroku-app bugbug &&
taskboot deploy-heroku --artifact-filter public/bugbug/bugbug-http-service-bg-worker.tar --heroku-app bugbug --heroku-dyno-type worker"
Implement deployment to Heroku (#458)
2019-05-23 21:39:02 +03:00
routes:
- notify.email.release-mgmt-analysis@mozilla.com.on-failed"
- notify.irc-channel.#bugbug.on-failed
metadata:
name: bugbug docker http service deploy
description: bugbug docker http service deploy
owner: release-mgmt-analysis@mozilla.com
source: https://github.com/mozilla/bugbug/raw/master/data-pipeline.yml