* Add a run_pipeline utility
* Add more tests for training
* Rewrite train.sh into train.py
* Add the pipeline to the PYTHONPATH
* Ensure that the W&B tracker throws errors in CI
* Add the Taskcluster environment variables so test-fast works on the train test
* Address review comments
* Use task group ID as suffix for offline Taskcluster publication from GCP
* Fix group_logs publication
* Add a mode to support GCP experiments from Taskcluster in a generic way
* Fix metrics path for GCP experiments that ran on Taskcluster
* Ignore old snakemake metrics that cannot be parsed
* Update tests
* Do not parse metrics name for new GCP experiments (taskcluster)
* Add tests for metrics filename parser
* Add a parser for GCP metrics filename support
* Support Taskcluster metrics structure in WandB.publish_group_logs
* Patch model name in group_logs
* Patch model suffix in group_logs
* Add details to value error exceptions
* Do continue on unsupported filename (Snakemake)
* Preserve legacy metrics dir for snakemake experiments
* Rework the GCP file structure browsing
* Fixes
* Include quantized metrics
* Update tests
* Publish Marian, OpusTrainer configs and datasets statistics
* Update tests
* Fixes
* Fix tests in CI context
* Nit
* Store extra config files as new keys of the main config
* Plot datasets in a custom chart
* Fixes
* Support extra-args for offline publication
* Fix tests
* Suggestions
* TRASHME Test publication from CI
* Revert "TRASHME Test publication from CI"
This reverts commit 2da4a9a3cd.
* Suggestion
* Trigger CI
* Fix training and model key detection
* TRASHME: Trigger publication from CI
* Revert "TRASHME: Trigger publication from CI"
This reverts commit ad4a3b7368.
---------
Co-authored-by: Evgeny Pavlov <epavlov@mozilla.com>
Co-authored-by: Evgeny Pavlov <pavlov89@gmail.com>
* Publish experiments from the CI
* Disable cache for CI runs
* Revert "Disable cache for CI runs"
This reverts commit ca4593a39846a1a5cddf5ebf41a02fc698e23bea.
* Use unique names for W&B runs
* Update tests
* Fix online group_logs publication
* Use task group ID suffix in offline publication from Taskcluster
* Use task group ID suffix in offline publication from GCP old Snakemake experiments
* Fix
* TRASHME Test publication from CI
* Revert "TRASHME Test publication from CI"
This reverts commit 4e15ed4eb4.
* Configure evaluation tasks
* Extract w&b code into module
* Do not check taskcluwter when publication is disabled
* Publish evaluation metrics to W&B
* Fix running eval tracking on CI
* Use args.wandb_run_name instead of default teacher
* Remove duplicated arguments
* Retrieve dataset from Taskcluster directly
* Add missing calls to publisher and logging
* Allow publishing metrics as a table on existing runs (i.e. previous trainings)
* Update regex to parse labels ending with '-1'
* Generic support for train/eval different naming
* Update tests
* Support disabled publication
* Publish group_logs from taskcluster
* Update tests
* Refactor group_log publication between online and offline taskcluster
* Restore missing input-file argument
* Rebase and fixes
* TRASHME test parameters to trigger train in CI
* Fix metrics_tasks default value
* Fix import
* Run linter
* Publish config first
* Revert "TRASHME test parameters to trigger train in CI"
This reverts commit ede4245786.
---------
Co-authored-by: Bastien Abadie <bastien@nextcairn.com>
Co-authored-by: Bastien Abadie <abadie@teklia.com>
Co-authored-by: Evgeny Pavlov <epavlov@mozilla.com>
* Configure evaluation tasks
* Extract w&b code into module
* Do not check taskcluwter when publication is disabled
* Publish evaluation metrics to W&B
* Fix running eval tracking on CI
* Use args.wandb_run_name instead of default teacher
* Remove duplicated arguments
* Retrieve dataset from Taskcluster directly
* Add missing calls to publisher and logging
* Allow publishing metrics as a table on existing runs (i.e. previous trainings)
* Update regex to parse labels ending with '-1'
* Generic support for train/eval different naming
* Update tests
* Support disabled publication
---------
Co-authored-by: Bastien Abadie <bastien@nextcairn.com>
Co-authored-by: Bastien Abadie <abadie@teklia.com>
Co-authored-by: Evgeny Pavlov <epavlov@mozilla.com>
* Show the stack trace when there is an error in tracking
* Rename parse_tag to parse_task_label
* Document the regexes
* Turn the parsed label into a NamedTuple for better typing hints
* Get the tag parsing working with the full task graph
* Allow for 3 letter language codes
* Temporarily disable an evaluate task that is failing
* Update code docs a bit
* Fix tests for finetune-student
* Add test
* Update name detection from tasks
* Update other tests
* Support renaming for old experiments and quantized run for consistency
* Update tests
* Suggestions
* Fixes
* Add W&B publication setting
* Switch to boolean
* Add wandb setting to the train action
* Run linter
* Change type name
* Make bool values lower case
* Configure Taskcluster secret for w&b through taskgraph transform
* Direct secret usage
* Support recursive join on tags
* Do not use taskcluster filtering with --from-stream
* Enable verbose mode
* More debugging lines
* Redirect opustrainer stderr to stdout
* Log marian lines in verbose output
* Fix test
* Log raw lines & add prefix to our own logging
* Correct project name
* Convert Taskcluster trigger from transform to to kind
* Fix lint
* Setup tracking on all training tasks
* Set WandB group & run name
* Skip publication on unit tests
* Make perplexity optional
* Update test fixture
* Add training parameter to control publication
* Move trigger control to python
* Use task config to get wandb names
* Bashism
* Use taskcluster group logic to build task name
* Run on WANDB_PUBLICATION=false, but do not publish
* Expose weight & biases tags
* Consistent evaluation tags parsing
* Add test
* Support backwards training task label
* Support evaluation task with suffix
* Support suffixes with form -1/2
---------
Co-authored-by: Evgeny Pavlov <epavlov@mozilla.com>
* Skip checking existing runs for new projects
Follow up of #484
Otherwise a value Error was raised by the wandb client
* Suggestion
---------
Co-authored-by: Evgeny Pavlov <epavlov@mozilla.com>
* Allow group traversal from training tasks dependencies during Taskcluster task group publication
* Fix lint + Apply Valentin's suggestions
* Comment to clarify the usage of a shared set variable
* Do not publish empty groups
* Base taskcluster task group publication
* Move tag parser to utils module
* Support metrics
* Support multiple teacher training
* Fix parsing for evaluation folder
* Generic group logs parser
* Parse extra evaluation tasks and publish group_logs fake run
* Publish Marian config on runs
* Publish marian config on runs instead of experiment config
* Rebase vrigal:publish-experiment-config
* Publish experiment config on group_logs
* Publish metrics at the end
* Add missing steps
* Publish all metrics to a group table
* Support more metrics formats
* Update tests
* Publish runs for extra metrics
* Prefix metrics with group
* Improve metrics publication
* Remove unused Metric.model_name
* Update tests
Valentin Rigal
Greg Tatum
Bastien Abadie
Evgeny Pavlov
EvaBardou
EvaBardou