add get_additional_environment_variables to Lightning container to be
able to flexibly add experiments specific env variables, e.g. Azure
mounting env variables for large file experiments... this can be
overridden in the config file directly.
Ensure that all AML experiments that originate from a PR are submitted
to the same experiment.
- Add capability to hardcode experiment name for AML submission via an
environment variable
- Add a PR job that looks for running AML jobs and cancels them
- Add timeouts for all smoke test jobs
- Add coordinates to slides pipeline
- Refactor DeepMILModule into a single class
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Refactor LoadRoiD/LoadPandaRoid transforms to work with OpenSlide and
removed params redundancy with LoadingParams class.
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Upgrade monai to 1.0.1
Note that we now need to convert monai outputs to torch tensors since
Rand/GridPatchd transforms return MetaTensors a new data structure in
the latest releases. This is handled in the collate function
Main changes:
- Change prints to logging.info to prevent all ranks from logging
- Run smoke tests with --striclty_aml_v1 flag to be able to wait for
completion
- Set progress=False of LoadTilesBtachd to avoid unnecessary verbosity
in logs
- Fix regression after merge: kill_ddp_processes right after
run_training because we run extra val epoch on a single device
- Remove a test that does nothing
- Use binary/multiclass metrics
- Ignore PL warnings
v2 get_ml_client and therefore get_credential gets called when using
--strictly_azureml_v1. Here we move this call to v2 scope only to be
able to use v1 strictly
log_hyperparameters fails with ChainedTokenCredential failure because we
were using an mlflow client that is not linked to the local tracking uri
so it was trying to authenticate to Azure to retrieve the run.
Add openslide to `hi-ml-cpath conda` environment, so that code can use
either `cucim` or `openslide` as backend.
Fixed a bug in github action `prepare_cpath_environment`: updated environment was not getting activated
Tests for histopathology use a hacky environment build from only
the pip section of the Conda environment. Switch tests to using a full Conda environment.
In this PR:
- Fix is introduced to handle missing cross-validation rounds (e.g., when a round fails)
- Extra validation epoch is handled
- Downloading the AML metrics json and saving it as a dataframe is separated into two functions (current version gives a `KeyError` when the json is downloaded for the first time)
Co-authored-by: Harshita Sharma <t-hsharma@microsoft.com>
Enable checkpoints transfer across AML workspaces
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
Dissociate thumbnails and heatmaps plots to flexibly remove/add each of
these plot options.
Motivations: thumbnail plots can be costly, so we might want to omit it
sometime while including heatmap plots and vice versa.
Kill DDP processes after run_training and initialize a new trainer
instance with a single device for inference
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
I accidentally created a v2 data asset with the same name as a v1
dataset in our test suite. The data asset cannot be removed or renamed,
hence I am renaming the v1 dataset in the test to get it passing again
This solves the error that we encounter when submitting a cross validation run with Amulter via hi-ml runner that expect a cluster argument to be able to launch a crossvalidation. This is necessary as k folds splits are enabled for crossval_count > 1.
Fernando Pérez-García
Kenza Bouzid
Melissa Bristow
Anton Schwaighofer
Kenza Bouzid
Harshita Sharma
maxilse