Add custom forward that checkpoints activations to trade compute for
memory
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
LoadMaskSubROId loads a sub section from a slide based a foreground png
mask with categorical labels for each tissue section.
---------
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
In this PR:
- `stratify_plots_by` parameter is introduced in BaseMIL to specify
metadata column on which the output PR and ROC curves can be stratified
- Functionality is added to plot the output curves stratified by
metadata
- Corresponding tests are added
---------
Co-authored-by: Harshita Sharma <t-hsharma@microsoft.com>
ROC curve is added as a plot option. Tests are updated for ROC and PR
curves
---------
Co-authored-by: Harshita Sharma <t-hsharma@microsoft.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
We discovered a bug after merging #754 val_set_is_dist was used as a
flag to gather_results. It was defined as
val_set_is_dist = pl_replace_sampler_ssp and max_num_gpus > 1
However max_num_gpus is set to -1 by default. So val_set_is_dist was
always False.
Now that we deprecated this flag results got gathered. This blows up the
memory when we use very large bag_size (1200).
Therefore we only gather if save_intermediate_outputs is True or if it's
an extra validation epoch.
A test is added to make sure gather_results is called when expected only
-No need to check if validation set is distributed now that we can shard
evaluation (test and val) dataloaders without duplications #743 (it will
be distributed anyways)
-Tests tiles_selction, outputs_gathering and loss_callback are updated
to account for uneven samples scenario
Current transform `SubSampled` to sample and shuffle the tiles does so
with replacement. In this PR, we ensure that tiles are sampled without
replacement.
Co-authored-by: Harshita Sharma <t-hsharma@microsoft.com>
Re-enable tests that had been disabled in #487. For that, remove the
code duplication we had in getting an AML workspace specifically for
unit tests. `get_workspace` now can read settings from environment
variables.
- Add attention histogram plot option
- Add original slide without overlay to heatmaps plots
- Change colormap from Reds to Spectral_r
- Remove rectangle borders for smoother heatmaps
Jobs that get interrupted after training and then re-start will not find
an inference checkpoint and fail. This PR adds logic to search for that
checkpoint in AzureML.
Closes#716
Test with OpenSlide backend as well now that the tiffiles are not
corrupted anymore -> properly compressed and saved in chunks by
specifying tile_size in tiffwriter options
Add TimerWrapper transform and elapsed_timer + refactor load to print
warnings about failure slides
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
* Run `pytest` and `pytest` on GPUs in parallel rather than sequentially.
* Cancel existing runs from previous pushes
* Fix dependency problem in hi-ml PR
* Reduce logging noise from `numba`
- Add coordinates to slides pipeline
- Refactor DeepMILModule into a single class
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Refactor LoadRoiD/LoadPandaRoid transforms to work with OpenSlide and
removed params redundancy with LoadingParams class.
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Upgrade monai to 1.0.1
Note that we now need to convert monai outputs to torch tensors since
Rand/GridPatchd transforms return MetaTensors a new data structure in
the latest releases. This is handled in the collate function
In this PR:
- Fix is introduced to handle missing cross-validation rounds (e.g., when a round fails)
- Extra validation epoch is handled
- Downloading the AML metrics json and saving it as a dataframe is separated into two functions (current version gives a `KeyError` when the json is downloaded for the first time)
Co-authored-by: Harshita Sharma <t-hsharma@microsoft.com>
Enable checkpoints transfer across AML workspaces
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
Dissociate thumbnails and heatmaps plots to flexibly remove/add each of
these plot options.
Motivations: thumbnail plots can be costly, so we might want to omit it
sometime while including heatmap plots and vice versa.
Add LossAnalysisCallback to plot loss heatmaps + scatter plots and save
loss ranks and their statistics.
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add commandline arguments to parametrise child runs arg identifier and primary metric to get best_epoch.
* Rename all crossval functions to hyperdrive ones to make it generic
For small validation sets and poor performance models, we sometimes end up with some slides falsely belonging to TP/FN cases.
We need to make sure thay top_heaps contain only true cases and bottom ones, only false cases.
Check predicted vs true label before pushing into slides_heaps to make sure we only push true predictions in top_slides_heaps and false ones in bottom_slides_heaps
In this PR:
- transformer_dropout parameter is added to TransformerPooling and TransformerPoolingBenchmark pooling layers.
- Tests are updated with the transformer_dropout parameter.
Kenza Bouzid
Peter Hessey
Anton Schwaighofer
Harshita Sharma
Kenza Bouzid
vale-salvatelli
Melissa Bristow