* Merged PR 42: Python package structure
Created Python package structure
* Merged PR 50: Röth-Tarantola generative model for velocities
- Created Python package structure for generative models for velocities
- Implemented the [Röth-Tarantola model](https://doi.org/10.1029/93JB01563)
* Merged PR 51: Isotropic AWE forward modelling using Devito
Implemented forward modelling for the isotropic acoustic wave equation using [Devito](https://www.devitoproject.org/)
* Merged PR 52: PRNG seed
Exposed PRNG seed in generative models for velocities
* Merged PR 53: Docs update
- Updated LICENSE
- Added Microsoft Open Source Code of Conduct
- Added Contributing section to README
* Merged PR 54: CLI for velocity generators
Implemented CLI for velocity generators
* Merged PR 69: CLI subpackage using Click
Reimplemented CLI as subpackage using Click
* Merged PR 70: VS Code settings
Added VS Code settings
* Merged PR 73: CLI for forward modelling
Implemented CLI for forward modelling
* Merged PR 76: Unit fixes
- Changed to use km/s instead of m/s for velocities
- Fixed CLI interface
* Merged PR 78: Forward modelling CLI fix
* Merged PR 85: Version 0.1.0
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* adding cgmanifest to staging
* adding a yml file with CG build task
* added prelim NOTICE file
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Merged PR 126: updated notice file with previously excluded components
updated notice file with previously excluded components
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* added cela copyright headers to all non-empty .py files (#3)
* switched to ACR instead of docker hub (#4)
* sdk.v1.0.69, plus switched to ACR push. ACR pull coming next
* full acr use, push and pull, and use in Estimator
* temp fix for dcker image bug
* fixed the az acr login --username and --password issue
* full switch to ACR for docker image storage
* Vapaunic/metrics (#1)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* BUILD: added build setup files. (#5)
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added pytest to environmetn, and pytest job to the main build (#18)
* Update main_build.yml for Azure Pipelines
* minor stylistic changes (#19)
* Update main_build.yml for Azure Pipelines
Added template for integration tests for scripts and experiments
Added setup and env
Increased job timeout
added complete set of tests
* BUILD: placeholder for Azure pipelines for notebooks build.
BUILD: added notebooks job placeholders.
BUILD: added github badgets for notebook builds
* CLEANUP: moved non-release items to contrib (#20)
* Updates HRNet notebook 🚀 (#25)
* Modifies pre-commit hook to modify output
* Modifies the HRNet notebook to use Penobscot dataset
Adds parameters to limit iterations
Adds parameters meta tag for papermil
* Fixing merge peculiarities
* Updates environment.yaml (#21)
* Pins main libraries
Adds cudatoolkit version based on issues faced during workshop
* removing files
* Updates Readme (#22)
* Adds model instructions to readme
* Update README.md (#24)
I have collected points to all of our BP repos into this central place. We are trying to create links between everything to draw people from one to the other.
Can we please add a pointer here to the readme?
I have spoken with Max and will be adding Deep Seismic there once you have gone public.
* CONTRIB: cleanup for imaging. (#28)
* Create Unit Test Build.yml (#29)
Adding Unit Test Build.
* Update README.md
* Update README.md
* Create Unit Test Build.yml (#29)
Adding Unit Test Build.
Update README.md
Update README.md
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* TESTS: added notebook integreation tests. (#65)
* TESTS: added notebook integreation tests.
* TEST: typo in env name
* Addressing a number of minor issues with README and broken links (#67)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fix for seyviewer and mkdir splits in README + broken link in F3 notebook
* issue edits to README
* download complete message
* Added Yacs info to README.md (#69)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added info on yacs files
* MODEL.PRETRAINED key missing in default.py (#70)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added MODEL.PRETRAINED key to default.py
* Update README.md (#59)
* Update README.md (#58)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* Adds premium storage (#79)
* Adds premium storage method
* update test.py for section based approach to use command line arguments (#76)
* added README documentation per bug bush feedback (#78)
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* https://github.com/microsoft/DeepSeismic/issues/71 (#80)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* addressing multiple issues from first bug bash (#81)
* added README documentation per bug bush feedback
* DOC: added HRNET download info to README
* added hrnet download script and tested it
* added legal headers to a few scripts.
* changed /data to ~data in the main README
* added Troubleshooting section to the README
* Dciborow/build bug (#68)
* Update unit_test_steps.yml
* Update environment.yml
* Update setup_step.yml
* Update setup_step.yml
* Update unit_test_steps.yml
* Update setup_step.yml
* Adds AzureML libraries (#82)
* Adds azure dependencies
* Adds AzureML components
* Fixes download script (#84)
* Fixes download script
* Updates readme
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* modified hrnet notebook, addressing bug bash issues (#95)
* Update environment.yml (#93)
* Update environment.yml
* Update environment.yml
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* notebook integration tests complete (#106)
* added README documentation per bug bush feedback
* HRNet notebook works with tests now
* removed debug material from the notebook
* corrected duplicate build names
* conda init fix
* changed setup deps
* fixed F3 notebook - merge conflict and pytorch bug
* main and notebook builds have functional setup now
* Mat/test (#105)
* added README documentation per bug bush feedback
* Modifies scripts to run for only afew iterations when in debug/test mode
* Updates training scripts and build
* Making names unique
* Fixes conda issue
* HRNet notebook works with tests now
* removed debug material from the notebook
* corrected duplicate build names
* conda init fix
* Adds docstrings to training script
* Testing somehting out
* testing
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* adds seresnet
* Modifies to work outside of git env
* test
* test
* Fixes typo in DATASET
* reducing steps
* test
* test
* fixes the argument
* Altering batch size to fit k80
* reducing batch size further
* test
* test
* test
* test
* fixes distributed
* test
* test
* adds missing import
* Adds further tests
* test
* updates
* test
* Fixes section script
* test
* testing everyting once through
* Final run for badge
* changed setup deps, fixed F3 notebook
* Adds missing tests (#111)
* added missing tests
* Adding fixes for test
* reinstating all tests
* Maxkaz/issues (#110)
* added README documentation per bug bush feedback
* added missing tests
* closing out multiple post bug bash issues with single PR
* Addressed comments
* minor change
* Adds Readme information to experiments (#112)
* Adds readmes to experiments
* Updates instructions based on feedback
* Update README.md
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88) (#2)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88) (#3)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* Remove related projects on AI Labs
* Added a reference to Azure machine learning (#115)
Added a reference to Azure machine learning to show how folks can get started with using Azure Machine Learning
* Update README.md
* update fork from upstream (#4)
* fixed merge conflict resolution in LICENSE
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* Remove related projects on AI Labs
* Added a reference to Azure machine learning (#115)
Added a reference to Azure machine learning to show how folks can get started with using Azure Machine Learning
* Update README.md
* Update AUTHORS.md (#117)
* Update AUTHORS.md (#118)
* pre-release items (#119)
* added README documentation per bug bush feedback
* added missing tests
* closing out multiple post bug bash issues with single PR
* new badges in README
* cleared notebook output
* notebooks links
* fixed bad merge
* forked branch name is misleading. (#116)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88) (#2)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* fixed link for F3 download
* MINOR: python version fix to 3.6.7 (#72)
* Adding system requirements in README (#74)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* DOC: forking dislaimer and new build names. (#9)
* Updating README.md with introduction material (#10)
* Update README with introduction to DeepSeismic
Add intro material for DeepSeismic
* Adding logo file
* Adding image to readme
* Update README.md
* Updates the 3D visualisation to use itkwidgets (#11)
* Updates notebook to use itkwidgets for interactive visualisation
* Adds jupytext to pre-commit (#12)
* Add jupytext
* Adds demo notebook for HRNet (#13)
* Adding TF 2.0 to allow for tensorboard vis in notebooks
* Modifies hrnet config for notebook
* Add HRNet notebook for demo
* Updates HRNet notebook and tidies F3
* removed my username references (#15)
* moving 3D models into contrib folder (#16)
* Weetok (#17)
* Update it to include sections for imaging
* Update README.md
* Update README.md
* added system requirements to readme
* sdk 1.0.76; tested conda env vs docker image; extented readme
* removed reference to imaging
* minor md formatting
* minor md formatting
* clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 - Issue #83
* Add Troubleshooting section for DSVM warnings #89
* Add Troubleshooting section for DSVM warnings, plus typo #89
* tested both yml conda env and docker; udated conda yml to have docker sdk
* tested both yml conda env and docker; udated conda yml to have docker sdk; added
* NVIDIA Tesla K80 (or V100 GPU for NCv2 series) - per Vanja's comment
* Update README.md
* BugBash2 Issue #83 and #89: clarify which DSVM we want to use - Ubuntu GPU-enabled VM, preferably NC12 (#88) (#3)
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing
* merge upstream into my fork (#1)
* MINOR: addressing broken F3 download link (#73)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build stat…
* Minor fix: broken links in README (#120)
* fully-run notebooks links and fixed contrib voxel models (#123)
* added README documentation per bug bush feedback
* added missing tests
* - added notebook links
- made sure orginal voxel2pixel code runs
* update ignite port of texturenet
* resolved merge conflict
* formatting change
* Adds reproduction instructions to readme (#122)
* Update main_build.yml for Azure Pipelines
* Update main_build.yml for Azure Pipelines
* BUILD: added build status badges (#6)
* Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7)
* Finished version of numpy data loader
* Working training script for demo
* Adds the new metrics
* Fixes docstrings and adds header
* Removing extra setup.py
* Log config file now experiment specific (#8)
* Merging work on salt dataset
* Adds computer vision to dependencies
* Updates dependecies
* Update
* Updates the environemnt files
* Updates readme and envs
* Initial running version of dutchf3
* INFRA: added structure templates.
* VOXEL: initial rough code push - need to clean up before PRing.
* Working version
* Working version before refactor
* quick minor fixes in README
* 3D SEG: first commit for PR.
* 3D SEG: removed data files to avoid redistribution.
* Updates
* 3D SEG: restyled batch file, moving onto others.
* Working HRNet
* 3D SEG: finished going through Waldeland code
* Updates test scripts and makes it take processing arguments
* minor update
* Fixing imports
* Refactoring the experiments
* Removing .vscode
* Updates gitignore
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script
* minor wording fix
* minor wording fix
* enabled splitting dataset into sections, rather than only patches
* enabled splitting dataset into sections, rather than only patches
* merged duplicate ifelse blocks
* merged duplicate ifelse blocks
* refactored prepare_data.py
* refactored prepare_data.py
* added scripts for section train test
* added scripts for section train test
* section train/test works for single channel input
* section train/test works for single channel input
* Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py
This PR includes the following changes:
- added README instructions for running f3dutch experiments
- prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic.
- ran black formatter on the file, which created all the formatting changes (sorry!)
* Merged PR 204: Adds loaders to deepseismic from cv_lib
* train and test script for section based training/testing
* train and test script for section based training/testing
* Merged PR 209: changes to section loaders in data.py
Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts:
- get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders.
- SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py
h, w = img.shape[-2], img.shape[-1] # height and width
* Merged PR 210: BENCHMARKS: added placeholder for benchmarks.
BENCHMARKS: added placeholder for benchmarks.
* Merged PR 211: Fixes issues left over from changes to data.py
* removing experiments from deep_seismic, following the new struct
* removing experiments from deep_seismic, following the new struct
* Merged PR 220: Adds Horovod and fixes
Add Horovod training script
Updates dependencies in Horovod docker file
Removes hard coding of path in data.py
* section train/test scripts
* section train/test scripts
* Add cv_lib to repo and updates instructions
* Add cv_lib to repo and updates instructions
* Removes data.py and updates readme
* Removes data.py and updates readme
* Updates requirements
* Updates requirements
* Merged PR 222: Moves cv_lib into repo and updates setup instructions
* renamed train/test scripts
* renamed train/test scripts
* train test works on alaudah section experiments, a few minor bugs left
* train test works on alaudah section experiments, a few minor bugs left
* cleaning up loaders
* cleaning up loaders
* Merged PR 236: Cleaned up dutchf3 data loaders
@<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments.
The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders.
This will affect your code if you access these attributes. E.g. if you have something like this in your experiments:
```
train_set = TrainPatchLoader(…)
patches = train_set.patches[train_set.split]
```
or
```
train_set = TrainSectionLoader(…)
sections = train_set.sections[train_set.split]
```
* training testing for sections works
* training testing for sections works
* minor changes
* minor changes
* reverting changes on dutchf3/local/default.py file
* reverting changes on dutchf3/local/default.py file
* added config file
* added config file
* Updates the repo with preliminary results for 2D segmentation
* Merged PR 248: Experiment: section-based Alaudah training/testing
This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment.
* Merged PR 253: Waldeland based voxel loaders and TextureNet model
Related work items: #16357
* Merged PR 290: A demo notebook on local train/eval on F3 data set
Notebook and associated files + minor change in a patch_deconvnet_skip.py model file.
Related work items: #17432
* Merged PR 312: moved dutchf3_section to experiments/interpretation
moved dutchf3_section to experiments/interpretation
Related work items: #17683
* Merged PR 309: minor change to README to reflect the changes in prepare_data script
minor change to README to reflect the changes in prepare_data script
Related work items: #17681
* Merged PR 315: Removing voxel exp
Related work items: #17702
* sync with new experiment structure
* sync with new experiment structure
* added a logging handler for array metrics
* added a logging handler for array metrics
* first draft of metrics based on the ignite confusion matrix
* first draft of metrics based on the ignite confusion matrix
* metrics now based on ignite.metrics
* metrics now based on ignite.metrics
* modified patch train.py with new metrics
* modified patch train.py with new metrics
* Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo.
Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script.
Related work items: #18264
* modified metrics with ignore_index
* modified metrics with ignore_index
* Merged PR 405: minor mods to notebook, more documentation
A very small PR - Just a few more lines of documentation in the notebook, to improve clarity.
Related work items: #17432
* Merged PR 368: Adds penobscot
Adds for penobscot
- Dataset reader
- Training script
- Testing script
- Section depth augmentation
- Patch depth augmentation
- Iinline visualisation for Tensorboard
Related work items: #14560, #17697, #17699, #17700
* Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators
Related work items: #16362
* Merged PR 452: decouple docker image creation from azureml
removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb
All other changes are due to trivial reruns
Related work items: #18346
* Merged PR 512: Pre-commit hooks for formatting and style checking
Opening this PR to start the discussion -
I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added:
- .pre-commit-config.yaml - defines git hooks to be installed
- .flake8 - settings for flake8 linter
- pyproject.toml - settings for black formatter
The last two files define the formatting and linting style we want to enforce on the repo.
All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors.
Some questions to start the discussion:
- Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that.
- Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file.
- Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this?
Thanks!
Related work items: #18350
* Merged PR 513: 3D training script for Waldeland's model with Ignite
Related work items: #16356
* Merged PR 565: Demo notebook updated with 3D graph
Changes:
1) Updated demo notebook with the 3D visualization
2) Formatting changes due to new black/flake8 git hook
Related work items: #17432
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* Merged PR 341: Tests for cv_lib/metrics
This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged.
I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing.
Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest.
Related work items: #16955
* merged tests into this branch
* merged tests into this branch
* Merged PR 569: Minor PR: change to pre-commit configuration files
Related work items: #18350
* Merged PR 586: Purging unused files and experiments
Purging unused files and experiments
Related work items: #20499
* moved prepare data under scripts
* moved prepare data under scripts
* removed untested model configs
* removed untested model configs
* fixed weird bug in penobscot data loader
* fixed weird bug in penobscot data loader
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* penobscot experiments working for hrnet, seresnet, no depth and patch depth
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* removed a section loader bug in the penobscot loader
* fixed bugs in my previous 'fix'
* fixed bugs in my previous 'fix'
* removed redundant _open_mask from subclasses
* removed redundant _open_mask from subclasses
* Merged PR 601: Fixes to penobscot experiments
A few changes:
- Instructions in README on how to download and process Penobscot and F3 2D data sets
- moved prepare_data scripts to the scripts/ directory
- fixed a weird issue with a class method in Penobscot data loader
- fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue)
- removed config files that were not tested or working in Penobscot experiments
- modified default.py so it's working if train.py ran without a config file
Related work items: #20694
* Merged PR 605: added common metrics to Waldeland model in Ignite
Related work items: #19550
* Removed redundant extract_metric_from
* Removed redundant extract_metric_from
* formatting changes in metrics
* formatting changes in metrics
* modified penobscot experiment to use new local metrics
* modified penobscot experiment to use new local metrics
* modified section experimen to pass device to metrics
* modified section experimen to pass device to metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* moved metrics out of dutchf3, modified distributed to work with the new metrics
* fixed other experiments after new metrics
* fixed other experiments after new metrics
* removed apex metrics from distributed train.py
* removed apex metrics from distributed train.py
* added ignite-based metrics to dutch voxel experiment
* added ignite-based metrics to dutch voxel experiment
* removed apex metrics
* removed apex metrics
* modified penobscot test script to use new metrics
* pytorch-ignite pre-release with new metrics until stable available
* removed cell output from the F3 notebook
* deleted .vscode
* modified metric import in test_metrics.py
* separated metrics out as a module
* relative logger file path, modified section experiment
* removed the REPO_PATH from init
* created util logging function, and moved logging file to each experiment
* modified demo experiment
* modified penobscot experiment
* modified dutchf3_voxel experiment
* no logging in voxel2pixel
* modified dutchf3 patch local experiment
* modified patch distributed experiment
* modified interpretation notebook
* minor changes to comments
* Updates notebook to use itkwidgets for interactive visualisation
* Further updates
* Fixes merge conflicts
* removing files
* Adding reproduction experiment instructions to readme
* checking in ablation study from ilkarman (#124)
tests pass but final results aren't communicated to github. No way to trigger another commit other than to do a dummy commit
maxkazmsft