* cleaning up files which are no longer needed
* fixes after removing forking workflow (#322)
* PR to resolve merge issues
* updated main build as well
* added ability to read in git branch name directly
* manually updated the other files
* fixed number of classes for main build tests (#327)
* fixed number of classes for main build tests
* corrected DATASET.ROOT in builds
* added dev build script
* Fixes for development inside the docker container (#335)
* Fix the mound command for the HRNet pretrained model in the docker readme
* Properly catch InvalidGitRepository exception
* make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed
* Properly catch InvalidGitRepository exception in train.py
* Readme update (#337)
* README updates
* Removing user specific path from config
Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com>
* Fixing #324 and #325 (#338)
* update colormap to a non-discrete one -- fixes#324
* fix mask_to_disk to normalize by n_classes
* changes to test.py
* Updating data.py
* bug fix
* increased timeout time for main_build
* retrigger build
* retrigger the build
* increase timeout
* fixes 318 (#339)
* finished 318
* increased checkerboard test timeout
* fix 333 (#340)
* added label correction to train gradient
* changing the gradient data generator to take inline/crossline argument conssistent with the patchloader
* changing variable name to be more descriptive
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* bug fix to model predictions (#345)
* replace hrnet with seresnet in experiments - provides stable default model (#343)
* PR to fix#342 (#347)
* intermediate work for normalization
* 1) normalize function runs based on global MIN and MAX 2) has a error handling for division by zero, np.finfo 3) decode_segmap normalizes the label/mask based on the n_calsses
* global normalization added to test.py
* increasing the threshold on timeout
* trigger
* revert
* idk what happened
* increase timeout
* picking up global min and max
* passing config to TrainPatchLoader to facilitate access to global min and max and other attr in low level functions, WIP
* removed print statement
* changed section loaders
* updated test for min and max from config too
* adde MIN and MAX to config
* notebook modified for loaders
* another dataloader in notebook
* readme update
* changed the default values for min max, updated the docstring for loaders, removed suppressed lines
* debug
* merging work from CSE team into main staging branch (#357)
* Adding content to interpretation README (#171)
* added sharat, weehyong to authors
* adding a download script for Dutch F3 dataset
* Adding script instructions for dutch f3
* Update README.md
prepare scripts expect root level directory for dutch f3 dataset. (it is downloaded into $dir/data by the script)
* Adding readme text for the notebooks and checking if config is correctly setup
* fixing prepare script example
* Adding more content to interpretation README
* Update README.md
* Update HRNet_Penobscot_demo_notebook.ipynb
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* Updates to prepare dutchf3 (#185)
* updating patch to patch_size when we are using it as an integer
* modifying the range function in the prepare_dutchf3 script to get all of our data
* updating path to logging.config so the script can locate it
* manually reverting back log path to troubleshoot build tests
* updating patch to patch_size for testing on preprocessing scripts
* updating patch to patch_size where applicable in ablation.sh
* reverting back changes on ablation.sh to validate build pass
* update patch to patch_size in ablation.sh (#191)
Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com>
* TestLoader's support for custom paths (#196)
* Add testloader support for custom paths.
* Add test
* added file name workaround for Train*Loader classes
* adding comments and clean up
* Remove legacy code.
* Remove parameters that dont exist in init() from documentation.
* Add unit tests for data loaders in dutchf3
* moved unit tests
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* select contiguous data splits for val and train (#200)
* select contiguous data splits for test and train
* changed data-dir to data_dir as arg to prepare_dutchf3.py
* update script with new required parameter label_file
* ignoring split_alaudah_et_al_19 as it is not updated
* changed TEST to VALIDATION for clarity in the code
* included job to run scripts unit test
* Fix val/train split and add tests
* adjust to consider the whole horz_lines
* update environment - gitpython version
* Segy Converter Utility (#199)
* Add convert_segy utility script and related notebooks
* add segy files to .gitignore
* readability update
* Create methods for normalizing and clipping separately.
* Add comment
* update file paths
* cleanup tests and terminology for the normalization/clipping code
* update notes to provide more context for using the script
* Add tests for clipping.
* Update comments
* added Microsoft copyright
* Update root README
* Add a flag to turn on clipping in dataprep script.
* Remove hard coded values and fix _filder_data method.
* Fix some minor issues pointed out on comments.
* Remove unused lib.
* Rename notebooks to impose order; set env; move all def funtions into utils; improve comments in notebooks; and include code example to run prepare_dutchf3.py
* Label missing data with 255.
* Remove cell with --help command.
* Add notebooks to test pipeline.
* grammer edits
* update notebook output and utils naming
* fix output dir error and cleanup notebook
* fix yaml indent error in notebooks_build.yml
* fix merge issues and job name errors
* debugging the build pipeline
* combine notebook tests for segy converter since they are dependent on each other
Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com>
* Azureml train pipeline (#195)
* initial add of azure ml pipeline
* update references and dependencies
* fix integration tests
* remove incomplete tests
* add azureml requirements.txt for dutchf3 local patch and update pipeline config
* add empty __init__.py to cv_lib dutchf3
* Get train,py to run in pipeline
* allow output dir in train.py
* Clean up README and __init__
* only pass output if available and use input dir for output in train.py
* update comment in train.py
* updating azureml_requirements to only pull from /master
* removing windows guidance in azureml_pipelines/README.md
* adding .env.example
* adding azureml config example
* updating documentation in azureml_pipelines README.md
* updating main README.md to refer to AML guidance documentation
* updating AML README.md to include additional guidance to cancel runs
* adding documentation on AzureML pipelines in the AML README.me
* adding files needed section for AML training run
* including hyperlink in format poiniting to additional detail on Azure Machine Learning pipeslines in AML README.md
* removing the mention of VSCode in the AML README.md
* fixing typo
* modifying config to pipeline configuration in README.md
* fixing typo in README.md
* adding documentation on how to create a blob container and copy data onto it
* adding documentation on blob storage guidance
* adding guidance on how to get the subscription id
* adding guidance to activate environment and then run the kick off train pipeline from ROOT
* adding ability to pass in experiement name and different pipeline configuration to kickoff_train_pipeline.py
* adding Microsoft Corporation Copyright to kickoff_train_pipeline.py
* fixing format in README.md
* adding trouble shooting section in README.md for connection to subscription
* updating troubleshooting title
* adding guidance on how to download the config.json from the Azure Portal in the README.md
* adding additional guidance and information on AzureML compute targets and naming conventions
* changing the configuation file example to only include the train step that is currently supported
* updating config to pipeline configuration when applicable
* adding link to Microsoft docs for additional information on pipeline steps
* updated AML test build definitions
* updated AML test build definitions
* adding job to aml_build.yml
* updating example config for testing
* modifying the test_train_pipeline.py to have appropriate number of pipeline steps and other required modifications
* updating AML_pipeline_tests in aml_build.yml to consume environment variables
* updating scriptType, sciptLocation, and inlineScript in aml_build.yml
* trivial commit to re-trigger broken build pipelines
* fix to aml yml build to use env vars for secrets and everything else
* another yml fix
* another yml fix
* reverting structure format of jobs for aml_build pipeline tests
* updating path to test_train_pipeline.py
* aml_pipeline_tests timed out, extending timeoutInMinutes from 10 to 40
* adding additional pytest
* adding az login
* updating variables in aml pipeline tests
Co-authored-by: Anna Zietlow <annamzietlow@gmail.com>
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* moved contrib contributions around from CSE
* fixed dataloader tests - updated them to work with new code from staging branch
* segyconverter notebooks and tests run and pass; updated documentation
* added test job for segy converter notebooks
* removed AML training pipeline from this release
* fixed training model tolerance precision in the tests - wasn't working
* fixed train.py build issues after the merge
* addressed PR comments
* fixed bug in check_performance
Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com>
Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com>
Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com>
Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com>
Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com>
Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com>
Co-authored-by: Anna Zietlow <annamzietlow@gmail.com>
* make tests simpler (#368)
* removed Dutch F3 job from main_build
* fixed a bug in data subset in debug mode
* modified epoch numbers to pass the performance checks, checkedout check_performance from Max's branch
* modified get_data_for_builds.sh to set up checkerboard data for smaller size, minor improvements on gen_checkerboard
* send all the batches, disabled the performance checks for patch_deconvnet
* added comment to enable tests for patch_deconvnet after debugging, renamed gen_checkerboard, added options to new arg per Max's suggestion
* Replace HRNet with SEResNet model in the notebook (#362)
* replaced HRNet with SEResNet model in the notebook
* removed debugging cell info
* fixed bug where resnet_unet model wasn't loading the pre-trained version in the notebook
* fixed build VM problems
* Multi-GPU training support (#359)
* Data flow tests (#375)
* renamed checkerboard job name
* restructured default outputs from test.py to be dumped under output dir and not debug dir
* test.py output re-org
* removed outdated variable from check_performance.py
* intermediate work
* intermediate work
* bunch of intermediate works
* changing args for different trainings
* final to run dev_build"
* remove print statements
* removed print statement
* removed suppressed lines
* added assertion error msg
* added assertion error msg, one intential bug to test
* testing a stupid bug
* debug
* omg
* final
* trigger build
* fixed multi-GPU termination in train.py (#379)
* PR to fix#371 and #372 (#380)
* added learning rate to logs
* changed epoch for patch_deconvnet, and enabled the tests
* removed TODOs
* changed tensorflow pinned version (#387)
* changed tensorflow pinned version
* trigger build
* closes 385 (#389)
* Fixing #259 by adding symmetric padding along depth direction (#386)
* BYOD Penobscot (#390)
* minor updates to files
* added penobscot conversion code
* docker build test (#388)
* added a new job to test bulding the docker, for now it is daisy-chained to the end
* this is just a TEST
* test
* test
* remove old image
* debug
* debug
* test
* debug
* enabled all the jobs
* quick fix
* removing non-tagged iamges
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* added missing license headers and fixed formatting (#391)
* added missing license headers and fixed formatting
* some more license headers
* updated documentation to close 354 and 381 (#392)
* fix test.py and notebook issues (#394)
* resolved conflicts for 0.2 release (#396)
* V00.01.00003 release (#356)
* cleaning up files which are no longer needed
* fixes after removing forking workflow (#322)
* PR to resolve merge issues
* updated main build as well
* added ability to read in git branch name directly
* manually updated the other files
* fixed number of classes for main build tests (#327)
* fixed number of classes for main build tests
* corrected DATASET.ROOT in builds
* added dev build script
* Fixes for development inside the docker container (#335)
* Fix the mound command for the HRNet pretrained model in the docker readme
* Properly catch InvalidGitRepository exception
* make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed
* Properly catch InvalidGitRepository exception in train.py
* Readme update (#337)
* README updates
* Removing user specific path from config
Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com>
* Fixing #324 and #325 (#338)
* update colormap to a non-discrete one -- fixes#324
* fix mask_to_disk to normalize by n_classes
* changes to test.py
* Updating data.py
* bug fix
* increased timeout time for main_build
* retrigger build
* retrigger the build
* increase timeout
* fixes 318 (#339)
* finished 318
* increased checkerboard test timeout
* fix 333 (#340)
* added label correction to train gradient
* changing the gradient data generator to take inline/crossline argument conssistent with the patchloader
* changing variable name to be more descriptive
Co-authored-by: maxkazmsft <maxkaz@microsoft.com>
* bug fix to model predictions (#345)
* replace hrnet with seresnet in experiments - provides stable default model (#343)
Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com>
Co-authored-by: Fatemeh <fazamani@microsoft.com>
* typos
Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com>
Co-authored-by: Fatemeh <fazamani@microsoft.com>
Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com>
Co-authored-by: Fatemeh <fazamani@microsoft.com>
Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com>
Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com>
Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com>
Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com>
Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com>
Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com>
Co-authored-by: Anna Zietlow <annamzietlow@gmail.com>
080cf46fe9