* Remove dead code and fix file-cache usage percentage check UT
* Add coverage test for gen-test-config
* Adding code coverage for doc command
* Adding build script to build with different options
* Delete old binary before regenerating
* Delete old binaries before regenerating
* Adding pipeline test cases
* Remove agent cleanup steps as 1ES is spawning new vms for every run
* Disable sas validation on ubn-22 for now
* Disable sas validation on ubn-22 for now
* Disable sas validation on ubn-22 for now
* Removing ubn-22 from all pipelines
* Correcting adls sas param
* Re-add sas validation for all ubn distros
* Disable chmod test cases for SAS auth
* Set default value for sas to be false
* Converting account sas to container sas for chmod test to work
* Enable e2e test again
* Correcting e2e failure with chmod test case
* Add Ubn-22 to release pipeline
* Change pool of ubnuntu to ms hosted
* Remove dead param from yaml file
* Adding Ubn-22 to pipelines
* Adding lint as standard ci step
* correcting lint command path
* Add copyright checks to ci pipeline
* Copyright fixes and ci check for copyright statements
* Adding go format check to CI
* Fixing some lint issues
* More lint fixes
* More lint fixes
* Fixing some lint issues
* Correcting code linting check in ci
* correcting lint status check
* correcting lint status check
* correcting lint status check
* correcting lint status check
* correcting lint status check
* correcting lint status check
* correcting lint status check
* correcting bugs reported by sa check
* notices_fix.sh update
* Add step to validate NOTICE file
* Add step to release for ubn-22
* Add logs for success
* Fixing copyright bug
* Correcting tag formats
* Correcting percentage check
* Spell error correction
* Ignore exectime from code coverage
* Correcting log string
* Removing dead file
* Correcting go format error
* Adding test cases for mount command
* Adding mount test to nightly pipeline
* Adding test case where the config file is not provided
* Typo fix
* Adding test case for mounting using environment variables
* added new pipeline
* python cleanup
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* test run
* removed print
* testing binary
* testing binary
* testing binary
* testing binary
* testing binary
* testing binary
* testing
* testing
* testing
* testing
* testing
* finished
* finished
* finished
* attempting to fix py issue
* cleanup
* testing
* removed direct read
* removed direct read
* removed direct read
* removed direct read
* fail if regression is too bad
* cleanup
* added dependency step to pipeline
* cleanup
* Adding v2 galactus from devops
* Copyright update
* Correction baseconfig file
* Renaming test libs to extension
* Remvoing binary
* Sync with main and updates as per review comments
Ubn18/20 giving error in pipelines as cmake version is outdated now. Upgrading version in all CMake files and disabling GTest as they are not compiling with newer version
* Set retry policy to explonential for adls account
* Encode % in the file name
* Removing temp file
* Print permission string in case of failure
* Do not trucate file if the size remains same
* acl.Permissions returning extra bits for sticky. Ignore these bits as blobfuse does not support them
* Make % encoding in file name to be a configurable option
* Run multi special char file name test only for adls
* Do not upload if truncate results in same file size or truncate is setting size to 0 for a new file
* Maintain last upload time to stop double uploads in case of truncate and flush
* Encode % in url for folder name and other things
* no need to encode prefix seperatly as now entire url will be encoded
* Unused reference removed
* Correcting nightly yml error
* Increasing timeout for nightly
* Run multi special char file test for block blob as well
* Add test case of directory with special characters
* Search the file under directory where it was created
* Add file cache timeout to be 0 in feature_test
* Add file cache timeout to be 0 in feature_test
* Correcting failing rename test cases in pytest
* Try deb for nightly
* Fuse installation for mounting on debs
* Install fuse to mount and test
* Install golang to run gotests
* Install golang to run gotests
* Sudo permissions added as go tests are failing
* pre cleanup to unmount and deleted mount directory in case last run failed
* code correction
* Pytest failure to move file due to non closure
* nightly express test correction
* Add Rhel 7.5 to express test
* Adding recursive dir deletion case
* Pipeline correction
* Adding more flavours to nightly
* correcting recursive dir deletion case
* Checkout branch code added
* Adding suse to nightly
* golang install in suse
* Print curl version before mount
* Adding ubuntu 16-18-20 to express test
* No need for cleanup in case of ubuntu
* test1
* Making pipeline more configurable
* Rename parameter
* condition correction
* Adding rhel 7.5 with old curl
* Adding rhel 7.5 with old curl
* Instead of last update time watch for create or write call to decide whether to upload the file on flush or not
Co-authored-by: vibhansa-msft <>
* Starting ADLS integration using Amanda's diff
* Upscaling stresstest
* Restructuring globals
* Adding storage client interface
* Adding adls wrapper and cpplite adls compilation steps
* Build gTest with new libs
* Renaming files
* Modifying includes with updated names
* Restructure attr cache wrapper
* Moving OAuth to src directory
* Bringing to compilable state
* Adding goStress test
* Updating stressTest
* Adding more test
* Correcting script
* Correcting script
* Correcting script
* Correcting script
* Script enhancement
* Reordering tests
* Reordering tests
* Adding cleanup stages
* Correcting go errors
* Stresstest for dynamic dir name
* Stress test cleanup
* Stresstest for dynamic dir name
* Increasing stress
* Test issue correction
* Add Huge file test case
* Correcting test name
* Correcting test
* Making Cache size configurable
* Remove stress dir on completion
* Adding GO feature test
* Adding file feature test
* Save time and reduce stress test
* Cleanup stress test
* Correcting failed cases
* Adding adls feature test
* Add mount log dump for failure
* Add mount log dump for failure
* Add mount log dump for failure
* Correcting failures
* feature test failure resolution
* Parsing of metadata for ADLS account corrected
* Adding dir last modified time support
* Correcting chmod operation
* Correcting chmod test case
* Squashed 'cpplite/' changes from e8e70d7..a3a9c21
a3a9c21 Fix build error on VS2019
4eb4ce3 fix link error on Windows when building adls
git-subtree-dir: cpplite
git-subtree-split: a3a9c21de3fa9932dcfceb2b6617757a511c75c3
* Adding last access and change time support
* Correcting metadata parsing for block blob
* Addressing pipeline failure
* Pipeline update
* Correcting feature_test
* Adding atrribute cache wrapper
* Remove attr cache from nightly
* Attr cache correction
* Enable attr cache in test
* Making version number change at a common place
* Run nightly only on master
* Reverting connection.cfg to default
* Resolving mount failure saying invalid credentials
* Review comments integration
* Making it run on master as well
* Addressing review comments
* Unicode support correction
* Make empty temp dir check optional
* Mounted directorys stat corrected for different timers
* got blob directory list working
* Reverting back Makefile changes
* Making max results configurable for List and ListAll
* Remove dir timers for block blob account
* Making attr-cache-wrapper work with block blob account
* Rename flow correction
* Based on ubn version choose adls or block blob test
* Enable stress and feature test
* Correcting MSI connection
* Cred manager from cpplite needs to do a callback to get the updated token from blobfuse
* Allow tmp directory if an empty root dir is there
* For readlink dump the file details to debug failure in pipeline
* Making account type configurable from config file
* Reverting block blob modified time update, as for block blob dir last modified time is actually the creation time in azure so even with this change w are showing incorrect values only
* Set last modified date of directory in case of block blob account to the creation date, as azure storage does not maintain the last modified date for the directories
* Bumping up version to 1.3.1
* Correcting symlink realted test cases
* Correction log message for SPN
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* removing extra debug statements
* Expand '~' in tmppath to home directory
* trying to get SPN working
* trying to get SPN working
* Adding POSTFields to curl post request, if set explicitly
* Remove debug logs
* Adding SPN Validation to nightly
* trying to get SPN working
* trying to get SPN working
* trying to get SPN working
* trying to get SPN working
* incremetal
* incremetal
* incremetal
* incremetal
* incremetal
* Adding unicode file name test
* Adding SAS token test in nightly and .label test in feature test
* Test name corrected
* Making refresh token a seperate thread
* Making refresh token thread base for MSI and SPN auth
* Adding ubn20.04 to pipeline
* Build script to stop if cpplite build fails
* Converting '\' to '/' in getattr to convert it to path seperator
* Squashed 'cpplite/' changes from a3a9c21..dfbf757
dfbf757 Add performance test code
f1444b4 Fix inconsistent RFC1123 date format in different system locales
git-subtree-dir: cpplite
git-subtree-split: dfbf757f6d0b900f0c6889cee06b258fa6557e25
* Performance optimization trial
* Making default list size to be 5K as server never returns more then this
* No need to invalidate cache if its a new directory being created
* Update CA options for curl client
* Ignore system files for exact match only
* Making max result a constant for ease of change
* Enable preserving last_change time for chmod
* Cache failure also in blob property so avoid extra REST calls
* Count only for valid cached properties
* Enable attrib caceh in block blob
* Check perf with disabling Attr_caching
* GetAttr reversion as block blob failed with the change
* Removing adls end point for get property
* Reverting back to dfs endpoint for data lake account
* modestring to permission string conversion unified
* Converting upload file to dfs endpoint for upload
* Testing with higher level of concurrency
* Remove unwanted code
* Disable adls endpoint upload
* Use new cache only for ADLS
* Resetting the max_concurrency limits
* Reverting back unused changes
* Remove new cache from block blob accounts
* Remove exists check for truncate
* Dont init attr cache wrapper for adls
* Dont init attr cache wrapper for adls
* Adding exists check
* Merge correction with master
* Supressing new adls caching
* Adding new caching common for block and adls
* Correcting adls check
* Correcting infinte delete call
* Remove un-used things from blob property
* ADLS property can not be cached on listing
* Make attr cache enable on list
* On deletion of directory invalidate cache recursively
* Correct data validation test
* Resetting curl timeout to original
* Removing dead code
* Correcting timers for file after upload
* Testing list api with blob endpoint
* Adding file permissions to be retreived seperatly
* Ignore the acl to validate the performance
* Get acl for blob during get properties
* Performance tuning
* Retrying blob endpoint for list
* Add metadata support for dfs endpoint
* Performance tuning test with metadata in adls
* Get metadata seperatly for adls
* Get metadata only for file
* Add directory listing time test
* Correcting dir listing test case
* Check perf without metadata
* Optimize directory empty flow
* Remove local checking of directory empty
* Put back the empty directory check on cache
* Making symlink support configurable, by default its enable
* No need to check ctime while opening the file
* Vector replication memory issue fix
* Passing vectors by value for list apis
* Convert return type to integer and return errno
* Reverting back commented fields
* Reverting field deletion from the item list
* Readding missing fields
* Correcting cmd line option for no-symlinks
* If attr cache is enable, provide file properties from there instead of calling list again
* Optimizing list all operation for memory saving
* Rearranging code
* Try : replace list all with list in getatttr and readdir
* Correcting getattr implementation to handle empty list
* Put caching logic in List api
* Correcting cache on list with configurable option
* Reducing logs in getattr
* Clear the intermediate vector in list operation
* Enhancing attr cache for the huge list of files
* Resolving compilation failure
* Controlling attr-cache size to 12mil
* Adding cache-on-list in nightly
* return ENOENT only when list returned back empty list
* Squashed 'cpplite/' changes from dfbf757..2aa4e31
2aa4e31 Replace old RFC1123 impl which cannot be built until GCC 5
bdc6435 Fix build error on Windows
35dfba8 remove comparison in build_adls
c8f2a63 modified cmakelists
git-subtree-dir: cpplite
git-subtree-split: 2aa4e3155777ccddcd4a8562456d50059f34492c
* Symlink was broken in last commit, fixing that
* Clear metadata details when file is created
* Ignore azs_flush with null path. Github issue #470
* Parse metadata in upload file api
* Making cache-on-list a default option
* Changing the flag orders
* Try removing the cache item lock to conserve memory
* 1. azs_flush expects a null pointer in file name so correcting code to replace slashes accordingly.
2. getattr need not create and lock the file which are not even cached.
3. gc-cache need to only check the last modified time as we do not maintain other timers
* Add exponential retry policy to blob client
* fix for small github issues
* Add request id in logs
* Increasing size of attr-cached
* Adding rename directory test case
* Reducing the test case size for rename dir
* Adding rename dir to feature test script
* Correcting rename dir test case
* Correcting python test script
* PyTest correctoin
Co-authored-by: Vikas Bhansali <vibhansa@microsoft.ccom>
Co-authored-by: NaraVen <narven@microsoft.com>
Co-authored-by: Ubuntu <AzureUser@ubuntu1604identity.ef4nn4lo4nbedgoq0aeagsr1pd.dx.internal.cloudapp.net>
* Adjusting timeout in nightly pipeline
* Scaling down stresstest as nightly is taking couple of hours to complete
* Reduce stress
* Correcting pyTest results for time comparison
* Put upper limit on time on stresstest
* Starting ADLS integration using Amanda's diff
* Upscaling stresstest
* Restructuring globals
* Adding storage client interface
* Adding adls wrapper and cpplite adls compilation steps
* Build gTest with new libs
* Renaming files
* Modifying includes with updated names
* Restructure attr cache wrapper
* Moving OAuth to src directory
* Bringing to compilable state
* Adding goStress test
* Updating stressTest
* Adding more test
* Correcting script
* Correcting script
* Correcting script
* Correcting script
* Script enhancement
* Reordering tests
* Reordering tests
* Adding cleanup stages
* Correcting go errors
* Stresstest for dynamic dir name
* Stress test cleanup
* Stresstest for dynamic dir name
* Increasing stress
* Test issue correction
* Add Huge file test case
* Correcting test name
* Correcting test
* Making Cache size configurable
* Remove stress dir on completion
* Adding GO feature test
* Adding file feature test
* Save time and reduce stress test
* Cleanup stress test
* Correcting failed cases
* Adding adls feature test
* Add mount log dump for failure
* Add mount log dump for failure
* Add mount log dump for failure
* Correcting failures
* feature test failure resolution
* Parsing of metadata for ADLS account corrected
* Adding dir last modified time support
* Correcting chmod operation
* Correcting chmod test case
* Squashed 'cpplite/' changes from e8e70d7..a3a9c21
a3a9c21 Fix build error on VS2019
4eb4ce3 fix link error on Windows when building adls
git-subtree-dir: cpplite
git-subtree-split: a3a9c21de3fa9932dcfceb2b6617757a511c75c3
* Adding last access and change time support
* Correcting metadata parsing for block blob
* Addressing pipeline failure
* Pipeline update
* Correcting feature_test
* Adding atrribute cache wrapper
* Remove attr cache from nightly
* Attr cache correction
* Enable attr cache in test
* Making version number change at a common place
* Run nightly only on master
* Reverting connection.cfg to default
* Resolving mount failure saying invalid credentials
* Review comments integration
* Making it run on master as well
* Addressing review comments
* Unicode support correction
* Make empty temp dir check optional
* Mounted directorys stat corrected for different timers
* got blob directory list working
* Reverting back Makefile changes
* Making max results configurable for List and ListAll
* Remove dir timers for block blob account
* Making attr-cache-wrapper work with block blob account
* Rename flow correction
* Based on ubn version choose adls or block blob test
* Enable stress and feature test
* Correcting MSI connection
* Cred manager from cpplite needs to do a callback to get the updated token from blobfuse
* Allow tmp directory if an empty root dir is there
* For readlink dump the file details to debug failure in pipeline
* Making account type configurable from config file
* Reverting block blob modified time update, as for block blob dir last modified time is actually the creation time in azure so even with this change w are showing incorrect values only
* Set last modified date of directory in case of block blob account to the creation date, as azure storage does not maintain the last modified date for the directories
* Bumping up version to 1.3.1
* Correcting symlink realted test cases
* Correction log message for SPN
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* trying to get SPN wokring
* removing extra debug statements
* Expand '~' in tmppath to home directory
* trying to get SPN working
* trying to get SPN working
* Adding POSTFields to curl post request, if set explicitly
* Remove debug logs
* Adding SPN Validation to nightly
* trying to get SPN working
* trying to get SPN working
* trying to get SPN working
* trying to get SPN working
* incremetal
* incremetal
* incremetal
* incremetal
* incremetal
* Adding unicode file name test
* Adding SAS token test in nightly and .label test in feature test
* Test name corrected
* Making refresh token a seperate thread
* Making refresh token thread base for MSI and SPN auth
* Adding ubn20.04 to pipeline
* Build script to stop if cpplite build fails
* Converting '\' to '/' in getattr to convert it to path seperator
* Squashed 'cpplite/' changes from a3a9c21..dfbf757
dfbf757 Add performance test code
f1444b4 Fix inconsistent RFC1123 date format in different system locales
git-subtree-dir: cpplite
git-subtree-split: dfbf757f6d0b900f0c6889cee06b258fa6557e25
* Performance optimization trial
* Making default list size to be 5K as server never returns more then this
* No need to invalidate cache if its a new directory being created
* Update CA options for curl client
* Ignore system files for exact match only
* Making max result a constant for ease of change
* Enable preserving last_change time for chmod
* Cache failure also in blob property so avoid extra REST calls
* Count only for valid cached properties
* Enable attrib caceh in block blob
* Check perf with disabling Attr_caching
* GetAttr reversion as block blob failed with the change
* Removing adls end point for get property
* Reverting back to dfs endpoint for data lake account
* modestring to permission string conversion unified
* Converting upload file to dfs endpoint for upload
* Testing with higher level of concurrency
* Remove unwanted code
* Disable adls endpoint upload
* Use new cache only for ADLS
* Resetting the max_concurrency limits
* Reverting back unused changes
* Remove new cache from block blob accounts
* Remove exists check for truncate
* Dont init attr cache wrapper for adls
* Dont init attr cache wrapper for adls
* Adding exists check
* Merge correction with master
* Supressing new adls caching
* Adding new caching common for block and adls
* Correcting adls check
* Correcting infinte delete call
* Remove un-used things from blob property
* ADLS property can not be cached on listing
* Make attr cache enable on list
* On deletion of directory invalidate cache recursively
* Correct data validation test
* Resetting curl timeout to original
* Removing dead code
* Correcting timers for file after upload
* Testing list api with blob endpoint
* Adding file permissions to be retreived seperatly
* Ignore the acl to validate the performance
* Get acl for blob during get properties
* Performance tuning
* Retrying blob endpoint for list
* Add metadata support for dfs endpoint
* Performance tuning test with metadata in adls
* Get metadata seperatly for adls
* Get metadata only for file
* Add directory listing time test
* Correcting dir listing test case
* Check perf without metadata
* Optimize directory empty flow
* Remove local checking of directory empty
* Put back the empty directory check on cache
* Making symlink support configurable, by default its enable
* No need to check ctime while opening the file
* Vector replication memory issue fix
* Passing vectors by value for list apis
* Convert return type to integer and return errno
* Reverting back commented fields
* Reverting field deletion from the item list
* Readding missing fields
* Correcting cmd line option for no-symlinks
* If attr cache is enable, provide file properties from there instead of calling list again
* Optimizing list all operation for memory saving
* Rearranging code
* Try : replace list all with list in getatttr and readdir
* Correcting getattr implementation to handle empty list
* Put caching logic in List api
* Correcting cache on list with configurable option
* Reducing logs in getattr
* Clear the intermediate vector in list operation
* Enhancing attr cache for the huge list of files
* Resolving compilation failure
* Controlling attr-cache size to 12mil
* Adding cache-on-list in nightly
* return ENOENT only when list returned back empty list
* Squashed 'cpplite/' changes from dfbf757..2aa4e31
2aa4e31 Replace old RFC1123 impl which cannot be built until GCC 5
bdc6435 Fix build error on Windows
35dfba8 remove comparison in build_adls
c8f2a63 modified cmakelists
git-subtree-dir: cpplite
git-subtree-split: 2aa4e3155777ccddcd4a8562456d50059f34492c
* Symlink was broken in last commit, fixing that
* Clear metadata details when file is created
* Ignore azs_flush with null path. Github issue #470
* Parse metadata in upload file api
* Making cache-on-list a default option
* Changing the flag orders
* Try removing the cache item lock to conserve memory
* 1. azs_flush expects a null pointer in file name so correcting code to replace slashes accordingly.
2. getattr need not create and lock the file which are not even cached.
3. gc-cache need to only check the last modified time as we do not maintain other timers
* Add exponential retry policy to blob client
* fix for small github issues
* Add request id in logs
* Increasing size of attr-cached
* Adding rename directory test case
* Reducing the test case size for rename dir
* Adding rename dir to feature test script
Co-authored-by: Vikas Bhansali <vibhansa@microsoft.ccom>
Co-authored-by: NaraVen <narven@microsoft.com>
Co-authored-by: Ubuntu <AzureUser@ubuntu1604identity.ef4nn4lo4nbedgoq0aeagsr1pd.dx.internal.cloudapp.net>
- Improvement in attribute caching
- Fixed broken symlink support
- Added support for '&' in file name
- Replace '\' with '/' in any file or directory names
- CPPLite upgraded to latest
- Adding support for ADLS account
- Symlink support added
- chmod command is supported for ADLS accounts
- Unicode filename bug fixed
- Block blob directory last modified time showing 1970 fixed
- last modified / access / change time support for ADLS account
- last modified / access / change time for mount directory corrected
Co-authored-by: Vikas Bhansali <vibhansa@microsoft.ccom>
Co-authored-by: NaraVen <narven@microsoft.com>
* fixed filenotfounderror
* mdoified readme
* fixed the bug where an invalid authtype assumes MSI and repeatedly tries to get an authtoken
* modified Yaml to include tests
* modified Yaml to include tests
* modified Yaml to include tests
* modified Yaml to include tests
* modified Yaml to include tests
* modified Yaml to include tests
* yaml fix
* yaml fix
* yaml fix
* yaml fix
* yaml fix
* yaml fix
* yaml fix
* yaml fix
* test dir in yaml fix
* fixing build
* fixing build
* yaml -> adding permissions to user ci user to execute tests binary
* yaml -> adding permissions to user ci user to execute tests binary
* yaml -> troubleshooting test run failure
* yaml -> fixing test running during build
* tet case time to match utc if the machine is not set to UTC
Vikas Bhansali
Sourav Gupta
Gauri Prasad
Tamer Sherif
Gauri Prasad
nara
vibhansa-msft
Nara
adreed-msft
Amanda Nguyen
Adam Sorrin