Граф коммитов

588 Коммитов

Автор SHA1 Сообщение Дата
Gauri Prasad 3856dcb1f0
Fixed % in log issue (#778) 2022-05-05 08:24:26 -07:00
Tamer Sherif 1f339ff0dd
Deprecate using 3rd party eviction policies (#769)
* Optimized Streaming

* close handle fix

* locking

* fixed tests

* cleanup

* Deprecate eviction library

* more cleanup

* remove timeout

* fix

* cleanup

* fixed bug

* fixed bug

* added purge

* added purge

* added purge

* added purge

* comment cleanup

* comment cleanup

* removed lock and added tests

* cleanup

* added handle buffer check

* cleanup - feedback

* moved to new dir
2022-04-28 10:43:01 -07:00
Vikas Bhansali ad3749b556
Correction with Block Size calculation for upload (#768)
* Fix issue with block-size running into overflow of number of blocks

* Add blocksize and file size to logs

* Make blocksize function to work on file size instead of fi object and add UT for blocksize calculations

* Adding more ut case

* Codespell error fix

* Rounding off and making it 8 multiple for edge cases

* Adding comments for rounding off

* Ceil the value directly while calculating the block size
2022-04-20 13:43:04 +05:30
Gauri Prasad dce44c67aa
Temporarily fixed fio error caused due to race condition between OpenFile and CloseFile (#765) 2022-04-18 07:28:07 -07:00
Sourav Gupta bf6ce0b96e
Test cases for mount command (#757)
* Adding test cases for mount command

* Adding mount test to nightly pipeline

* Adding test case where the config file is not provided

* Typo fix

* Adding test case for mounting using environment variables
2022-04-16 10:43:57 +05:30
Gauri Prasad 4b64cb712a
Set flag parsing to exit on error to ensure exit status is non zero (#762) 2022-04-15 16:35:57 -07:00
Tamer Sherif e624502e06
[BUG] Fixed block size max blocks exceeded bug when blocksize not set (#759)
* Fixed block size max blocks exceeded

* println

* println

* add const

* changed default config to 0

* cleanup

* cleanup

* fix test

* test
2022-04-13 17:31:54 -07:00
Gauri Prasad 6fc0a3cae7
Support . and .. in ls (#756) 2022-04-11 09:57:57 -07:00
Gauri Prasad c7e419bb7d
Fixed issue where DIRECT flag was not masked out in libfuse3 component (#755)
* Fixed issue where DIRECT flag was not masked out in libfuse3 component

* Added a unit test

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-04-06 08:54:58 -07:00
Sourav Gupta 7a676a38b8
removing trailing '/' from path (#754) 2022-04-01 10:40:49 +05:30
Gauri Prasad 5d14ff5310
Codespell fix (#750) 2022-03-30 08:09:46 -07:00
Gauri Prasad d005f6e0ae
Fixed bug where blobfuse2 could not be installed in docker container (#751) 2022-03-30 08:09:29 -07:00
Sourav Gupta 5cda7edeab
E2E data validation tests (#740)
* Adding e2e test for data validation

* Correcting tmp-path flag name

* Not run as unit test

* Adding negative test case
2022-03-28 21:26:12 +05:30
Vikas Bhansali 50c2267008
Fixing pipeline issue where providing config file is leading to a mount failure (#749) 2022-03-26 12:28:59 +05:30
Vikas Bhansali b1ff604a39
Vibhansa/v2/noconfigfile (#747)
* Allow blobfuse2 to mount without config file, in case all required params are set as env variables

* Updating readme for perf comparison with v1

* Removing test code

* Changes as per review comments

* Corrected perf numbers in readme
2022-03-25 10:54:41 +05:30
Vikas Bhansali 103149ba7c
File-cache open-file complexity with respect to file locks removed (#737)
* Improving file locks for better perf and less complexity

* Addressing review comments

* Flaky timeout test correction

* Replace os.remove with deleteFile call as we need to handle permission error in all such cases

* Adding trace logs to deletefile api

* Resolving file cache UT for rename when file is not in local cache
2022-03-25 08:30:33 +05:30
Vikas Bhansali 7f4417ef1b
V2 Longhaul scripts (#732)
* Adding longhaul test script

* Update test/longhaul/longhaul.sh

* Correcting scripts for longhaul

* Correcting mail command

* Correcting typo as per review comments

Co-authored-by: Ubuntu <vibhansa@blobfuse.com>
2022-03-24 11:48:30 +05:30
Gauri Prasad ece81e7c8d
Preserve metadata when reuploading files (#739)
* Added metadata to handle to attempt to preserve metadata. WIP

* Added metadata preservation feature to attribute cache

* Removed test from file test

* Fixed unit tests

* Added code to ensure metadata is retreived

* metadata preservation

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-03-23 21:15:35 -07:00
Gauri Prasad 15b68d598a
Fixed some codespell errors (#745)
Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-03-23 11:00:13 -07:00
Tamer Sherif 168ea5d03a
[PERF] ResNet50 Weekly Perf Runs (#731)
* added new pipeline

* python cleanup

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* test run

* removed print

* testing binary

* testing binary

* testing binary

* testing binary

* testing binary

* testing binary

* testing

* testing

* testing

* testing

* testing

* finished

* finished

* finished

* attempting to fix py issue

* cleanup

* testing

* removed direct read

* removed direct read

* removed direct read

* removed direct read

* fail if regression is too bad

* cleanup

* added dependency step to pipeline

* cleanup
2022-03-17 16:03:09 -07:00
Vikas Bhansali c5d1c7f3ff
Adding deb-11 and Mariner to nightly and release pipeline (#729)
* Adding deb-11 to the pipelines

* Adding mariner in pipeline

* Correcting steps for mari

* Correcting steps for mari

* Correcting steps for mari

* Correcting steps for mari

* Correcting steps for mari

* Correcting steps for mari

* Correcting stage name for mari

* Correcting stage name for mari

* Correcting install step in mari

* Adding ubn-22 to both pipelines

* Comment out ubn-22 as devops does not support it yet
2022-03-16 11:42:10 +05:30
Sourav Gupta b466677539
Gen1 mount using adlsgen1fuse (#721)
* Adding code for gen1 mount with datasetfuse

* Creating pipeline for adlsgen1fuse

* Adding unit test cases

* Adding sample datasetfuse config file
2022-03-16 10:45:23 +05:30
Gauri Prasad 7cd6fbe554
Fixed bug where endpoint is populated incorrectly when passed as envi… (#730)
* Fixed bug where endpoint is populated incorrectly when passed as environment variable

* Fixed unit tests

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-03-15 08:40:39 -07:00
Sourav Gupta 3afd9a2ae3
Adding version and security warnings check (#726)
* Changes for checking version and showing warnings

* Using sas url for version container

* Checking only the status code

* Pointing to blobfuse2

* Having python script instead of keeping it inline
2022-03-15 09:59:21 +05:30
Tamer Sherif 1d05fcc431
Merge pull request #711 from Azure/write-stream-block
[FEATURE] Support Write Stream on AzStorage Component
2022-03-14 13:52:05 -07:00
Tamer Sherif 36292f0342 fixed test 2022-03-14 13:24:58 -07:00
Tamer Sherif 2fb1dd44cc more tests 2022-03-14 12:57:25 -07:00
Tamer Sherif fd5f725173 added test with offset larger than file 2022-03-14 12:48:27 -07:00
Tamer Sherif 7eebccbf57 fixed tests 2022-03-13 19:15:44 -07:00
Tamer Sherif c012eeafac cleanup 2022-03-13 00:15:47 -08:00
Vikas Bhansali d9ffe5cba9
chmod fix to preserve additional principals added to to ACL (#727)
* chmod shall result into SetAccessControl call with permissions and not ACL to preserve the existing principals added to the file

* Adding loopbackfs config to base config
2022-03-11 09:11:05 +05:30
Tamer Sherif 0d6c0f42dc component 2022-03-10 12:52:49 -08:00
Tamer Sherif 7d9ceb9185 removed mod block list 2022-03-10 12:21:09 -08:00
Tamer Sherif c9ae4d1f79 removed small file filed 2022-03-10 10:39:27 -08:00
Vikas Bhansali ba1a18c34d
Double mount validation corrected and error string generalised (#722)
Double mount validation corrected and error string generalised
2022-03-10 09:18:05 +05:30
Ashish Mishra 17310a3927
Renaming the header of migration.md (#725) 2022-03-08 20:23:54 -08:00
Tamer Sherif 1001a9ade5 struct cleaning 2022-03-08 15:07:39 -08:00
Tamer Sherif 60a2ab5b28 create new blocks cleanup 2022-03-08 15:00:15 -08:00
Tamer Sherif 2f4320bf98 further cleanup 2022-03-08 12:30:59 -08:00
Tamer Sherif 84a095613c reduce copy call 2022-03-07 15:08:51 -08:00
Tamer Sherif 40f997c0c7 added tests 2022-03-07 14:37:05 -08:00
Gauri Prasad dd401750a3
Fixed bug where some user provided values that can be set to 0 were s… (#714)
* Fixed bug where some user provided values that can be set to 0 were set to default value

* Addressed comments

* Fixed unit test for file cache empty config

* Allow for cli params to be set to 0

* Fixed defaults and merged isSet

* Added test

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-03-07 10:23:52 -08:00
Vikas Bhansali 827506595a
For re-download of readonly file reset the permissions before we download (#720)
Read-only file redownload was failing, need permission change on local file before we redownload
2022-03-04 11:09:05 +05:30
Vikas Bhansali b0e7125836
Extension support with CGO (#705)
* Adding v2 galactus from devops

* Copyright update

* Correction baseconfig file

* Renaming test libs to extension

* Remvoing binary

* Sync with main and updates as per review comments
2022-03-04 10:30:15 +05:30
Vikas Bhansali 870fb10955
Block level disk cache for read streaming (#706)
* Correcting UT for disk cache

* Sync with main and review comment addressed

* Keeping default persisted block timeout to 30 mins

* changes as per review comments

* adding comments to UT for disk-persistence

* Copyright info update
2022-03-03 09:57:11 +05:30
Tamer Sherif 2b170a69d3 added dl tests 2022-03-02 16:46:16 -08:00
Tamer Sherif 68b8d05026 Merge branch 'main' into write-stream-block 2022-03-02 15:19:23 -08:00
Tamer Sherif ac6d625790 added dl tests 2022-03-02 15:19:01 -08:00
Tamer Sherif 9885fb7421
Merge pull request #717 from Azure/change-mod-after-create
[BUG] Upload after creating does not work if > 256MB
2022-03-02 14:17:40 -08:00
Tamer Sherif 3f9326cc85 fixed datalake tests 2022-03-01 15:55:52 -08:00