Граф коммитов

93 Коммитов

Автор SHA1 Сообщение Дата
syeleti-msft ba739b6206
Fix ReadPanic (#1533)
* Fix Read Panic
2024-11-04 18:50:08 +05:30
Vikas Bhansali 8c6a53e527
Reverting custom component patch (#1541) 2024-10-15 14:25:54 +05:30
abhiguptacse 1f78a4b59e adding copyright 2024-10-15 08:10:23 +00:00
abhiguptacse 497a39985a rename + moving package 2024-10-08 15:14:39 +00:00
abhiguptacse 630e31cdc5 resolved comments 2024-09-25 11:39:13 +00:00
abhiguptacse c38ddd967f first cut implementation of custom components. 2024-09-19 06:37:49 +00:00
Sourav Gupta 0aa064f066
Data integrity issues in block cache (#1508)
* Data Integrity fixes
2024-08-22 18:18:42 +05:30
Sourav Gupta 891d0032e4
Race condition in random write between uploading and writing to a same block (#1481)
* Block upload and write race condition
2024-08-03 12:02:36 +05:30
Vikas Bhansali 2c1781ec92
Depdendency upgrade for next release (#1486)
* Depdendency upgrade for next release
2024-08-02 14:20:52 +05:30
ashruti-msft da78133520
Block Cache Read correction (#1483)
Fixed: We copied the entire block regardless of whether it was fully used, leading to copying over garbage data.
Fixed: Error in read when disk cache was enabled
2024-07-31 11:00:28 +05:30
Sourav Gupta ee19eff072
Block cache random write in sparse files (#1475) 2024-07-30 16:27:20 +05:30
Vikas Bhansali cefb0226f9
Removing usage of SPN in pipelines (#1431)
* Removing usage of SPN in pipelines
2024-06-17 16:19:22 +05:30
Sourav Gupta 8e06e67633
GO SDK track2 migration (#1380) 2024-04-04 15:59:29 +05:30
Vikas Bhansali bd7dffcfc4
Adding e2e tests for block-cache in pipeline (#1325)
* Adding e2e tests for block-cache in pipeline
2024-02-27 14:32:48 +05:30
Vikas Bhansali fdfe001287
Write support using Block-Cache (#1258)
* Adding support for writing using block-cache
2024-01-22 17:38:30 +05:30
Sourav Gupta d918c11e5a
Mount on non-empty path (#1311) 2024-01-16 11:12:53 +05:30
Vikas Bhansali 439f7dbccd
Copyright fix" (#1302)
Copyright fix for 2024
2024-01-02 15:40:39 +05:30
ashruti-msft a8edfb919a
Truncate function changed for streaming (#1248) 2023-11-15 11:07:12 +05:30
Vikas Bhansali 9c10ae0880
Correcting sync to flush behavior (#1281)
* Correcting sync to flush behaviour
2023-11-07 17:18:11 +05:30
Sourav Gupta 610ef3af8a
Symlink fix for ADLS accounts (#1283) 2023-11-07 15:56:28 +05:30
Vikas Bhansali 0f20ace345
Refresh token 5 minutes before the expiry (#1253)
* Refresh token 5 minutes before the expiry
2023-10-03 21:57:59 +05:30
Vikas Bhansali 5ecfe4004c
Adaptive Prefetch Streaming (#1181)
* Adaptive Prefetch Streaming
2023-08-25 12:20:34 +05:30
Vikas Bhansali 49154dd62f
lseek() on dir handle does not work as expected (#1224)
* Document lseek on dir will not work as expected
2023-08-14 14:41:26 +05:30
Vikas Bhansali 3f756e4131
Telemetry header addition (#1214)
* Adding telemetry string
2023-08-01 16:28:04 +05:30
Sourav Gupta 461fad73e9
Adding FUSE option direct_io (#1162)
* Adding direct_io flag
2023-06-06 21:23:25 +05:30
Vikas Bhansali 05a10949bd
Upgrading to go 1.19 (#1135)
* Upgrading to go 1.19
2023-05-10 14:53:05 +05:30
Vikas Bhansali 4763ee5f33
DFS rename operation not encoding source url (#1082)
* Adding fix for dfs rename where source is not url encoded
2023-03-15 15:14:54 +05:30
Sourav Gupta 855b27c9f1
Set ignore-open-flags to true by default (#1029)
* Making ignore-open-flags as true
2023-01-21 10:21:38 +05:30
Sourav Gupta 399ad95fb1
Mount all fix (#1034)
* Mount all fix
2023-01-20 12:32:56 +05:30
Vikas Bhansali e02acffd38
Copyright fix (#1025)
* Correcting copyright statement
2023-01-07 08:11:07 +05:30
Vikas Bhansali 5f9260442f
Correct version and update Changelog post preview.4 release (#960)
* Correct version and update Changelog post preview.4 release

* Update as per review comments

* Remove unwanted files
2022-11-04 11:23:14 +05:30
Gauri Prasad 1c18b19bef
Fix line to disable for stream direct (#957) 2022-11-02 12:11:19 -07:00
Gauri Prasad 135843b122
Added stream data validation test cases (#955) 2022-10-31 10:49:27 -07:00
Vikas Bhansali 34add4579e
FIO Test additions (#943)
* Add random fio read/write tests to perf pipeline

* Adding more fio tests to per pipeline

* Rename first stage

* Correct output file name

* Correcting output file name

* display command being tested out

* Correct fio command

* Temp disable other tests

* Make resnet50 test a configurable option

* Correcting fio command

* Correcting fio command

* Correcting fio command

* Correcting fio command

* Re-enable disabled perf tests

* Re-enable disabled perf tests

* Print error in azauth test failures
2022-10-18 14:36:29 +05:30
Vikas Bhansali 92ea765040
Vibhansa/v2/newvmaddition (#929)
* Trying out rehl 8.6 and 9.0

* Adding RHEL 8.6 to pipeline

* Correcting perf pipeline outputs

* Correct service file to contain env variables regarding auth

* Add fuse in suse so that unmount can work without fusermount3
2022-10-11 21:53:54 +05:30
Vikas Bhansali 6f8d90ad71
Docker installation and container image build (#914)
* adding commands to build and launch a container image with blobfuse2 installed

* Adding docker installation steps

* spell correction

* Adding support for fuse2 docker

* Correct fuse installation step

* Correct fuse installation step

* Adding steps to publish container image to azure container registry

* Adding script to map dependency graph

* Changes as per review comments

* Adding docker container related help in README

* Add flag to ignore open flag for write to work

* Correcting ignore open flags related option in dockerfile

* Correct handling of ignore open flags

* Correcting flag on cli

* Run blobfuse in foreground so that container stays up untill unmount is done

* Spell correction

Co-authored-by: Ubuntu <vibhansa@vibhansa-ubn20-perf.x53acyytduluvkundcmr1j05nd.sx.internal.cloudapp.net>
2022-10-01 09:59:38 +05:30
Vikas Bhansali 4efad88930
Reset per file coverage threshold to 70% (#922) 2022-09-30 08:13:28 +05:30
Gauri Prasad 816ec4c77f
Run scripts in performance pipeline (#888) 2022-09-26 07:52:07 -07:00
Tamer Sherif ec6eb72a29
Script to test attr calls after writes + perf vm fix (#911)
* pipeline fix

* pipeline fix
2022-09-23 10:32:58 -07:00
Gauri Prasad f067d5ca23
Clean up exit 1 errors for blobfuse2 (#893) 2022-09-08 13:09:32 -07:00
Gauri Prasad 23e7af7528
Make writeback cache default (#887)
* stream data validation

* stream data validation

* Enable writeback caching safely

* Flipped boolean semantics for writeback cache

* Added CLI param

* Add debug log for write only in case it is a problem

* Modified test to work with enable writeback cache

* Added a flag to ignore append flag

* github comments

* Added tests for ignore append

* changelog

* Added fuse2 test

* correcting file mode when ignore-append-file flag is true

* fix unit test

* Changes for ignore open flags

* test for libfuse2

* add ignore open flag to all config

* Updated README

* Added to Changelog

* Quick test only in data validation for streaming

Co-authored-by: Tamer Sherif <tasherif@microsoft.com>
Co-authored-by: vibhansa-msft <vibhansa@microsoft.com>
Co-authored-by: souravgupta <souravgupta@microsoft.com>
2022-09-02 17:04:42 +05:30
Sourav Gupta 6ba1e91a7e
Expand path (#880)
Expanding tilde(~) character in mount, config file or cache paths
2022-09-01 19:30:08 +05:30
Gauri Prasad 0a28bfba1d
Added scripts to generate output conveniently (#872) 2022-08-26 10:45:48 -07:00
Vikas Bhansali cb4d5445ed
MD5 Sum updation on upload and validation on download (#863)
* Testing migration to 1es hosted pool

* remove indent

* Agent Image

* Remove demands

* Set imageName

* Add correct demand

* Just echo hello world

* Correct indent

* remove the quotes =

* lets try this

* maybe this works

* yeet

* added quotes around the pool name, added agent name demand

* added vmImage name?

* illegal node name?

* hostedAgent = true

* Remove Agent Name

* checkout none

* simple yaml

* try other pool

* no quotes

* UPdate

* update tab

* checkout ne

* Try blobfuse pool

* Added steps to pipeline nightly test

* Fixed indent

* Remove checkout: none

* Add distro test

* fixed indent

* remove checkout

* Fix checkout

* fuse3

* Added RHEL 8

* Override image

* Install

* Added Centos

* Added Oracle

* Added debian

* Added Mariner and SUSE

* Added C build

* Added git to Deban

* Try to get centos 8 to work

* try another cond

* conditional print

* conditional print

* agent condition try 2

* try with variable

* Update OS mirrors

* Fix sed command

* Add to yum

* try removing -e

* no bedst

* Conditional script

* Add more if script

* fix centos 7 stuff

* add flags

* fix check

* tdnf with y option

* Added a stdio lib

* try to fix oracle 8.1 and centos 8.5

* RHEL add flags

* remove ski pborken

* try debian 10

* added build essentials pacakge

* try another rhel

* also install fusermount

* added whoami

* added groups

* trying to run as root

* trying to not run as root

* install fuse

* mariner build essential

* try to use yum to install git

* Add -y

* Get sources list

* remove suse

* spell sources

* find the file that looks for microsoft repos

* remove azure list

* add to debiam

* INstall fuse lib

* try to fix debian 11

* Added MSI test

* reenable suse

* Install cmake and gcc on rhel

* Adding cmake install step for MSI

* Failover methods are not supported in EL8 so remove that confi from default yum repo config file

* install cmake on suse

* Correcting config file name in MSI ste

* Manually install fuse3-devel on SUSE

* Ignore gpg checks in suse to allow installing fuse3-devel

* Correcting suse install step

* Install fuse-libs on rhel vms

* user fusermount3 to unmount as we are not installing fuse on all of the vms

* Trying rhel 7.3 image

* Add RHEL 7.5

* Try to install rh-python

* Trying to install libfuse in RHEL

* sudo for echo

* Use sudo correctly

* add -e

* Updated container and python package for rhrl

* Try to get RHEL 8 to work

* Try to use vault 8 repos

* try mirror

* try without a mrror

* try with enable epel

* add flags

* Get all RHEL to work

* Get rhel 7.5 to work

* Added Ubuntu nightly

* added params

* Use msft managed

* indent

* Use normal nightly pipeline

* Added Blob STE retry policy

* Add credential policy

* Added Datalake retry policy

* Correcting UT compilation failure

* Ignore data validation test on distros other than ubn

* Change retry policy factory name for datalake

* Reducing cache config for streaming test

* Remove data validation test from different auth options

* Remove quick_test as default is anyway true

* Move data validation as a seperate stage

* Move data validation as a seperate stage

* Move data validation as a seperate stage

* Add data validation as a configurable step

* Make exhaustive test disable most of the test cases

* Adding support for md5sum validation and updation

* Add tags for data validation builds

* Try running fuse2 on ubn20 for data-validation

* Try running fuse2 on ubn20 for data-validation

* Merge all git related test cases to save time

* Remove ubn18 from pipelines, and use ubn20 for fuse2 testing

* Use ubn18 for release pipeline as glibc version error is coming on other distros due to this

* Remove clone and e2e test from release pipelines

* Adding basic e2e checks as part of ci

* Undo e2e from ci as multiple ci can run in parallel and running e2e may make it flaky

* Updating changelog for retry policy

* Enable git and tar test cases in nightly data validation test

* Enable git tests in verbose tests

* Remove clone test from verbose test

* Correct code lint command

* Ignore fuse3 test wrapper as well in linting

* Have two different flags for setting and validation md5 sums

* Correct code linting issue

* Adding UT for various combinations of MD5

* File cache shall delete the local file if download fails

* Adding few test cases to azstorage utils

* Remove dead code

* Resolve UT issue with logging

* Use 1ES for ubn-18 in artifact pipeline

* Remove redundancy

* Add dependency

* Add dependency

* Move code coverage to 1ES but remove ubn-22

* Move code coverage to 1ES but remove ubn-22

* Add account cleanup step to code coverage

* Add cleanup stage in code coverage

* Add cleanup stage in code coverage

* Correcting work dir path

* correction as per review comments

* Always run account cleanup at the end

* install go before doing account cleanup

* Make coverage and account cleanup a configurable step

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-08-25 16:49:56 +05:30
Tamer Sherif bbd19b2f3f
Streaming small fixes + more tests (#861)
* added tests and cleaned up for prev

* added tests and cleaned up for prev

* more fixes

* fix

* added e2e

* added e2e

* added e2e

* added e2e

* added empty temp path

* fixed temp dir

* removed read only from config

* git clone disable

* added tests

* more tests

* read in buffer err:

* added stream only error test

* more tests

* close file and flush

* removed comment

* cleanup

* fix

* cleanup
2022-08-24 09:22:51 -07:00
Gauri Prasad ec4817d1ac
Added Azurite Test (#865) 2022-08-18 12:33:19 -07:00
Vikas Bhansali 50d3f9ebe9
Retry on timeout error from server (#858) 2022-08-18 09:00:13 -07:00
Vikas Bhansali dfe3da52e5
Stablizing pipelines and adding more test cases (#856)
* Disable Data validation tests in artifact pipeline

* Correcting test filer

* Removing tests for debian-9 for now

* Correcting paths for deb 9 distro

* Correct config file printing with path

* Adding git status and stash related test cases

* Add directory tar test case

* correct string check as different version of git are giving different output strings

* Check git cli output only if there was something dumped on console

* Always unmount before mount in release tests

* Use different directory for each git related test

* Adding timeout to release steps

* Do minimal test with LFU

* Taking a smaller repo for all git related test cases
2022-08-05 16:54:50 +05:30
Vikas Bhansali 154ba6c7ab
Code coverage addition for generate config cli (#849)
* Remove dead code and fix file-cache usage percentage check UT

* Add coverage test for gen-test-config

* Adding code coverage for doc command

* Adding build script to build with different options

* Delete old binary before regenerating

* Delete old binaries before regenerating

* Adding pipeline test cases

* Remove agent cleanup steps as 1ES is spawning new vms for every run

* Disable sas validation on ubn-22 for now

* Disable sas validation on ubn-22 for now

* Disable sas validation on ubn-22 for now

* Removing ubn-22 from all pipelines

* Correcting adls sas param

* Re-add sas validation for all ubn distros

* Disable chmod test cases for SAS auth

* Set default value for sas to be false

* Converting account sas to container sas for chmod test to work

* Enable e2e test again

* Correcting e2e failure with chmod test case

* Add Ubn-22 to release pipeline

* Change pool of ubnuntu to ms hosted

* Remove dead param from yaml file
2022-08-02 12:52:06 +05:30
Vikas Bhansali 4f805cc1ab
Adding code linting (#844)
* Adding Ubn-22 to pipelines

* Adding lint as standard ci step

* correcting lint command path

* Add copyright checks to ci pipeline

* Copyright fixes and ci check for copyright statements

* Adding go format check to CI

* Fixing some lint issues

* More lint fixes

* More lint fixes

* Fixing some lint issues

* Correcting code linting check in ci

* correcting lint status check

* correcting lint status check

* correcting lint status check

* correcting lint status check

* correcting lint status check

* correcting lint status check

* correcting lint status check

* correcting bugs reported by sa check

* notices_fix.sh update

* Add step to validate NOTICE file

* Add step to release for ubn-22

* Add logs for success

* Fixing copyright bug

* Correcting tag formats

* Correcting percentage check

* Spell error correction

* Ignore exectime from code coverage

* Correcting log string

* Removing dead file

* Correcting go format error
2022-07-28 10:09:41 +05:30