Граф коммитов

674 Коммитов

Автор SHA1 Сообщение Дата
Gauri Prasad 50ee5aa064 made separate step for secure config 2022-10-24 00:39:25 -07:00
Gauri Prasad 6a921cd9ff Add -y to apt commands 2022-10-19 01:30:56 -07:00
Gauri Prasad bbcc2db261 Add proxy to code cov 2022-10-18 23:57:57 -07:00
Gauri Prasad 196a9085a0
Fixed a bug where mount all required a config file (#939) 2022-10-18 20:07:14 -07:00
Vikas Bhansali 34add4579e
FIO Test additions (#943)
* Add random fio read/write tests to perf pipeline

* Adding more fio tests to per pipeline

* Rename first stage

* Correct output file name

* Correcting output file name

* display command being tested out

* Correct fio command

* Temp disable other tests

* Make resnet50 test a configurable option

* Correcting fio command

* Correcting fio command

* Correcting fio command

* Correcting fio command

* Re-enable disabled perf tests

* Re-enable disabled perf tests

* Print error in azauth test failures
2022-10-18 14:36:29 +05:30
Tamer Sherif 95754b705d
MSI Object ID Credential Support (#926)
* msi support for obj id

* cleanup

* added validation to msi

* error

* debug

* debug

* test

* testing

* more testing

* more testing

* more testing

* cleanup

* object id

* UT

* fixed test

* fixed test

* UT

* fix

* error change

* cleanup

* test

* autodetect

* fix tests
2022-10-13 10:04:08 -07:00
Tamer Sherif 6fa067d419
Document required config for azstorage + fix samples (#936)
* added required comment and fixed sampels

* more detail
2022-10-12 09:11:22 +05:30
Tamer Sherif ee2b5a182a
Stream Filename Caching E2E and Data Validation (#934)
* added e2e filename caching

* data validation

* data validation

* time fix

* test

* test
2022-10-11 18:49:19 -07:00
Vikas Bhansali 92ea765040
Vibhansa/v2/newvmaddition (#929)
* Trying out rehl 8.6 and 9.0

* Adding RHEL 8.6 to pipeline

* Correcting perf pipeline outputs

* Correct service file to contain env variables regarding auth

* Add fuse in suse so that unmount can work without fusermount3
2022-10-11 21:53:54 +05:30
Nick Piper 5cf214eb43
Change https_proxy / http_proxy example to a working example (#933)
I found that

`https_proxy=http://172.10.10.10:8888 blobfuse2...`

did not work. Logs show `proxyconnect tcp: dial tcp: lookup http://172.10.10.10: no such host`

However this works: `https_proxy=172.10.10.10:8888 blobfuse2...`

So I'm offering this patch to the readme accordingly
2022-10-11 01:16:55 -07:00
Gauri Prasad 0369d6098f
Added support to work with virtual directories without the special marker directory blob (#927)
* Added support to work with virtual directories

* fix spelling

* Fix lint

* Fix subdir issue

* Correcting infinte loop in getattr using list failure condition

* Add cli flag to enable virtual directory support

* Update ReadME for virtual-dir cli options

* Corrected

Co-authored-by: vibhansa-msft <vibhansa@microsoft.com>
2022-10-07 10:25:01 +05:30
Sourav Gupta a92ef4339b
Mount using /etc/fstab (#924)
* parsing os args

* Removing auto completion

* Update changelog

* Add more flags to ignore

* Small change'

* coverage edit

* Removing dead code

* Add test cases for parse and validate config

* Init logger for UT on config

* Correctly define const slice for fuse ignored flags

* Adding unmount test cases

* Add comments as per review comments

* Correcting unmount tests

* Add logs if mkdir fails

* spell correction in comments

* Run all unmount tests serially with different mount points

* correcting unmount test cases

* Making unmount test parallel

* Adding handle map test cases

Co-authored-by: vibhansa-msft <vibhansa@microsoft.com>
2022-10-05 21:33:52 +05:30
Gauri Prasad 04a0d09d43
Fix release pipeline (#925)
* Fix release

* Fix release

* Fix release
2022-10-03 09:30:26 +05:30
Vikas Bhansali 6f8d90ad71
Docker installation and container image build (#914)
* adding commands to build and launch a container image with blobfuse2 installed

* Adding docker installation steps

* spell correction

* Adding support for fuse2 docker

* Correct fuse installation step

* Correct fuse installation step

* Adding steps to publish container image to azure container registry

* Adding script to map dependency graph

* Changes as per review comments

* Adding docker container related help in README

* Add flag to ignore open flag for write to work

* Correcting ignore open flags related option in dockerfile

* Correct handling of ignore open flags

* Correcting flag on cli

* Run blobfuse in foreground so that container stays up untill unmount is done

* Spell correction

Co-authored-by: Ubuntu <vibhansa@vibhansa-ubn20-perf.x53acyytduluvkundcmr1j05nd.sx.internal.cloudapp.net>
2022-10-01 09:59:38 +05:30
Gauri Prasad cdde152a0d
Fix for ignore open flags (#917)
* Fix typo for ignore open flags

* Changelog

* Fixed error and Readme

* correct the yaml

* try to fix ut

* revert test

* revert test

* revert test
2022-09-30 10:15:09 +05:30
Vikas Bhansali 4efad88930
Reset per file coverage threshold to 70% (#922) 2022-09-30 08:13:28 +05:30
Gauri Prasad 3802cdd853
Added subcommands to health monitor (#916) 2022-09-29 10:28:20 -07:00
Sourav Gupta 855347ccdc
Monitoring cpu and memory in one goroutine (#919) 2022-09-29 14:47:57 +05:30
Vikas Bhansali dba6ed537b
Adding UT for mount all and gen1fuse CLIs (#909)
* Adding more test cases for mount all and gen1 fuse

* Reset flag when its done
2022-09-27 08:41:00 +05:30
Tamer Sherif 770e1f523e
Stream Filename Caching Support (#871)
* filename caching

* filename caching

* locking

* resolved some feedback comments

* baseconfig changes

* fixed tests

* added tests

* comments

* added comments
2022-09-26 12:42:25 -07:00
Gauri Prasad 816ec4c77f
Run scripts in performance pipeline (#888) 2022-09-26 07:52:07 -07:00
Tamer Sherif ec6eb72a29
Script to test attr calls after writes + perf vm fix (#911)
* pipeline fix

* pipeline fix
2022-09-23 10:32:58 -07:00
Tamer Sherif d4a3b4e0c7
Health Monitor Readme + Generated CLI Docs (#907)
* Health Monitor Readme and generate new docs

* Health Monitor Readme and generate new docs
2022-09-19 09:44:40 -07:00
Gauri Prasad 595382bd3f
Update Migration guide (#904) 2022-09-15 11:02:13 -07:00
Vikas Bhansali 765dbbb4a9
Adding component governance to ci pipeline (#902) 2022-09-15 08:18:46 +05:30
Gauri Prasad 7d445c73c9
Fixed parsing of endpoint to support blob endpoints for ADLS accounts (#898) 2022-09-12 07:51:01 -07:00
Sourav Gupta 719d81a434
Unit tests for mount, mountv1 and doc commands (#897) 2022-09-12 11:30:34 +05:30
Sourav Gupta 43e88d02f5
Readme for health monitor (#899)
Adding readme for health monitor
2022-09-12 11:11:51 +05:30
Vikas Bhansali 35d4db0a22
Auto detect auth mode if user has not provided it as config (#896)
* Add code to autodetect the auth mode based on other config

* Add test cases to validate autodetection of authtype
2022-09-12 10:52:34 +05:30
Gauri Prasad f067d5ca23
Clean up exit 1 errors for blobfuse2 (#893) 2022-09-08 13:09:32 -07:00
Sourav Gupta b6d07cb4ab
Update version (#891)
* Ignoring case

* Updating version to preview 4

* Changing to ubn-20

* temporary change

* some changes

* Update to preview 4
2022-09-05 11:09:12 +05:30
Gauri Prasad 23e7af7528
Make writeback cache default (#887)
* stream data validation

* stream data validation

* Enable writeback caching safely

* Flipped boolean semantics for writeback cache

* Added CLI param

* Add debug log for write only in case it is a problem

* Modified test to work with enable writeback cache

* Added a flag to ignore append flag

* github comments

* Added tests for ignore append

* changelog

* Added fuse2 test

* correcting file mode when ignore-append-file flag is true

* fix unit test

* Changes for ignore open flags

* test for libfuse2

* add ignore open flag to all config

* Updated README

* Added to Changelog

* Quick test only in data validation for streaming

Co-authored-by: Tamer Sherif <tasherif@microsoft.com>
Co-authored-by: vibhansa-msft <vibhansa@microsoft.com>
Co-authored-by: souravgupta <souravgupta@microsoft.com>
2022-09-02 17:04:42 +05:30
Sourav Gupta 6ba1e91a7e
Expand path (#880)
Expanding tilde(~) character in mount, config file or cache paths
2022-09-01 19:30:08 +05:30
Tamer Sherif 05630307a1
Stream data validation nightly pipeline step (#886) 2022-08-31 19:01:51 -07:00
Gauri Prasad 1978243d43
Added supported v1 CLI parameters to v2 (#881) 2022-08-31 12:12:14 -07:00
Gauri Prasad 35a032144f
Build fuse package on Ubuntu 20 (#885)
* try to build fuse2 package on ubuntu 20

* try to build fuse2 package on ubuntu 20
2022-08-30 13:00:25 +05:30
Vikas Bhansali 2065b6e84e
Vibhansa/v2/healthmon coverage supress (#883)
* Igmore healthmon code coverage for now

* Correcting command

* Ignore stats-manager
2022-08-30 12:49:35 +05:30
Sourav Gupta 0ac75c29e7
Health monitor (#868)
Adding health monitor for blobfuse2
Co-authored-by: vibhansa-msft <vibhansa@microsoft.com>
2022-08-27 19:47:39 +05:30
Vikas Bhansali 6e05993f75
Add missing cli options to readme (#877)
* Add missing cli options to readme

* Adding missing features and unsupported scenarios

* Spell corrections

* Adding list block cli parameter as csi driver uses that over cli

* Update README.md

Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com>
2022-08-27 11:37:49 +05:30
Gauri Prasad 0a28bfba1d
Added scripts to generate output conveniently (#872) 2022-08-26 10:45:48 -07:00
Gauri Prasad 53399593dc
Changes for enabling writeback cache (#875) 2022-08-26 10:44:04 -07:00
Vikas Bhansali cb4d5445ed
MD5 Sum updation on upload and validation on download (#863)
* Testing migration to 1es hosted pool

* remove indent

* Agent Image

* Remove demands

* Set imageName

* Add correct demand

* Just echo hello world

* Correct indent

* remove the quotes =

* lets try this

* maybe this works

* yeet

* added quotes around the pool name, added agent name demand

* added vmImage name?

* illegal node name?

* hostedAgent = true

* Remove Agent Name

* checkout none

* simple yaml

* try other pool

* no quotes

* UPdate

* update tab

* checkout ne

* Try blobfuse pool

* Added steps to pipeline nightly test

* Fixed indent

* Remove checkout: none

* Add distro test

* fixed indent

* remove checkout

* Fix checkout

* fuse3

* Added RHEL 8

* Override image

* Install

* Added Centos

* Added Oracle

* Added debian

* Added Mariner and SUSE

* Added C build

* Added git to Deban

* Try to get centos 8 to work

* try another cond

* conditional print

* conditional print

* agent condition try 2

* try with variable

* Update OS mirrors

* Fix sed command

* Add to yum

* try removing -e

* no bedst

* Conditional script

* Add more if script

* fix centos 7 stuff

* add flags

* fix check

* tdnf with y option

* Added a stdio lib

* try to fix oracle 8.1 and centos 8.5

* RHEL add flags

* remove ski pborken

* try debian 10

* added build essentials pacakge

* try another rhel

* also install fusermount

* added whoami

* added groups

* trying to run as root

* trying to not run as root

* install fuse

* mariner build essential

* try to use yum to install git

* Add -y

* Get sources list

* remove suse

* spell sources

* find the file that looks for microsoft repos

* remove azure list

* add to debiam

* INstall fuse lib

* try to fix debian 11

* Added MSI test

* reenable suse

* Install cmake and gcc on rhel

* Adding cmake install step for MSI

* Failover methods are not supported in EL8 so remove that confi from default yum repo config file

* install cmake on suse

* Correcting config file name in MSI ste

* Manually install fuse3-devel on SUSE

* Ignore gpg checks in suse to allow installing fuse3-devel

* Correcting suse install step

* Install fuse-libs on rhel vms

* user fusermount3 to unmount as we are not installing fuse on all of the vms

* Trying rhel 7.3 image

* Add RHEL 7.5

* Try to install rh-python

* Trying to install libfuse in RHEL

* sudo for echo

* Use sudo correctly

* add -e

* Updated container and python package for rhrl

* Try to get RHEL 8 to work

* Try to use vault 8 repos

* try mirror

* try without a mrror

* try with enable epel

* add flags

* Get all RHEL to work

* Get rhel 7.5 to work

* Added Ubuntu nightly

* added params

* Use msft managed

* indent

* Use normal nightly pipeline

* Added Blob STE retry policy

* Add credential policy

* Added Datalake retry policy

* Correcting UT compilation failure

* Ignore data validation test on distros other than ubn

* Change retry policy factory name for datalake

* Reducing cache config for streaming test

* Remove data validation test from different auth options

* Remove quick_test as default is anyway true

* Move data validation as a seperate stage

* Move data validation as a seperate stage

* Move data validation as a seperate stage

* Add data validation as a configurable step

* Make exhaustive test disable most of the test cases

* Adding support for md5sum validation and updation

* Add tags for data validation builds

* Try running fuse2 on ubn20 for data-validation

* Try running fuse2 on ubn20 for data-validation

* Merge all git related test cases to save time

* Remove ubn18 from pipelines, and use ubn20 for fuse2 testing

* Use ubn18 for release pipeline as glibc version error is coming on other distros due to this

* Remove clone and e2e test from release pipelines

* Adding basic e2e checks as part of ci

* Undo e2e from ci as multiple ci can run in parallel and running e2e may make it flaky

* Updating changelog for retry policy

* Enable git and tar test cases in nightly data validation test

* Enable git tests in verbose tests

* Remove clone test from verbose test

* Correct code lint command

* Ignore fuse3 test wrapper as well in linting

* Have two different flags for setting and validation md5 sums

* Correct code linting issue

* Adding UT for various combinations of MD5

* File cache shall delete the local file if download fails

* Adding few test cases to azstorage utils

* Remove dead code

* Resolve UT issue with logging

* Use 1ES for ubn-18 in artifact pipeline

* Remove redundancy

* Add dependency

* Add dependency

* Move code coverage to 1ES but remove ubn-22

* Move code coverage to 1ES but remove ubn-22

* Add account cleanup step to code coverage

* Add cleanup stage in code coverage

* Add cleanup stage in code coverage

* Correcting work dir path

* correction as per review comments

* Always run account cleanup at the end

* install go before doing account cleanup

* Make coverage and account cleanup a configurable step

Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-08-25 16:49:56 +05:30
Vikas Bhansali ddeb57c6f3
Add missing file-cache cli (compared with v1) (#874)
* Adding cli options for cache configuration

* correct default values
2022-08-24 22:22:02 +05:30
Tamer Sherif bbd19b2f3f
Streaming small fixes + more tests (#861)
* added tests and cleaned up for prev

* added tests and cleaned up for prev

* more fixes

* fix

* added e2e

* added e2e

* added e2e

* added e2e

* added empty temp path

* fixed temp dir

* removed read only from config

* git clone disable

* added tests

* more tests

* read in buffer err:

* added stream only error test

* more tests

* close file and flush

* removed comment

* cleanup

* fix

* cleanup
2022-08-24 09:22:51 -07:00
Gauri Prasad c1a7890461
Upgrade version to preview 3 (#867) 2022-08-19 09:02:38 +05:30
Gauri Prasad ec4817d1ac
Added Azurite Test (#865) 2022-08-18 12:33:19 -07:00
Gauri Prasad 1533810ab7
fix some codespell errors (#870) 2022-08-18 09:26:10 -07:00
Vikas Bhansali 50d3f9ebe9
Retry on timeout error from server (#858) 2022-08-18 09:00:13 -07:00
Gauri Prasad bc4fd768ef
Added more storage tests (#860) 2022-08-15 08:32:26 -07:00
Gauri Prasad 26030f2754
Added MSI Tests and more to code coverage (#857) 2022-08-09 08:49:23 -07:00