azure-storage-fuse/internal/attribute.go

107 строки
3.7 KiB
Go
Исходник Обычный вид История

2022-02-09 19:42:28 +03:00
/*
_____ _____ _____ ____ ______ _____ ------
| | | | | | | | | | | | |
| | | | | | | | | | | | |
| --- | | | | |-----| |---- | | |-----| |----- ------
| | | | | | | | | | | | |
| ____| |_____ | ____| | ____| | |_____| _____| |_____ |_____
Licensed under the MIT License <http://opensource.org/licenses/MIT>.
Copyright © 2020-2023 Microsoft Corporation. All rights reserved.
2022-02-09 19:42:28 +03:00
Author : <blobfusedev@microsoft.com>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
*/
package internal
import (
"os"
"time"
"github.com/Azure/azure-storage-fuse/v2/common"
2022-02-09 19:42:28 +03:00
)
Perf and Memory Optimizations (#704) * First commit of blobfuse v2 code * Adding hadle optimization for v2 * Making direct read a configurable option, disabled by default * Rearrange structures to reduce padding and save memory * Correcting attr-cache UT for flag changes * Merging file-cache-test * Make flag check a function * Adding comments to handle methods * Adding more fuse options for optimization * Adding more fuse options for optimization * Removing exectime from red flows * Correcting profiling code to work for both cpu and memory * Cleanup old pprof reports * Adding error log for policy local file removal case and adding 10mil max file limit in attr-cache * Correcting file-cache UT for notInCache cases * Correcting libfuse UT to get handle object out of open return parameters * Addressing review comments * Removing 7.5 old curl from nightly as it does not make sense for v2 to test different curl versions * Tuning fuse parameters for better perf * Fuse parameter tuning * Removing some fuse options as mount is failing on some platforms * Exit pipeline stage if mount failed * Populate fuse connection flag based on kernel capabilities * Log kernel fuse capabilities and the one chosen by blobfuse on given platform * Fail pipeline if blobfuse2 binary is not found in df output * Wait for some time for mount to go through before validating * Correcting validation of mount point * Correcting validation of mount point * Use fuse2 on Deb-10 as with fuse3 we are not able to mount on that distro * Update mount validation for huge listing * Image signing to happen only when release option is chosen Co-authored-by: Gauri Prasad <gapra@microsoft.com> Co-authored-by: Ubuntu <vibhansa@vibhansa-perftest.l1vcg1e4u55e3gwz1dsfkxy2if.tx.internal.cloudapp.net>
2022-02-19 13:26:43 +03:00
func NewDirBitMap() common.BitMap16 {
bm := common.BitMap16(0)
2022-02-09 19:42:28 +03:00
bm.Set(PropFlagIsDir)
return bm
}
Perf and Memory Optimizations (#704) * First commit of blobfuse v2 code * Adding hadle optimization for v2 * Making direct read a configurable option, disabled by default * Rearrange structures to reduce padding and save memory * Correcting attr-cache UT for flag changes * Merging file-cache-test * Make flag check a function * Adding comments to handle methods * Adding more fuse options for optimization * Adding more fuse options for optimization * Removing exectime from red flows * Correcting profiling code to work for both cpu and memory * Cleanup old pprof reports * Adding error log for policy local file removal case and adding 10mil max file limit in attr-cache * Correcting file-cache UT for notInCache cases * Correcting libfuse UT to get handle object out of open return parameters * Addressing review comments * Removing 7.5 old curl from nightly as it does not make sense for v2 to test different curl versions * Tuning fuse parameters for better perf * Fuse parameter tuning * Removing some fuse options as mount is failing on some platforms * Exit pipeline stage if mount failed * Populate fuse connection flag based on kernel capabilities * Log kernel fuse capabilities and the one chosen by blobfuse on given platform * Fail pipeline if blobfuse2 binary is not found in df output * Wait for some time for mount to go through before validating * Correcting validation of mount point * Correcting validation of mount point * Use fuse2 on Deb-10 as with fuse3 we are not able to mount on that distro * Update mount validation for huge listing * Image signing to happen only when release option is chosen Co-authored-by: Gauri Prasad <gapra@microsoft.com> Co-authored-by: Ubuntu <vibhansa@vibhansa-perftest.l1vcg1e4u55e3gwz1dsfkxy2if.tx.internal.cloudapp.net>
2022-02-19 13:26:43 +03:00
func NewSymlinkBitMap() common.BitMap16 {
bm := common.BitMap16(0)
2022-02-09 19:42:28 +03:00
bm.Set(PropFlagSymlink)
return bm
}
Perf and Memory Optimizations (#704) * First commit of blobfuse v2 code * Adding hadle optimization for v2 * Making direct read a configurable option, disabled by default * Rearrange structures to reduce padding and save memory * Correcting attr-cache UT for flag changes * Merging file-cache-test * Make flag check a function * Adding comments to handle methods * Adding more fuse options for optimization * Adding more fuse options for optimization * Removing exectime from red flows * Correcting profiling code to work for both cpu and memory * Cleanup old pprof reports * Adding error log for policy local file removal case and adding 10mil max file limit in attr-cache * Correcting file-cache UT for notInCache cases * Correcting libfuse UT to get handle object out of open return parameters * Addressing review comments * Removing 7.5 old curl from nightly as it does not make sense for v2 to test different curl versions * Tuning fuse parameters for better perf * Fuse parameter tuning * Removing some fuse options as mount is failing on some platforms * Exit pipeline stage if mount failed * Populate fuse connection flag based on kernel capabilities * Log kernel fuse capabilities and the one chosen by blobfuse on given platform * Fail pipeline if blobfuse2 binary is not found in df output * Wait for some time for mount to go through before validating * Correcting validation of mount point * Correcting validation of mount point * Use fuse2 on Deb-10 as with fuse3 we are not able to mount on that distro * Update mount validation for huge listing * Image signing to happen only when release option is chosen Co-authored-by: Gauri Prasad <gapra@microsoft.com> Co-authored-by: Ubuntu <vibhansa@vibhansa-perftest.l1vcg1e4u55e3gwz1dsfkxy2if.tx.internal.cloudapp.net>
2022-02-19 13:26:43 +03:00
func NewFileBitMap() common.BitMap16 {
bm := common.BitMap16(0)
2022-02-09 19:42:28 +03:00
return bm
}
Perf and Memory Optimizations (#704) * First commit of blobfuse v2 code * Adding hadle optimization for v2 * Making direct read a configurable option, disabled by default * Rearrange structures to reduce padding and save memory * Correcting attr-cache UT for flag changes * Merging file-cache-test * Make flag check a function * Adding comments to handle methods * Adding more fuse options for optimization * Adding more fuse options for optimization * Removing exectime from red flows * Correcting profiling code to work for both cpu and memory * Cleanup old pprof reports * Adding error log for policy local file removal case and adding 10mil max file limit in attr-cache * Correcting file-cache UT for notInCache cases * Correcting libfuse UT to get handle object out of open return parameters * Addressing review comments * Removing 7.5 old curl from nightly as it does not make sense for v2 to test different curl versions * Tuning fuse parameters for better perf * Fuse parameter tuning * Removing some fuse options as mount is failing on some platforms * Exit pipeline stage if mount failed * Populate fuse connection flag based on kernel capabilities * Log kernel fuse capabilities and the one chosen by blobfuse on given platform * Fail pipeline if blobfuse2 binary is not found in df output * Wait for some time for mount to go through before validating * Correcting validation of mount point * Correcting validation of mount point * Use fuse2 on Deb-10 as with fuse3 we are not able to mount on that distro * Update mount validation for huge listing * Image signing to happen only when release option is chosen Co-authored-by: Gauri Prasad <gapra@microsoft.com> Co-authored-by: Ubuntu <vibhansa@vibhansa-perftest.l1vcg1e4u55e3gwz1dsfkxy2if.tx.internal.cloudapp.net>
2022-02-19 13:26:43 +03:00
// Flags represented in common.BitMap16 for various properties of the object
2022-02-09 19:42:28 +03:00
const (
PropFlagUnknown uint16 = iota
PropFlagNotExists
PropFlagIsDir
PropFlagEmptyDir
PropFlagSymlink
PropFlagMetadataRetrieved
PropFlagModeDefault // TODO: Does this sound better as ModeDefault or DefaultMode? The getter would be IsModeDefault or IsDefaultMode
)
// ObjAttr : Attributes of any file/directory
type ObjAttr struct {
MD5 Sum updation on upload and validation on download (#863) * Testing migration to 1es hosted pool * remove indent * Agent Image * Remove demands * Set imageName * Add correct demand * Just echo hello world * Correct indent * remove the quotes = * lets try this * maybe this works * yeet * added quotes around the pool name, added agent name demand * added vmImage name? * illegal node name? * hostedAgent = true * Remove Agent Name * checkout none * simple yaml * try other pool * no quotes * UPdate * update tab * checkout ne * Try blobfuse pool * Added steps to pipeline nightly test * Fixed indent * Remove checkout: none * Add distro test * fixed indent * remove checkout * Fix checkout * fuse3 * Added RHEL 8 * Override image * Install * Added Centos * Added Oracle * Added debian * Added Mariner and SUSE * Added C build * Added git to Deban * Try to get centos 8 to work * try another cond * conditional print * conditional print * agent condition try 2 * try with variable * Update OS mirrors * Fix sed command * Add to yum * try removing -e * no bedst * Conditional script * Add more if script * fix centos 7 stuff * add flags * fix check * tdnf with y option * Added a stdio lib * try to fix oracle 8.1 and centos 8.5 * RHEL add flags * remove ski pborken * try debian 10 * added build essentials pacakge * try another rhel * also install fusermount * added whoami * added groups * trying to run as root * trying to not run as root * install fuse * mariner build essential * try to use yum to install git * Add -y * Get sources list * remove suse * spell sources * find the file that looks for microsoft repos * remove azure list * add to debiam * INstall fuse lib * try to fix debian 11 * Added MSI test * reenable suse * Install cmake and gcc on rhel * Adding cmake install step for MSI * Failover methods are not supported in EL8 so remove that confi from default yum repo config file * install cmake on suse * Correcting config file name in MSI ste * Manually install fuse3-devel on SUSE * Ignore gpg checks in suse to allow installing fuse3-devel * Correcting suse install step * Install fuse-libs on rhel vms * user fusermount3 to unmount as we are not installing fuse on all of the vms * Trying rhel 7.3 image * Add RHEL 7.5 * Try to install rh-python * Trying to install libfuse in RHEL * sudo for echo * Use sudo correctly * add -e * Updated container and python package for rhrl * Try to get RHEL 8 to work * Try to use vault 8 repos * try mirror * try without a mrror * try with enable epel * add flags * Get all RHEL to work * Get rhel 7.5 to work * Added Ubuntu nightly * added params * Use msft managed * indent * Use normal nightly pipeline * Added Blob STE retry policy * Add credential policy * Added Datalake retry policy * Correcting UT compilation failure * Ignore data validation test on distros other than ubn * Change retry policy factory name for datalake * Reducing cache config for streaming test * Remove data validation test from different auth options * Remove quick_test as default is anyway true * Move data validation as a seperate stage * Move data validation as a seperate stage * Move data validation as a seperate stage * Add data validation as a configurable step * Make exhaustive test disable most of the test cases * Adding support for md5sum validation and updation * Add tags for data validation builds * Try running fuse2 on ubn20 for data-validation * Try running fuse2 on ubn20 for data-validation * Merge all git related test cases to save time * Remove ubn18 from pipelines, and use ubn20 for fuse2 testing * Use ubn18 for release pipeline as glibc version error is coming on other distros due to this * Remove clone and e2e test from release pipelines * Adding basic e2e checks as part of ci * Undo e2e from ci as multiple ci can run in parallel and running e2e may make it flaky * Updating changelog for retry policy * Enable git and tar test cases in nightly data validation test * Enable git tests in verbose tests * Remove clone test from verbose test * Correct code lint command * Ignore fuse3 test wrapper as well in linting * Have two different flags for setting and validation md5 sums * Correct code linting issue * Adding UT for various combinations of MD5 * File cache shall delete the local file if download fails * Adding few test cases to azstorage utils * Remove dead code * Resolve UT issue with logging * Use 1ES for ubn-18 in artifact pipeline * Remove redundancy * Add dependency * Add dependency * Move code coverage to 1ES but remove ubn-22 * Move code coverage to 1ES but remove ubn-22 * Add account cleanup step to code coverage * Add cleanup stage in code coverage * Add cleanup stage in code coverage * Correcting work dir path * correction as per review comments * Always run account cleanup at the end * install go before doing account cleanup * Make coverage and account cleanup a configurable step Co-authored-by: Gauri Prasad <gapra@microsoft.com>
2022-08-25 14:19:56 +03:00
Mtime time.Time // modified time
Atime time.Time // access time
Ctime time.Time // change time
Crtime time.Time // creation time
Size int64 // size of the file/directory
Mode os.FileMode // permissions in 0xxx format
Flags common.BitMap16 // flags
Path string // full path
Name string // base name of the path
MD5 []byte
Metadata map[string]string // extra information to preserve
2022-02-09 19:42:28 +03:00
}
// IsDir : Test blob is a directory or not
func (attr *ObjAttr) IsDir() bool {
return attr.Flags.IsSet(PropFlagIsDir)
}
// IsSymlink : Test blob is a symlink or not
func (attr *ObjAttr) IsSymlink() bool {
return attr.Flags.IsSet(PropFlagSymlink)
}
// IsMetadataRetrieved : Whether or not metadata has been retrieved for this path.
// Datalake list paths does not support returning x-ms-properties (metadata), so we cannot be sure if the path is a symlink or not.
func (attr *ObjAttr) IsMetadataRetrieved() bool {
return attr.Flags.IsSet(PropFlagMetadataRetrieved)
}
// IsModeDefault : Whether or not to use the default mode.
// This is set in any storage service that does not support chmod/chown.
func (attr *ObjAttr) IsModeDefault() bool {
return attr.Flags.IsSet(PropFlagModeDefault)
}