* Changes to support new user delegation SAS
* Added support for user delegation SAS
* Support generation of Directory SAS
* Correcting UDK struct and updating SAS & Blob SAS query params
* Pinning down old SASVersion
* Renaming for SDK consistency, adding tests
* Reorganizing for CI
* Cleaning up tests
* Cleaning up tests due to CI
* Adding change to resolve CI issues
* Addressing PR comments
* Test for sdd
* Update sdd test
* Fix tests
Co-authored-by: Narasimha Kulkarni <63087328+nakulkar-msft@users.noreply.github.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
* Adding BlobDeleteType to Delete API
* Added test for soft delete and undelete
* Adding skip to this test as verisioning must be disabled in advance
* Permanent Delete method
* Updated test to run with alternate bsu
* Error handling in test
* Adding sleep in test
* Adding sleep after creation
* Improving test for Delete API
* Checking test
* Adding more sleeps
* Updating pipeline
* Break up tests
* Clean up, removing comments
* Update readme directives to allow for obtaining ACLs
* Put the properties in the correct place
* Actually regenerate the code
* Add SAS token permissions for get/set acl
* add extra perms to blob SAS
- TestFilterBlobsUsingAccountSAS: The SAS token didn't have permission
to delete containers. Add the permission to allow cleanup to succeed.
- TestPageBlockWithCPKByScope: The deferred cleanup was commented out,
it's not clear why.
* - Keep compliance with Golang 1.13.14. http.Header has no method Values
on older go versions.
* Do not return error when client request id is missing
* Regenerate SDK
* Regenerate SDK with proper version
* Adding support for PutBlobFromURL
* Correcting regen code and handling other errors
* Resolving comments on PR
* Correcting SetTags method
* Updating tests with GetTags and SetTags
* Remove swagger/Go_BlobStorage/code-model-v1
* Checks value of metadata & reorganized tests
Co-authored-by: Adele Reed <adreed@microsoft.com>
See: https://github.com/Azure/azure-storage-blob-go/issues/233
There needs to be a more robust way of controling memory reuse and threadpooling for streaming copies (or maybe all copies, I will not speak to that here).
This introduces a new interface called TransferManager. This provides 3 main fuctions:
- Control buffer allocation
- Control goroutine allocation
- Allow reuse
This allows a developer to control exactly how they wish to handle this for their use case. We provide two implementations:
- Static buffer allocation similar to the original implementation
- Threadpool allocation similar to how it currently works since my change
Additionally, both of these support having shared pools between calls which neither did before.
It also allows for custom creation of implementations by users that better support their needs (a hybrid of the two above, one that adjusts buffer or concurrency based on transfer speed, ...)
Note: the behavior by default returns to the static buffer method here. If azcopy likes the current behavior, it should switch using NewSyncPool()
Like the original change, this does not change the API compatibility. I think that is probably something that should be done, but not in this change.
There are 3 files here that were not part of my change, but were not go fmt, so they have been updated by the go tool:
- bytes_wrtier.go
- zc_*
All tests without the end to end test system that are expected to pass do (comparing the non-changed version against this one).
Co-authored-by: John Doak <jdoak@DANEWMAN-LENOVO.redmond.corp.microsoft.com>
The patch that tried to de-encapsulate the response status code assumed
the error would be of the `StorageError` type. We now test the error
type before trying to get to the response; if the error is not a
`StorageError`, the log severity will always be increased.
The condition that tests the HTTP status code to decide whether or not to upgrade log severity never sees the 4xx and 5xx responses because they are encapsulated in a `StorageError`.
Addresses #214
* [highlevel] Stop using memory-mapped files
While investigating this SDK for uploading and downloading large blobs
(e.g. 25GB or more) it became apparent that the memory-mapped approach
has some severe limitations:
1. Limits the file size on 32-bit systems (theoretically 4GB, but much
less in practice).
2. Has no backpressure when writing to slower storage mediums.
3. Appears to finish faster, but the OS spends several minutes flushing
the modified RAM to disk afterwards (depends on the speed of the
disk).
On a VM with 16GB of RAM and a slow disk (spinning in this case) the
algorithm quickly overwhelms the available memory and causes severe
performance degradation. It ended up simultaneously trying to flush to
the slow data disk and page out to the slightly faster OS disk.
The solution is to stop using memory-mapped files (at least the way the
SDK currently uses then) and switch to the `io.ReaderAt` and
`io.WriterAt` interfaces. They explicitly allow for parallel access to
non-overlapping regions which make them a good candidate for this
purpose.
Benchmarking large downloads (25GB file) between azcopy 10.4.3 and these
updates using a test app, the difference between them is within 10
seconds. When compared against the original code on a beefy machine with
plenty of RAM the measured execution time is faster, but there is a
little bit of delay while the last of the data flushes from RAM to disk.
* PR feedback
* Set up CI with Azure Pipelines
[skip ci]
* Remove Travis and outdated Gopkg files
* Updated GO Version to 1.15
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
* Get Blob Tag/Set Blob Tag/Find Blob by Tag
* Get Blob Tag/Set Blob Tag/Find Blob by Tag + Writing Tests
* Bug Fix
* Code Refactoring
* Code Refactoring 2
* Test case for findblobsbytags
* Minor Fix 1
* rectified blob tags comment corresponding to ifTags param
* Added tier parameter in upload block blob function signature + Fixed usage + Wrote a test case for validation.
* Added tier parameter in
a. CopyFromURL, CommitBlockList of Block Blob
b. Create (Page Blob)
Fixed all occurrence
* Minor Change
* Added test
* Generated code for 12-12-2019 spec
* Fix test
* Changes
* Basic Testing and modification in WithVersionId function.
* Added Tags and Versions in BlobListingDetails.
* Added Tests
* Added TestCases
* Commented out tests which require versioning disabled.
* Added Tests
* Testcases 1-on-1 with python SDK
* Moved all tests to same file for ease of accessibility
Co-authored-by: zezha-msft <zezha@microsoft.com>
Mohit Sharma
siminsavani-msft
adreed-msft
Ze Qian Zhang
Kyle Farnung
simin
Yang Peng
Ashish Mishra
Sreejith Kesavan
John Doak
Pedro Araujo
Alexey Shvechkov
mohsha-msft
Jonas-Taha El Sesiy