When the buffer isn't filled to capacity a new slice is created that's
smaller than the original. In that case the smaller slice is returned to
the pool which prevents the rest of the capacity from being used again.
The solution is to pass the original slice through and attach the
length. This allows the original slice to be returned to the pool once
the operation is complete.
This change also simplifies the `sendChunk` method, ensuring that the
buffer is returned to the TransferManager even when no bytes were read
from the reader.
* Implement Legal hold & Immutability Policy
* Add immutability policy options to relevant blob functions
* Fix tests, add on create tests
* Fix tests after merge
* Set environment variables
* Only scan for immutability if the container could have it
* Extend testing, close response bodies.
* Increase stack history size
* Larger history size
* Back to 7
* Only use SRP to cleanup immutability containers
* Handle containers not existing
* Add list testing
* Add immutability SAS and tests
* Disable race testing for now
* Added Generated Code | Service Version 2020-10-02 (#315)
* Added Generated Code | Service Version 2020-10-02
* Minor Edit
* TestListBlobsIncludeDeletedWithVersion
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
* Changes to support new user delegation SAS
* Added support for user delegation SAS
* Support generation of Directory SAS
* Correcting UDK struct and updating SAS & Blob SAS query params
* Pinning down old SASVersion
* Renaming for SDK consistency, adding tests
* Reorganizing for CI
* Cleaning up tests
* Cleaning up tests due to CI
* Adding change to resolve CI issues
* Addressing PR comments
* Test for sdd
* Update sdd test
* Fix tests
Co-authored-by: Narasimha Kulkarni <63087328+nakulkar-msft@users.noreply.github.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
* Adding BlobDeleteType to Delete API
* Added test for soft delete and undelete
* Adding skip to this test as verisioning must be disabled in advance
* Permanent Delete method
* Updated test to run with alternate bsu
* Error handling in test
* Adding sleep in test
* Adding sleep after creation
* Improving test for Delete API
* Checking test
* Adding more sleeps
* Updating pipeline
* Break up tests
* Clean up, removing comments
* Update readme directives to allow for obtaining ACLs
* Put the properties in the correct place
* Actually regenerate the code
* Add SAS token permissions for get/set acl
* add extra perms to blob SAS
- TestFilterBlobsUsingAccountSAS: The SAS token didn't have permission
to delete containers. Add the permission to allow cleanup to succeed.
- TestPageBlockWithCPKByScope: The deferred cleanup was commented out,
it's not clear why.
When the buffer isn't filled to capacity a new slice is created that's
smaller than the original. In that case the smaller slice is returned to
the pool which prevents the rest of the capacity from being used again.
The solution is to pass the original slice through and attach the
length. This allows the original slice to be returned to the pool once
the operation is complete.
This change also simplifies the `sendChunk` method, ensuring that the
buffer is returned to the TransferManager even when no bytes were read
from the reader.
* Removed requestId, Timeout, versionId, snapshot from the function signature of GetTags() and SetTags() (#252)
* Changed the function signature of getTags and setTags
* Minor Adjustment
* Review - 1
* Adding Support for PutBlobFromUrl (#251)
* Regenerate SDK
* Regenerate SDK with proper version
* Adding support for PutBlobFromURL
* Correcting regen code and handling other errors
* Resolving comments on PR
* Correcting SetTags method
* Updating tests with GetTags and SetTags
* Remove swagger/Go_BlobStorage/code-model-v1
* Checks value of metadata & reorganized tests
Co-authored-by: Adele Reed <adreed@microsoft.com>
* Keep compatibility with Go 1.13.14. (#264)
* - Keep compliance with Golang 1.13.14. http.Header has no method Values
on older go versions.
* Do not return error when client request id is missing
* Adding MSI Login Example (#241)
* Added MSI login example
* Minor edits
Co-authored-by: zezha-msft <zezha@microsoft.com>
* Offer Knob to Disable Syslog | Default logging to syslog enabled (#268)
* Provide Knob For Syslog
* Reverting changes to GetTags/SetTags & BlobPropertiesInternal (#269)
* Adding MSI Login Example (#241)
* Added MSI login example
* Minor edits
Co-authored-by: zezha-msft <zezha@microsoft.com>
* Reverting changes to GetTags/SetTags & BlobPropertiesInternal
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: zezha-msft <zezha@microsoft.com>
Co-authored-by: siminsavani-msft <77068571+siminsavani-msft@users.noreply.github.com>
Co-authored-by: zezha-msft <zezha@microsoft.com>
* Update go dependencies
* Resolving Concurrency Issues (#275)
* Resolving concurrency issues
* Fixing concurrency test
* Minor changes
* Incremented version and added to changelog
* Updated version to 0.14
Co-authored-by: Mohit Sharma <65536214+mohsha-msft@users.noreply.github.com>
Co-authored-by: Adele Reed <adreed@microsoft.com>
Co-authored-by: Sreejith Kesavan <sreejithemk@gmail.com>
Co-authored-by: zezha-msft <zezha@microsoft.com>
Co-authored-by: Yang Peng <42422145+yangp18@users.noreply.github.com>
* - Keep compliance with Golang 1.13.14. http.Header has no method Values
on older go versions.
* Do not return error when client request id is missing
* Regenerate SDK
* Regenerate SDK with proper version
* Adding support for PutBlobFromURL
* Correcting regen code and handling other errors
* Resolving comments on PR
* Correcting SetTags method
* Updating tests with GetTags and SetTags
* Remove swagger/Go_BlobStorage/code-model-v1
* Checks value of metadata & reorganized tests
Co-authored-by: Adele Reed <adreed@microsoft.com>
See: https://github.com/Azure/azure-storage-blob-go/issues/233
There needs to be a more robust way of controling memory reuse and threadpooling for streaming copies (or maybe all copies, I will not speak to that here).
This introduces a new interface called TransferManager. This provides 3 main fuctions:
- Control buffer allocation
- Control goroutine allocation
- Allow reuse
This allows a developer to control exactly how they wish to handle this for their use case. We provide two implementations:
- Static buffer allocation similar to the original implementation
- Threadpool allocation similar to how it currently works since my change
Additionally, both of these support having shared pools between calls which neither did before.
It also allows for custom creation of implementations by users that better support their needs (a hybrid of the two above, one that adjusts buffer or concurrency based on transfer speed, ...)
Note: the behavior by default returns to the static buffer method here. If azcopy likes the current behavior, it should switch using NewSyncPool()
Like the original change, this does not change the API compatibility. I think that is probably something that should be done, but not in this change.
There are 3 files here that were not part of my change, but were not go fmt, so they have been updated by the go tool:
- bytes_wrtier.go
- zc_*
All tests without the end to end test system that are expected to pass do (comparing the non-changed version against this one).
Co-authored-by: John Doak <jdoak@DANEWMAN-LENOVO.redmond.corp.microsoft.com>
Ze Qian Zhang
siminsavani-msft
adreed-msft
Mohit Sharma
Bertrand Mermet
jared-gs
Aleksandr Chebotov
Kyle Farnung
Xavier B
Gian Lorenzo Meocci
simin
Yang Peng
Ashish Mishra
Sreejith Kesavan
John Doak