AzureStor/NEWS.md

AzureStor 3.0.0.9000

• Uploading with ADLS now sets the Content-Type property correctly.
• Fixes to support blob/ADLS interoperability, which has just gone GA.

AzureStor 3.0.0

Significant user-visible changes

• Substantial enhancements to multiple-file transfers (up and down):
  • These can now accept a vector of pathnames as the source and destination arguments.
  • Alternatively, for a wildcard source, add the ability to recurse through subdirectories. Any directory structure in the source will be reproduced at the destination.
• Related to the above: the file transfer methods can now create subdirectories that are specified in their destination argument. For ADLS and blob uploading this happens automatically; for Azure file uploading it requires a separate API call which can be slow, so is optional.
• The default destination directory when transferring files (not connections) is now the (remote) root for uploading, and the (local) current directory for downloading.
• Significant changes to file storage methods for greater consistency with the other storage types:
  • The default directory for list_azure_files is now the root, mirroring the behaviour for blobs and ADLSgen2.
  • The output of list_azure_files now includes the full path as part of the file/directory name.
  • Add recursive argument to list_azure_files, create_azure_dir and delete_azure_dir for recursing through subdirectories. Like with file transfers, for Azure file storage this can be slow, so try to use a non-recursive solution where possible.
• Make output format for list_adls_files, list_blobs and list_azure_files more consistent. The first 2 columns for a data frame output are now always name and size; the size of a directory is NA. The 3rd column for non-blobs is isdir which is TRUE/FALSE depending on whether the object is a directory or file. Any additional columns remain storage type-specific.
• New get_storage_metadata and set_storage_metadata methods for managing user-specified properties (metadata) for objects.
• Revamped methods for getting standard properties, which are now all methods for get_storage_properties rather than having specific functions for blobs, files and directories.
• Creating a service-specific endpoint (file_endpoint, blob_endpoint, adls_endpoint) with an invalid URL will now warn, instead of throwing an error. This enables using tools like Azurite, which use a local address as the endpoint. Calling storage_endpoint with an invalid URL will still throw an error, as the function has no way of telling which storage service is required.

Other changes

• Remove the warning about ADLSgen2 not supporting shared access signatures (SAS).
• Background process pool functionality has been moved to AzureRMR. This removes code duplication, and also makes it available for other packages that may benefit.
• Only display the file transfer progress bar in an interactive session.
• Export do_container_op and call_storage_endpoint to allow direct calls to the storage account endpoint.
• Fix S3 dispatching bugs in copy_url_to_storage and multicopy_url_to_storage.
• A NULL download destination will now actually return a raw vector as opposed to a connection, matching what the documentation says.

AzureStor 2.1.1

• Correctly handle ADLSgen2 and file storage listings with more than 5000 files.
• Fix a bug in confirmation prompts. On R >= 3.5, prompts will now use utils::askYesNo; as a side-effect, Windows users who are using RGUI.exe will see a popup dialog box instead of a message in the terminal.

AzureStor 2.1.0

• Update client API versions to "2018-11-09".
• Fix bug with blob uploading using azcopy.
• Add copy_url_to_blob function, for directly copying a HTTP[S] URL to blob storage. The corresponding generic is copy_url_to_storage, with a method for blob containers (only).

AzureStor 2.0.2

• By default, HTTP(S) requests to the storage endpoint will now be retried on encountering a network error. To change the number of retries, call options(azure_storage_retries=N) where N >= 0. Setting this option to zero disables retrying.
• Downloading now proceeds in blocks, much like uploading. The default block size is set to 16MB for blob and ADLSgen2, and 4MB for file storage. While this reduces the throughput slightly (basically there is at least one extra REST call involved), it allows retrying a failed transfer on a per-block basis rather than having to redownload the entire file.
• Also display the progress bar for uploads. The command to enable/disable the progress bar is now options(azure_storage_progress_bar=TRUE|FALSE).

AzureStor 2.0.1

• Display a progress bar when downloading single files. To turn this on or off, call options(azure_dl_progress_bar=TRUE|FALSE).
• Fix upload_to_url/download_from_url bugs introduced in last update.

AzureStor 2.0.0

Significant user-visible changes

• Support authentication via Azure Active Directory tokens for blob and ADLSgen2 storage.
• Support uploading and downloading to in-memory R objects, without having to create a temporary file. Uploading can be done with src a rawConnection or textConnection object. For downloading, if dest is NULL, the downloaded data is returned as a raw vector, or if dest is a rawConnection, in the connection object. See the examples in the documentation.
• Implement parallel file transfers using a background pool of R processes. This can significantly speed up transfers when working with multiple small files.
• Experimental support for using the Microsoft AzCopy commandline utility to perform file transfers. Set the argument use_azcopy=TRUE in any upload or download function to call AzCopy rather than relying on internal R code. The call_azcopy function also allows you to run AzCopy with arbitrary arguments. Requires AzCopy version 10.
• New generics for storage operations:
  • storage_container, create_storage_container, delete_storage_container, list_storage_containers for managing containers (blob containers, file shares, ADLSgen2 filesystems)
  • storage_upload, storage_download, storage_multiupload, storage_multidownload for file transfers
  • list_storage_files, create_storage_dir, delete_storage_dir, delete_storage_file for managing objects within a container

Other changes

• Add ADLS upload/download support to upload_to_url and download_from_url.
• Set default blocksize for upload_azure_file to 4MB, the maximum permitted by the API (#5).
• Allow resource group and subscription accessor methods to work when AzureStor is not on the search path.

AzureStor 1.0.0

• Submitted to CRAN

AzureStor 0.9.0

• Moved to cloudyr organisation