### Packages impacted by this PR
### Issues associated with this PR
### Describe the problem that is addressed by this PR
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
### Are there test cases added in this PR? _(If not, why?)_
### Provide a list of related PRs _(if any)_
### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_
### Checklists
- [ ] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
### Packages impacted by this PR
### Issues associated with this PR
### Describe the problem that is addressed by this PR
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
### Are there test cases added in this PR? _(If not, why?)_
### Provide a list of related PRs _(if any)_
### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_
### Checklists
- [ ] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
### Packages impacted by this PR
### Issues associated with this PR
### Describe the problem that is addressed by this PR
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
### Are there test cases added in this PR? _(If not, why?)_
### Provide a list of related PRs _(if any)_
### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_
### Checklists
- [ ] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
Some build output file paths in code coverage config are now out-dated:
- shared policies are now from storage-blob so they are no longer needed
in `exclude` list.
- output files for queue are now under dist-esm/storage-queue
This PR updates these config files to have correct paths.
### Packages impacted by this PR
### Issues associated with this PR
### Describe the problem that is addressed by this PR
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
### Are there test cases added in this PR? _(If not, why?)_
### Provide a list of related PRs _(if any)_
### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_
### Checklists
- [ ] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
---------
Co-authored-by: Timo van Veenendaal <timov@microsoft.com>
Co-authored-by: Harsha Nalluru <sanallur@microsoft.com>
***NO_CI***
The combination of `nyc` + `esm` is broken in latest versions of
NodeJS (https://github.com/istanbuljs/nyc/issues/1530#issuecomment-1773403365).
Since `esm` package is no longer actively maintained and its repo is archived,
it's less likely that the issue will be fixed soon.
This change switches to use another code coverage tool `c8` which is not
affected. `c8` respects `.nycrc` config files so those are not renamed.
***NO_CI***
- update engine.node to >=16.0.0 in package.json
- update @types/node version to ^16.0.0
- update dev-tool sample's MIN_SUPPORTED_NODE_VERSION to 16
- update eslint-plugin's rules and tests related to engine.node version
- remove TextEncoder and TextDecoder stubs as they are now on global object
- fix tests compiler error due to better typings in v16
- update some README files to not reference version like 14.x.x
- along with it, `@types/mocha` version to `^10.0.0`
- add `esm` dev dependency as they are used, but not explicitly list
- use dev-tool run command for test scripts as much as possible
- fix test issue caused by Mocha behavior change around test name
- move `import "chai/register-should"` to mocha -r command line option for core
http tests
***NO_CI***
- bump `mocha` version
- update dev-tool test commmand to add
`TS_NODE_COMPILER_OPTIONS='{"module":"commonjs"}'` for TS input
- update dev-tool test command to apply a Mocha workaround of requiring
`esm` for ESM JS input
- add missing `cross-env` or `esm` dev dependencies
- adjust some test's import of `http` and `https`.
- update dependency testing script
It is desirable to centralize our NPM test scripts so that we can change
common options easily. Currently test-proxy is always launched, which
isn't useful for packages (e.g., core) that don't have recorded tests.
This PR adds an opt-in option to not launch test-proxy when running
tests.
### Packages impacted by this PR
`@azure/dev-tool`
***NO_CI***
Most of our packages, if not all, have dev dependency on `ts-node` either
directly (`mocha --require ts-node/register`) or indirectly via dev-tool (`run
test:node-ts-input`). Currently tests are running fine because mocha is able to
resolve ts-node currently. It may fail in other cases though (e.g., after
migrating a package to ESM).
This ensures `ts-node@^10.0.0` and `types/node` are included for our rush
packages.
Two tests are not awaiting on `dirClient.exists()` calls, which some
times caused weird test failure probably due to different timing in
async calls where test recorder couldn't find recording for some tests.
This PR adds await for the two calls.
### Packages impacted by this PR
`@azure/storage-file-share`
### Describe the problem that is addressed by this PR
When the empty string was passed as a continuationToken to the `byPage`
method of any `list` operation, the constructed url would have an empty
marker parameter `&marker=&`.
This was causing issues with SharedKey authentication since we now share
the same credential signing logic as `storage-blob` and the
implementation of `getURLQueries` in storage-blob ignores query
parameters without a value:
b4e1a63cd2/sdk/storage/storage-blob/src/utils/utils.common.ts (L371)
However, the previous copy of this method in `storage-file-share` did
not:
b4e1a63cd2/sdk/storage/storage-file-share/src/utils/utils.common.ts (L306)
Notice the missing `&& lastIndexOfEqual < value.length - 1` in the
second implementation.
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
While it is possible to change the way storage-blob does this signing to
match storage-file-share, given that storage-blob is a far more popular
package, there is considerable risk to changing behavior there.
Since existing storage-file-share customers (such as Storage Explorer)
may be already passing the empty string today, there is a strong desire
to maintain that contract.
Rather than diverge the signing logic around SharedKeyCredential again,
I believe the least invasive fix is to have the list methods in
storage-file-share ignore the empty string as demonstrated in this PR.
Port PR https://github.com/Azure/azure-sdk-for-js/pull/26159
- port source and test change for blob
- update one negative test. We used to throw an error when splting an
undefined `parsedUrl.pathname` in `getAccountNameFromUrl`. In core v2 we
changed to use `URL` instead of `URLBuilder`. `parsedUrl.pathname` is
now an empty string, an empty `accountName` is returned instead of error
thrown.
- duplicate source change for other storage package as we currently are
doing for internal util methods.
- update `sinon` version to `^15.0.0`. The breaking changes does not
affect our usages.
- upgrade `@types/sinon` to `^10.0.0`. This reveals test issue where
mocked tokens are not compatible with `AccessToken`. This PR fixes the
tests too.
Port PR https://github.com/Azure/azure-sdk-for-js/pull/25675
- update blob service version to 2023-01-03
- add "appendBlock - append large block" test
- port file-datalake source changes
- port file-datalake tests and add recordings
We used to re-export types, or have copies of same types from each
storage package so customers have the conveniences of not importing them
from another package. This PR adds back some missing types that were
exported.
### Packages impacted by this PR
`@azure/storage-datalake`
### Describe the problem that is addressed by this PR
This package has long since been replaced by
`@azure/storage-file-datalake`, hasn't been published in 4 years, and is
deprecated on npm.
### Packages impacted by this PR
`@azure-tests/perf-storage-blob`
`@azure-tests/perf-core-rest-pipeline`
`@azure/test-utils-perf`
### Issues associated with this PR
#24525
### Describe the problem that is addressed by this PR
Remove last references to `core-http` from the tree as it will soon be
deprecated.
- port tests
- use isBrowser from core-util
- port the source changes
- re-generate client
- update test resource names to use different variable names. New
recorder
unique names are slightly different from previous version and cause same
name to
be retrieved for "dir" variable.
- add recordings
### Packages impacted by this PR
`@azure/storage-queue`
### Issues associated with this PR
Fixes#15813Closes#15594
### Describe the problem that is addressed by this PR
This PR migrates storage-queue to the new core pipeline in the same way
that storage-file-share was migrated. There are no changes to the public
surface and existing recorded tests still pass.
### Provide a list of related PRs _(if any)_
https://github.com/Azure/azure-sdk-for-js/pull/26102
### Packages impacted by this PR
`@azure/storage-file-share`
### Issues associated with this PR
#23427
### Describe the problem that is addressed by this PR
Migrate to use the latest test recorder based on the standard EngSys
test proxy. This will allow us to migrate recordings out of the
repository in a future PR.
### Packages impacted by this PR
`@azure/storage-file-share`
### Issues associated with this PR
#15813
### Describe the problem that is addressed by this PR
This PR migrates storage-file-share to the new core pipeline in the same
way that storage-blob and storage-file-datalake were migrated. There are
no changes to the public surface and existing recorded tests still pass.
### Provide a list of related PRs _(if any)_
#24141, #24835
### Packages impacted by this PR
`@azure/storage-blob`
### Describe the problem that is addressed by this PR
When generating the client library, the OpenAPI input specification uses
path parameters to model sending the container and blob ids to the
service. However, since these ids require special handling to be escaped
properly, the SDK manages the complete URL itself.
This causes the path parameters to be duplicated when processed by
core-client. To avoid this problem, I previously used a pipeline policy
that removed the duplicate information.
While migrating `storage-file-datalake`, I thought of a different
solution, namely that of having the client itself modify the
OperationSpec directly before the request is generated. This results in
less work at runtime and is a bit cleaner in terms of making each
operation request.
This PR ports the same approach to storage-blob.
### Packages impacted by this PR
@azure/storage-queue
### Issues associated with this PR
Didn't find any when doing a quick search
### Describe the problem that is addressed by this PR
Current code example says that "iterator" should be used to obtain the
value, but it is item where one should get the value from
### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?
### Are there test cases added in this PR? _(If not, why?)_
No code change per say, just example / usage change
### Provide a list of related PRs _(if any)_
### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_
### Checklists
- [x] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
### Packages impacted by this PR
`@azure/storage-file-datalake`
### Issues associated with this PR
https://github.com/Azure/azure-sdk-for-js/issues/23427
### Describe the problem that is addressed by this PR
Migrate to use the latest test recorder based on the standard EngSys
test proxy. This will allow us to migrate recordings out of the
repository in a future PR.
### Packages impacted by this PR
`@azure/storage-file-datalake`
### Issues associated with this PR
#15813
### Describe the problem that is addressed by this PR
This PR migrates storage-file-datalake to the new core pipeline in the
same fashion that storage-blob was recently migrated. There are no
changes to the public surface and existing recorded tests still pass.
### Provide a list of related PRs _(if any)_
#24141
### Packages impacted by this PR
`@azure/storage-blob-changefeed`
### Issues associated with this PR
https://github.com/Azure/azure-sdk-for-js/issues/23427
### Describe the problem that is addressed by this PR
Migrate to use the latest test recorder based on the standard EngSys
test proxy. This will allow us to migrate recordings out of the
repository in a future PR.
use `removeHeaderSanitizer` to remove the `x-ms-encryption-key` header
from the recordings. The header value contains test resource keys that
made credscan unhappy.
### Packages impacted by this PR
`@azure/storage-blob-changefeed`
### Issues associated with this PR
Fixes#21311
### Describe the problem that is addressed by this PR
This PR migrates to core-tracing GA and updates tests accordingly.
### Provide a list of related PRs _(if any)_
#24773#24548
***NO_CI***
- `@rollup/plugin-commonjs` to `^24.0.0`
- `@rollup/plugin-json` to `^6.0.0`
- `@rollup/plugin-multi-entry` to `^6.0.0`
- `rollup-plugin-polyfill-node` to `^0.12.0`
- `@rollup/plugin-inject` to `^5.0.0`
- `@rollup/plugin-replace` to `^5.0.0`
Fix notification-hubs rollup test config: now sourcemaps plugin is complaining
about .ts file. Move typescript plugin before it.
Fix service-bus to include needed rollup dependencies. It was lucky to be able
to re-use on other packages to install them but is failing in some builds.
### Packages impacted by this PR
storage-blob-changefeed
### Describe the problem that is addressed by this PR
Migrated to use `core-rest-pipeline` (coreV2). Since this package
doesn't directly expose REST operations, there is very little change.
`core-http` only provided a few common interfaces that have been moved
to other packages.
Migrate `storage-blob` to `core-rest-pipeline`.
Stable development of storage has moved to the `feature/storage/stable` branch until acceptance testing of main has completed.
* [engsys] fix check-format failures
- keyvault-command doesn't have samples-dev directory
- result of of `rush format`. formatting changes that potentially caused by prettier upgrade
* form-recognizer api.md formatting change
* upgrade dev dependency `puppeteer` to `^19.2.2`
* update pnpm-lock.yaml
* Fix failing tests
by changing them to verify `error.name`. The error message from parsing an
incomplete JSON string has been improved in recent versions of chromium.
Unfortunately it doesn't match NodeJS version any more.
Currently we copy additional properties into the result if they are not marked
as handled. In cases where `xmlName` is different from `seralizedName`, we end
up copying unwanted additional property that has been de-serialized already
because the marked flag is set for `serializedName`.
This PR marks an XML property as handled after it is de-serialized.
***NO_CI***
- string replace in package json: 12 => 14 for engines/node and for dependency @types/node
- eslint: update `json-engine-is-present` rule to 14.0.0 as LTS version
- identity: react to typing improvements for `readFile()`'s `options.encoding`
- trivial generated api.md file changes due to @types/node version bump
* Use escape path like dotnet for storage blob names
* Rerecord test cases.
* Remove it.only to fix tests
* Rerecord test cases for testing against browser
Co-authored-by: Marco Wiedemeyer <mwiedeme@outlook.com>
Co-authored-by: Emma Zhu <emmazhu@microsoft.com>
# Fixing LRO cancellation by deprecating `cancelOperation`
Canceling a long-running operation is currently done by calling `cancelOperation`: eb0aa1da67/sdk/core/core-lro/src/poller.ts (L79-L82) However the cancellation behavior could vary between APIs in a way that makes `cancelOperation` a confusing abstraction. In particular, when sending a cancellation request, the service may respond with either a 200 or a 202 response, depending on the API and/or the operation, where the former indicates the operation has been canceled immediately and the latter indicates it could be some time before the operation is deemed as canceled. Handling 202 responses needs care to set the customer's expectation about wait times and this is typically done by providing a poller object that customer could use to optionally block until the operation finishes. This discrepancy in behavior makes `cancelOperation` not a suitable abstraction for the case when cancellation returns a 202 response.
In this PR, I am deprecating `cancelOperation` and leaving it up to the library author to implement operation cancellation as they see fit. This proposal has been discussed with @bterlson, @joheredi, and @witemple-msft and they're all on-board with it.
## Pollers without cancellation abstraction
The PR also introduces a second poller interface that doesn't have `cancelOperation` at all, named `SimplePollerLike`. This interface is meant to be used for pollers for new operations that don't support cancellation.
The other difference between `PollerLike` and `SimplePollerLike` is in how the operation state is defined. `PollerLike` defines its state shape as a type that extends `PollOperationState<TResult>`: eb0aa1da67/sdk/core/core-lro/src/pollOperation.ts (L17-L38) Mainly, it has `isStarted`, `isCanceled`, and `isCompleted` booleans. However, the semantics of `isCanceled` can be confusing; should it be set at the time when the cancellation request is sent or when the service responds that the operation is canceled?
To avoid this confusion, `OperationState` replaces those booleans with a status field typed as a union of states:
```ts
export interface OperationState<TResult> {
/**
* The current status of the operation.
*/
status: "notStarted" | "running" | "succeeded" | "canceling" | "canceled" | "failed";
/**
* Will exist if the operation encountered any error.
*/
error?: Error;
/**
* Will exist if the operation produced a result of the expected type.
*/
result?: TResult;
}
```
Which makes it clear that it reflects operation status after each poll.
## Use case: Implement LRO cancellation in @azure/ai-language-text (https://github.com/Azure/azure-sdk-for-js/pull/22852)
The proposal is implemented in a different PR: https://github.com/Azure/azure-sdk-for-js/pull/22852
This LRO cancellation returns a 202 response so cancellation itself is an LRO. I am proposing to implement cancellation by returning a function named `sendCancellationRequest` alongside the operation poller, so the return type is:
```ts
interface AnalyzeBatchOperation {
poller: AnalyzeBatchPoller;
sendCancellationRequest(): Promise<void>;
}
```
And client code can simply destruct the output object as follows to get access to the poller and the cancellation function:
```ts
const { poller, sendCancellationRequest } = await client.beginAnalyzeBatch(actions, documents, "en");
...
// Let's cancel the operation
await sendCancellationRequest(); // poller.cancelOperation() no longer exists
```
Design considerations:
- cancellation as a concept is not related to polling so a poller is not the right place to send cancellation requests
- canceling an operation requires knowing the operation ID/location, but we shouldn't ask customers for that information because it is an implementation detail
- cancellation is an LRO but it doesn't have its own operation location, i.e. the operation location is the same for the operation being cancelled and the cancellation operation itself. This means there is no need to create a dedicated poller for the cancellation operation and customers could still use the existing poller to await their operation or its cancellation
- cancellation is a POST request with no interesting response, so it should be modeled programmatically as `(): Promise<void>` function
***NO_CI***
This is a follow-up to PR #21536 and PR #21537.
- upgrade typescript to 4.6.0 for communication packages
- upgrade typescript to 4.6 for test-utils and dev-tool
- upgrade typescript to 4.6
* Change to return empty change feed when meta/segments.json is not created yet.
* Added ability to set Change Feed chunk download size
* Resolve comments
- Reverts unintentional move to westus in #13886
- Aligns storage-blob with storage-file-share and storage-file-datalake
- Ensures access is over ipv4, which is required for some SAS tests
- If both agent VMs and storage account are in westus, access is over ipv6
The perf framework previously did not support true multi-core operation. This PR provides an implementation of multicore support based on a message-passing model.
Two new options are added to the perf framework as part of this PR:
* `--cpus`/`-c`: number specifying the number of cores (CPUs) to use. Defaults to 1, set to 0 to match the number of cores on the machine.
* `--use-worker-threads`: boolean, defaults to false. Pass this flag to use `worker_threads` instead of `child_process` for multithreading (see Multithreading implementations, below)
* Fix issue to expose blob properties in raw response.
* Fix samples issues of: no request in error's detailed info, data lake file system name should be lower cased.
for packages that has their own rollup configurations to match what's used in dev-tool.
- rollup to v2.60.2
- rollup-plugin-sourcemaps to v0.6.3
- @rollup/plugin-node-resolve to v13.0.6
* Add support for file change time property support in storage file package.
Change timestamp headers to be optional, add support for last write time in putRange/putRangeFromUrl methods.
Add support for content type header in rename
* Fix code review comments
Jeremy Meng
EmmaZhu-MSFT
Ben Broderick Phillips
Deyaaeldeen Almahallawi
Jeremy Meng
kazrael2119
Azure SDK Bot
Jeff Fisher
Alexander Wennerstrøm
Harsha Nalluru
Mike Harder
Dina Berry
Timo van Veenendaal
Marco Wiedemeyer
azraelczw
Wei Jun
Sarangan Rajamanickam
praveenkuttappan
Matthew Podwysocki
