This PR implements `dev-tool migrate` and related commands.
Design document (PR):
https://github.com/Azure/azure-sdk-for-js/pull/25044/files
(rather than repeat what is already described in detail in that PR, I'll
only summarize it here)
## `dev-tool migrate`
- `dev-tool migrate` allows a developer to start a migration pass,
during which each pending migration will be applied in sequence to the
current package.
- Migrations run four phases, all of which are optional:
- Eligibility check: determines whether or not a migration is applicable
to a package
- Precondition check: determines if the migration can run successfully
(for example, the asset sync migration checks for PowerShell and that
the git config is set)
- Execution: runs the automated migration (if there is one -- migrations
_can_ be manual)
- Validation: checks the result of the migration
- Any of the four phases may fail, in which case a helpful error will be
printed and the migration will be suspended. You will then need to
manually correct the migration or give up.
- To give up, use `dev-tool migrate --abort`. This will cancel the
suspended migration and leave the working tree dirty.
- If you correct the migration manually, you can run `dev-tool migrate
--continue` to pick up where the previous command left off.
- You also have `--list` to list pending migrations, `--list-applied` to
list applied migrations, and `--has=<id>` to determine whether or not a
migration has been applied.
## `dev-tool admin create-migration`
Creating a migration is easy. You just run `dev-tool admin
create-migration`. It will prompt you for a migration id & description
as well as optional URL. Then it will instantiate the migration template
and open the new migration in VS Code (if you called the script from
within a vscode terminal). From there, writing a migration is as simple
as implementing a function for each of the phases you want to enable in
the migration.
## Initial migrations
- `sample-config-to-metadata`: moves the `//sampleConfiguration` to
`//metadata` in a package.json file (this is a minor hygiene thing that
can only be justified by doing it automatically).
- `onboard/test-proxy-asset-sync`: onboards a package that uses the
recorder to the test proxy asset sync tool. This migration demonstrates
all of the features of migrations, including how to check for
preconditions, check eligibility, execute & validate, and it shows how
to add new files to the git index if a migration creates a file.
---------
Co-authored-by: Jeff Fisher <xirzec@xirzec.com>
Co-authored-by: Timo van Veenendaal <me@timo.nz>
* [smoke-tests] skip overriding version for dependency not managed in this repo
`@azure/msal-node` is not managed in this repo, so we should keep the version
specified in package.json.
* Update common/smoke-test/Initialize-SmokeTests.ps1
Co-authored-by: Albert Cheng <38804567+ckairen@users.noreply.github.com>
Co-authored-by: Albert Cheng <38804567+ckairen@users.noreply.github.com>
- previously the runner load sample modules in main(). Any error would cause
main() to exit thus skipping rest of tests. Now the samples are loaded when they
are executed so errors in loading modules can be caught.
- the consumeEventsFromServiceBusQueue sample is calling `process.exit()` upon
completion, causing the runner to exit because it currently runs tests in-proc.
Update the test to exit with code 1 when failing.
- Use NodeTestVersion after samples are prepared
- dev-tool is not compatible with Node 8
- Use readdirSync() for Node 8 compat
- Add --harmony_async_iteration flag for Node 8
- Samples depend on async iterators
* Rename common test job files to be consistent with other repos
* Move ci tests into their own file and call via matrix generation job
* Use common platform matrix for ci tests
* [dev-tool] Use //sampleConfiguration for skips
* Replace all instances of smokeTestConfiguration with sampleConfiguration
* Formatting
* Tested file matching logic
* removed a left-over import
* Run smoke test samples after package release
* Simplify smoke test eligibility check
* Add DryRun mode to smoke test initialization script. Simplify smoke test on release check
* [smoke test] Rename Nightly argument to Daily
* Remove separate job and conditionals for smoke test eligibility output
* Minor pipeline fixes
* Use ubuntu 18 vmimage for smoke test eligibility check job
* Add node version template to smoke test eligibility check
* Skip smoke tests when run manifest is empty
* Remove redundant NodeVersion parameter
* Disable digital twins smoke tests temporarily
* Disable form recognizer smoke test files
* Cover more Node LTS versions in smoke test sparse matrix
* Use NodeTestVersion variable and template in smoke test matrix
* Revert to using use-node-version template directly, as smoke tests don't need pnpm
* Update smoke test titles to include node versions. Remove failing entries
* Refactor Initialize-SmokeTests.ps1 script for readability
* Parallelize smoke test resource deploys. Use idiomatic function names.
* Filter jobs started from the initialize smoke test script. Add job timeout.
* Use common logging functions
* [smoke tests] Remove deploy jobs in finally block
* Elaborate on smoke test initialize logic in docs and script. Update .gitignore.
* Clarify NODE_PATH variable setting when running smoke tests locally
* Move smoke test .gitignore entries to git root
* add ability to skip smoke tests based on the presence of environment variables
parameterize test-resources.json
disable cosmos db smoke testing in china cloud
add --ci to smoke test execution
add yargs to smoke tests
smoke tests: @types/yargs is unnecessary, add await in app
turns out @types/yargs is required
use KeyVaultSecrets instead of this
incremental sample compiling progress
first cut at using sample format for smoke tests
Smoke test deploys, onboards samples
Fix pre-samples.js
Add environment variable output to New-TestResources.ps1
Update smoke test package.json
Update-SmokeTests.ps1 - variable outputs for CI, start documenting
New-TestResources.ps1 output environment variables
Update smoke tests yaml
* Remove Smoke Test resources in yaml
* parameters
* Update smoke-tests.yml
* Fix prep-samples.js
* Remove extra *
* Formatting for prep-samples.js
* Revert "Remove extra *"
This reverts commit f0c1b3f36b474f09d2cd49ee2abb327ad0236f10.
* Move Node version higher so Update-SmokeTests.ps1 can use it
* node run.js
* Set AZURE_RESOURCEGROUP_NAME at the top level so delete scripts can clean up after
* Add run.js
* Give ability to specify blank service directory in remove-test-resources.yml
* Add ability to specify resource group name directly to New-TestResources.ps1
* Add devops logging for smoke test failures
* Better command layout
* Update KV test resources to include KEYVAULT_URI for samples
* Fix App Config and KV to use environment variables from the test-resources.json
* Add in package.json configuration to skip certain failing files in smoke tests
* Add ability to skip samples based on configuration information in package.json
* Remove ArmTemplateParameters from smoke tests
* Delete test resources after 24 hours
* Add contacts.js (timing out) to skip
* Add deleteAndRecover.js to skip list
* Skip backupAndRestore.js in KV Secrets
* Improve smoke test error logging
* Skip useProxy.js in Service Bus smoke tests
* Use ARM template params in sovereign clouds
* Remove base smoke test cases
* Use configruable storage endpoint suffix in event hub ARM template
* Use baseName for the basis of the GUID
* Set environment variables from standard config
* Set sample directory when running each test to preserve sample context
* Idempotent naming
* Force $CI to $false
* Use proces.cwd() instead of __dirname
* Skip samples where confiugrations are not present in test-resources.json
* Add support for storage endpoint suffix to storage test-resources.json
* Also exclude copyModel.js
* Compute account name for text analytics so we don't collied with test resources that have the same name
* Add endpoint information for cognitive services
* Skip some storage tests because of hardcoded endpoint suffix
* Skip ai-text-analytics pending further investigation
* Proper spelling for customPipeline
* New-TestResources -- if $CI is true AND a ResourceGroupName is specified, do not mututae the $BaseName
* Always generate a new $BaseName when $CI is $true. If we don't do this and reuse $BaseName then multiple resources of the same type might get deployed with the same name and there will be collisions
* Remove samples-manifest.json
* Use static name for cognitive services user role assignment to avoid smoke test collision
* Form Recognizer should use resource group location
* Role assignment names must be GUIDs
* Update search SKU so deployment succeeds
* Fix form recognizer and text analytrics ARM templates to assign RBAC roles directly to the resources
* Unique guid for Text Analytrics role assignment
* Use relative paths in Initialize-SmokeTests.ps1
* Add --use-packages to samples prep to use package references instead of source references
* Move //smokeTestConfiguration from samples package.json to the package's package.json
* Add in better path support
* Update Smoke Test README.md
* Use globbing to find samples/javascript folders inside packages
* Remove extra files
* Remove hard coded test cases
* Remove smoke-test/test-resources.json
* Use context to set AZURE_AUTHORITY_HOST
* Spacing
* Document -ResourceGroupName parameter
* workingDirector -> workingDirectory
* no inputs
* Skip template samples as the samples do not run properly
* Set environment variables for test application in the initilize script
* Exclude authenticationMethods.js and recognizeContent.js samples because of smoke test errors
* Remove setting AZURE_AUTHORITY_HOST in New-TestResources.ps1 and set it in Initialize-SmokeTest.ps1
* Review feedback: use APPCONFIG_CONNECTION_STRING in samples and tests
* Review feedback KEYVAULT_NAME -> KEYVAULT_URI in all samples and docs for KV
* Make sample.env files consistent
* Skip deploying and onboarding when there is no test-resources.json file
* smoke tests use subscription configuration
* update archetype-sdk-integration.yml as well
* editor confiugration always adds extra new line at end of file. Remove this so the files from eng/common remain untouched
* Update contributing sections to link to CONTRIBUTING.md
* Update the contributing section for rest of libraries
* Remove old CONTRIBUTING.md files
README should point to the main CONTRIBUTING.md under repository root.
We did similar changes earlier (e.g., 73f7f90c4) however these seem leftover
that got missed.
* Link to storage specific contributing guide
* add support for authority host and pipelines
* add direct @azure/identity dependency
* set AZURE_AUTHORITY_HOST
* smoke tests use connection string instead of account/key/url combination
* set start position explicitly
* smoke tests: add set-dev-packages.ts
* use current version of Event Hubs, use dev versions in azure dependencies in package.json
* move smoke tests pipeline to smoke test folder
* update smoke test variable locaiton
* update smoke test variable locaiton
* Revert "update smoke test variable locaiton"
This reverts commit a3a8911c1a.
* Revert "update smoke test variable locaiton"
This reverts commit 7982a9d4b8.
* Revert "move smoke tests pipeline to smoke test folder"
This reverts commit aeb9c27ea3.
* add logging for installed packages, remove package-lock.json
* add matrix
* matrix is part of a strategy
* it's an npm custom command
* delete `set-dev-packages.ts`
* now with quotes around the custom command...
* add step to install TypeScript on the agent
* correctly tab displayName and condition
* case insensitive
* move smoke-tests.yml for better alerting
* [Storage] Enable doc-warden checking for storage READMEs
as they should have required headings now.
* Change docSetting to expect JavaScript in headings instead of JS
* move smoke tests
* move documentation
* Revert "move documentation"
This reverts commit 5cc21e851effdc18572179a954fe8df14bef3da0.
* move smoke tests to common
* npm install for smoke-tests
* revert interface change (needs to happen in step with package version updates)
Jana R
Will Temple
Jeremy Meng
Jeremy Meng
Jonathan Cárdenas
ckairen
Chidozie Ononiwu
Mike Harder
Ben Broderick Phillips
Will Temple
Sima Zhu
Wes Haggard
Ben Broderick Phillips
Danny Amirault
Daniel Jurek
Jonathan Turner