* probably need to ensure that sanitize_setup and create_build_and_install both submit True as well, but let's see if this works
* version_set_dev implies that we will always be working in a dev context, so should pass along dev=True
* add close_vnext_issue function
* add close_vnext_issue function to ci checks that create issues
* add strict sphinx
* log GH issue number we're closing
* remove 3.7
* require pygithub (and from there pynacl) from the default requirements set. this should allow testing on windows pypy39 to successfully install requirements
* remove python_version spec
* rebalance to force windows onto py310. there iwn't a wheel for pywin32 that works with pypy
* Update tools/azure-sdk-tools/setup.py
Encountered some issues with the generation.py script
when comments are in the dev_requirements.txt file. This
change correctly parses them.
Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
* add strict-sphinx for issue creation
* add vnext_issue creator to run_sphinx_build
* add GH_TOKEN to generate docs to step
* update template
* update date to april deadline
* don't need args
* only open issues if running in analyze-weekly
* add in rohit's feedback
* try ci_enabled flag
* try using a gh env var
* fix import
* set env var, add filter in discover_targeted_packages
* add Public param to ci.yml test
* retry setting env var
* pass through template
* missing paren
* don't look for Public in ci.yml
* try again
* remove unrecognized arg
* print vars
* try checking build.reason
* few fixes + prints
* try testing skip
* try more prints
* try more prints
* try more prints
* try nonempty packages list
* ignore discover_targeted_packages for now
* try again
* try again
* try no parameter
* missed import
* accidentally deleted other var
* try again
* fix
* test set PULLREQUEST
* missed var
* set var correctly
* don't fail docs step
* remove condition
* comment
* fake pull request run
* with discover targeted packages filter run
* remove print debugs
* try only publish artifacts if they exist for docs
* remove filter for now
* try again
* set in right tempalte
* switch to a service we actually want to disable
* formatting
* feedback
* block on release if ci_enabled=false
* test disabling ci for template, should block on release
* remove test setting from azure-template
* remove unused import and add small doc
* updates to eng/ci_tools and eng/test_tools.txt to allow necessary additions for compatibility
* added new namespace to azure-sdk-tools: ci_tools.scenario.
* ci_tools.scenario is used to emplace any code that actually does the action of installing a dependency, replacing requirements, building on the fly, etc. This means that create_package_and_install and replace_dev_reqs both are moved into this module
* created tox environment 'optional'. This tox environment utilizes array items of pyproject.toml config item 'tool.azure-sdk-build.optional' to install requirements for specific test scenarios before invoking tests against them. These tests are run within a custom venv for each optional configuration item in pyproject.toml, location .tox/optional/<envname>/
* the new 'optional' environment is enhanced to allow users to pass in the target environment EG: 'tox -e optional --root . -c ../../../eng/tox/tox.ini -- --optional no_requests'
Co-authored-by: Laurent Mazuel <laurent.mazuel@gmail.com>
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* update docs and debugging info
* Update doc/dev/code_snippets.md
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
---------
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
* Temporarily Scaffold new azure-storage-extensions package
* Update parse_functions.py to pick up extension modules and package names
* Add additional code testing parse_function for the new functionality
* Update build.py to invoke cibuildwheel to produce platform-specific wheels where necessary
Co-authored-by: Jacob Lauzon <jalauzon@microsoft.com>
Co-authored-by: Azure SDK Bot <adxpysdk@microsoft.com>
* updated replace_dev_reqs to handle all cases of relative installs to handle local relative deeps _deeper_ than the target package. Previously only handled relative deps starting with ../
* added unit tests for replace_dev_reqs to ensure that packages are replaced (or left alone) as expected
* apply black
* update scenario to include 'here'. update test to use automatic tempdir cleanup per Kevin's feedback
---------
Co-authored-by: kdestin <101366538+kdestin@users.noreply.github.com>
* cleaning up setup_execute_tests. removing code paths that have gone completely unused since being implemented (to mirror previous methodology if necessary). cleaning up arguments that aren't used. cleaning up imports. generally just tightening up of the script and clarifying it's purpose.
* adding the ability to update a pyproject.toml easily
* adding tests for new update functionality
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Add python 3.12 to platform matrix for tests
* Various updates to test_tools.txt and ci_tools.txt to enable compatibility with python 3.12 at test run time
* Adjust build matrix for metricsadvisor and tables to route python 3.11 from windows to ubuntu machine, bypassing issue #32409
* Add manually built cffi and aiohttp to test_tools.txt to enable async recorded tests on python 3.12
* Various updates to minimum range in install_depend_packages.py. All due to various levels of support for py 3.12
* Add the concept of platform specific override for minimum version, allowing the invoking platform to further refine the selection from pypi
* Updates and extensions to tools/azure-sdk-tools to handle cleanup of git idx files with wonky permissions
* Disable latestdependency and mindependency in azure-eventhub-checkpointstoreblob-aio. Follow-up issue #32422
* Various other small updates to enable further compatibility
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
* updates to test sever, removing a couple of the pins from test_tools and ci_tools.txt
* remove bad readme_renderer requirement
* simplifying requirements. fixing requirement file for invoking analyze
* repair issues with pypy39 due to twine installation
* exclude azure-core-tracing-opencensus from sphinx build
* eliminate pytoml dependency
* Update Sphinx to v6.x
* safe import of tomli on <py311. import tomllib on >=py311. always pull in toml-w
Co-authored-by: Laurent Mazuel <laurent.mazuel@gmail.com>
* parse 'keywords' from setup.py in ParsedSetup class
* take advantage of parsed keywords to add new tox environment that verifies that 'azure sdk' is present in the keywords for our packages
* add check to 'analyze' step to invoke newly added keyword check
- Restructure per-service conda builds to a single, unified conda build located within `conda-sdk-client`
- This newly refactored pipeline takes the form of two stages. the first builds multiplatform conda packages against py38,39,310,311. the second builds all our universal (noarch) conda packages.
- Move all conda releaselogs under `conda/conda-releaselogs`
- Move all conda recipes from Xiang's personal build store into the official repo
- Refactor conda build pipeline from a script within `devops_tasks` directory to an entrypoint within `azure-sdk-tools`
- Update conda build within azure-sdk-tools to _also retrieve package data_ prior to assembling a conda package. This totally eliminates the need for a difference in CI and local invocation, and massively simplifies the CI pipeline steps
- Update conda version to 2023.09.01
* remove all references to *.zip files throughout CI
* source distribution and documentation artifact are now all tar.gz files
* update all references to artifact .zip -> .tar.gz
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* try getting build id
* try more env vars in ci
* try print env var
* try print env var
* wip
* add vnext script
* update label for creating issue
* move dep
* add envar to vnext steps
* add gh token to pipeline
* fix package name and lines too long
* move vnext script to azure-sdk-tools/gh_tools per feedback
* cspell
* fix more cspell
This first adds a keyword argument to the TokenCredential protocol method `get_token`.
This enables users and client SDKs to pass in a flag to denote that
`get_token` requests should be requesting CAE tokens.
If the underlying credential's `get_token` implementation supports this
flag, then a CAE token will be requested. Otherwise, a non-CAE token
will be requested.
Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
* selection of wheel now honors the invoking executable's pip compatibility tags to select the correct wheel from a list
* add a unit test testing the selection algo
---------
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* Remove references to {envbindir}
tox prepends the {env_bin_dir} to $PATH, so commands
will default to using the virtual environment first.
See: https://tox.wiki/en/4.4.11/config.html#commands
One of tox's core maintainers also discourages its use:
https://github.com/tox-dev/tox/issues/2909#issuecomment-1409806654
* refactor: Write paths relative to git root, instead of package dir
Uses an inline plugin (toxfile.py) to make a computed config
value avaiable to tox environments.
* refactor: Rename {toxinidir} to {tox_root}
{toxinidir} is an alias for {tox_root}, but {tox_root} is:
* More clear: {tox_root} isn't necessarily where the ini file is
* Shorter
* refactor: Flatten some commands that are no longer super long
* refactor: Make a pytest section
* refactor: Remove `changedir = {tox_root}`
This is the default behavior, see:
https://tox.wiki/en/4.4.11/config.html#change_dir-external
* fix: {distdir} is deprecated
See: https://tox.wiki/en/latest/upgrading.html#removed-tox-ini-keys
* fix: Remove --ignore-installed from pip command
--ignore-installed can break a python installation if multiple
conflicting versions of a package are installed
* refactor: Add descriptions to all tox environments
* fix: Remove ineffective platform config
platform = linux: linux
macos: darwin
windows: win32
Setting the above in the base environment has no effect:
* None of the environments have {linux,macos,windows} in the
name, so the platform config is always empty.
* chore: Bump `tox` to `>4.4.11` and no longer install `tox-monorepo`
tox4 natively solves what `tox-monorepo` was written to solve:
`--root` lets you set `toxinidir` independently of the config
file in use.
4.4.11 was chosen as the cutoff since it was the first release to
include a fix to `--root` that prevented `{work_dir}` from being
changed when `{toxinidir}` was changed by `--root`.
`tox-monorepo` should no longer be needed
* docs: Update CONTRIBUTING.md to use tox 4 commands
* References to tox-monorepo were removed
* Replaced discussion of `tox -l` with tox4's `tox list`
* Updated examples to use `--root`
* docs: Make `azure-media-{videoanalyzer,analytics}-edge`'s DevTips.md point to CONTRIBUTING.md for tox
* docs: Remove remaining references to tox-monorepo
* doc: Use `--root` throughout documentation
* engsys: Make tox_harness set `{toxinidir}` when invoking tox
* engsys: Set required tox version in config
* docs: Use the new url for tox docs
* docs: Add a callout that TOX_* environment variables control defaults
* TOX_CONFIG_DIR can be used to permanently set --conf
* TOX_ROOT_DIR can be used to permanently set --root
* update packaging requirement everywhere it is used.
* fix cspell
* docs: Use `tox run` syntax throughout documentation
* engsys: Use `tox run` syntax in tox_harness.py
* docs: Replace – (U+2013 : EN DASH) with - (ASCII code 45)
* update pytest requirements + coverage.
* update virtualenv
* more conflicts resolved
* bringing in urllib3 restriction, aligning ci_tools.txt with test_tools.txt
* check_call requires a list of type string. if we append ['blah', 'blah'] we are actually inserting another array. just use regular array concatenation
* engsys: Tox uses `run` for serial execution and `run-parallel` for parallel`
* azure-core: Use `@pytest_asyncio.fixture` to mark an asyncio fixture
See https://pytest-asyncio.readthedocs.io/en/latest/concepts.html#strict-mode
* pin coverage to newer version with no conflicts
* identify issue with missing code coverage being caused by wrongly named namespace (#30344)
* remove fix of dotfile!
* apply black formatting to resolve failing analyze step
* tox installing tox. probably not a good thing :)
* engsys: Remove `tox` from `azure-media-analytics-edge` dev_requirements
* align regression version of ci/test_tools.txt
* new coverage format requires that the package sources be present when combining .coverage files. amend our tox tree cleanup to exclude the whl directory when running coverage.
* ensure that missing hidden folder can't break cleanup operation
* ensure coverage isn't generated on whl_no_aio
* ensure that coverage is generated without wonky package paths
---------
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
* pin latest setuptools such that it won't be uninstalled during combined install
* Update eng/pipelines/templates/jobs/tests-nightly-python.yml
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* Moves the analyze_deps script into azure-sdk-tools as an entrypoint 'sdk_analyze_deps' and add typing information to definition file
* Updates the analyze_dependencies module to evaluate the dependency specifiers found across the repo. If any combination of specifiers eliminates all available versions from PyPI (combined with local if present) we will throw. This won't catch transitive dependencies, but it will definitely catch conflicts in directs. A follow-up PR enabling poetry will validate transitive deps.
* Updates our shared_requirements.txt to more of a known dependencies list. It is now more of a "Yes we do have an external dep on isodate. None of the specifiers cause conflicts though."
* add newline cushion between tags and python snippet in snippet updater script
* run snippet updater script on libraries that use sample tags in readme
- When a user uses the `--profile` argument with the PerfStress
framework, sorted pstats output of the cprofile data is now output
to stdout.
- Profiling now supported with Event perf tests
- The warmup stage will now correctly run when profiling is enabled.
- A `--profile-path` argument is added to allow customization
of the output file/path.
- Now we restrict parallelism to 1 when profiling
Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
Co-authored-by: antisch <antisch@microsoft.com>
* update to only apply the is_check_enabled when we are in CI, all tox environments will run regardless if target package is configured to skip them in CI or not.
* now skipping an entire analyze entire step if possible
* now marking skipped package / environment combinations
* don't install azure-core via editable install from dev reqs for verifytypes
* make same change for pyright so we aren't mixing whl/sdist install with editable install when running locally
* create whls locally
* create whls for deps for pyright/verifytypes
* move shared code to ci_tools.functions and share between tox harness and script
* cspell
* some typing improvements
* regen and rerecord now that we don't gen with maxpagesize anymore
* fix typing errs and dependencies
* shared reqs
* enable samples for pyright/mypy
* cspell
* fix cspell again
* update
* update recordings tag
* pep526 type hints
* regen
* eliminate remaining requests usage from azure-sdk-tools
* directly add Paul's recommendations, definitely on point
* ensure that a None header gets flattened to empty string.
* ensure that the urllib3 clients honor the same certificate bundle that requests does
* inactive packages are still assembled, but will not be automatically tested unless included using ENABLE_PACKAGE_NAME=true
* checks can now be individually disabled through the pyproject.toml
* added package classifiers to ParsedSetup properties
* add new function is_check_enabled which combines environment_exclusions, pyproject.toml, and environment overrides into one slick package
* all run_check scripts now honor is_check_enabled to enable/disable atomically
* update eng_sys_checks for new functionality
* add pyproject.toml for all core/ packages
* enable black on only track 2 core packages
* remove package build exclusion list in favor of honoring classifier
* update a couple packages with Inactive classifier so they are no longer built. azure-mgmt-scheduler, azure-mgmt-documentdb, azure-mgmt-regionmove
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* add azd
* apply changes on default, aio and test
* add changelog
* debug test
* modify tests and clear comment
* add --tenant-id
* --tenant-id test
* remove object, add option on init function
* merge
* fix test, add optional package
* Add updates on Readme later once we have the doc
* use https in the link
* pylint
* typing
* update changelog
* pylint
* pylint
* break long string in multiple lines
* verfitypes
* skip typing
* add comment
[Monitor][Query] Improve typing
This enables the mypy, pyright, verifytypes checks in the CI, and also
adds some typing improvements in order to pass the checks..
* Use class attribute style typing
* Class ordering in models file was changed a bit to allow
for class attribute typing.
Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
* opt in to sample type checking
* fix samples
* rename 3.1 async auth sample
* rename more conflicting files
* add typing fixes for copy, custom model, and strongly typed model samples
* fix strongly typed sample
* improve typing for strongly typed sample
* type recognized form result
* type the poller result in sample
* fix typing for invoice argument
* fix copy sample
* fix bounding box sample
* fix custom model samples
* ignore type mismatch in dict samples
* update typing vars
* ignore typing inconsistency
* pylint
* update docs with auth sample rename
* update docs with dict sample renames
* rename var
* using typing.List
* Draft multiproc support
* Improve barrier reliability
* Spelling
* Run global setup per proc
* Update cspell
* Add parameters to SleepTest
- Aligns with .NET and supports more scenarios
* Add LogTest to help observe and debug the framework
* Subprocess spawn instead of fork
* Fix status reporting
* Improved error handling
* Updated docs
* Apply suggestions from code review
Co-authored-by: Paul Van Eck <paulvaneck@microsoft.com>
* Cspell fixes
* Typo
Co-authored-by: Mike Harder <mharder@microsoft.com>
Co-authored-by: Paul Van Eck <paulvaneck@microsoft.com>
# Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
# All SDK Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes]**
- [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.**
- [ ] **I have read the [contribution guidelines](https://github.com/Azure/azure-sdk-for-python/blob/main/CONTRIBUTING.md).**
## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
### [Testing Guidelines](https://github.com/Azure/azure-sdk-for-python/blob/main/CONTRIBUTING.md##building-and-testing)
- [ ] Pull request includes test coverage for the included changes.
* regen with latest autorest
* enable pyright and verifytypes
* lint and add dep to shared reqs
* enable samples checked by mypy/pyright
* regen with latest autorest
* wip
* adding type hints where missing to library code
* fixing mypy errors in samples
* fix more typing issues
* formatting
* not ready for pyright until we can regenerate on newer autorest
* lint fixes
* fixes for models
* expand type hints in clients
* add missing type hints
* add back kwargs
* reorder optional
* regenerate with latest autorest
* run autorest postprocess
* enable pyright,verifytypes,samples for type checking
* exclude auth samples checking by pyright until 27704 is fixed
* fix sample that we skip for pyright
* move exclusion list to a file within sdk-tools that we can import from random places in the CI. clean up tox_harness as the serial invocation is unused, untested, and worthless.
* type and doccomment all touched functions
* replacing in_ci in favor of ci_tools common version that also honors github actions
* enable filtering based on the opt out lists from the unified exclusion set
* rename tox environment from lint -> pylint. update single reference to said environment. update documentation to reflect new location of exclusion list
* re-add azure-ai-textanalytics and azure-ai-metricsadvisor to opt_out lists
* adding further cspell exclusions
Azure SDK Bot
swathipil
Jacob Lauzon
McCoy Patiño
Scott Beddall
Krista Pratico
Yuchao Yan
Paul Van Eck
mccoyp
catalinaperalta
Patrick Hallisey
Xiang Yan
Scott Beddall
Laurent Mazuel
Chenjie Shi
ChenxiJiang333
kdestin
iscai-msft
Ray Chen
Limin Gu
Yalin Li
hemarina
Anna Tisch
Eric Xing
Wei Dong
Dongwon Shin