* apply black and prepare for parsing the ci.yml so that we can understand if testproxy is enabled or disabled
* The language specific cert trust in Language-Settings has been updated to no longer globally set environment variables SSL_CERT_DIR, SSL_CERT_FILE, and REQUEST_CA_BUNDLE. These are dynamically set in proxy_startup.py
* encode the fact that azure-eventhub does not support pypy machines
* add azure-sdk-tools test workflow
* update cspell to account for new workflow
* fix matrix generation when no direct packages are present
* fix a test relying on folder ordering to NOT depend on that folder ordering
* updates to eng/ci_tools and eng/test_tools.txt to allow necessary additions for compatibility
* added new namespace to azure-sdk-tools: ci_tools.scenario.
* ci_tools.scenario is used to emplace any code that actually does the action of installing a dependency, replacing requirements, building on the fly, etc. This means that create_package_and_install and replace_dev_reqs both are moved into this module
* created tox environment 'optional'. This tox environment utilizes array items of pyproject.toml config item 'tool.azure-sdk-build.optional' to install requirements for specific test scenarios before invoking tests against them. These tests are run within a custom venv for each optional configuration item in pyproject.toml, location .tox/optional/<envname>/
* the new 'optional' environment is enhanced to allow users to pass in the target environment EG: 'tox -e optional --root . -c ../../../eng/tox/tox.ini -- --optional no_requests'
Co-authored-by: Laurent Mazuel <laurent.mazuel@gmail.com>
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* updated replace_dev_reqs to handle all cases of relative installs to handle local relative deeps _deeper_ than the target package. Previously only handled relative deps starting with ../
* added unit tests for replace_dev_reqs to ensure that packages are replaced (or left alone) as expected
* apply black
* update scenario to include 'here'. update test to use automatic tempdir cleanup per Kevin's feedback
---------
Co-authored-by: kdestin <101366538+kdestin@users.noreply.github.com>
* cleaning up setup_execute_tests.
* removing code paths that have gone completely unused since being implemented (to mirror previous methodology if necessary).
* cleaning up arguments that aren't used. cleaning up imports.
* generally just tightening up of the script and clarifying it's purpose.
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* Add python 3.12 to platform matrix for tests
* Various updates to test_tools.txt and ci_tools.txt to enable compatibility with python 3.12 at test run time
* Adjust build matrix for metricsadvisor and tables to route python 3.11 from windows to ubuntu machine, bypassing issue #32409
* Add manually built cffi and aiohttp to test_tools.txt to enable async recorded tests on python 3.12
* Various updates to minimum range in install_depend_packages.py. All due to various levels of support for py 3.12
* Add the concept of platform specific override for minimum version, allowing the invoking platform to further refine the selection from pypi
* Updates and extensions to tools/azure-sdk-tools to handle cleanup of git idx files with wonky permissions
* Disable latestdependency and mindependency in azure-eventhub-checkpointstoreblob-aio. Follow-up issue #32422
* Various other small updates to enable further compatibility
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
* Remove references to {envbindir}
tox prepends the {env_bin_dir} to $PATH, so commands
will default to using the virtual environment first.
See: https://tox.wiki/en/4.4.11/config.html#commands
One of tox's core maintainers also discourages its use:
https://github.com/tox-dev/tox/issues/2909#issuecomment-1409806654
* refactor: Write paths relative to git root, instead of package dir
Uses an inline plugin (toxfile.py) to make a computed config
value avaiable to tox environments.
* refactor: Rename {toxinidir} to {tox_root}
{toxinidir} is an alias for {tox_root}, but {tox_root} is:
* More clear: {tox_root} isn't necessarily where the ini file is
* Shorter
* refactor: Flatten some commands that are no longer super long
* refactor: Make a pytest section
* refactor: Remove `changedir = {tox_root}`
This is the default behavior, see:
https://tox.wiki/en/4.4.11/config.html#change_dir-external
* fix: {distdir} is deprecated
See: https://tox.wiki/en/latest/upgrading.html#removed-tox-ini-keys
* fix: Remove --ignore-installed from pip command
--ignore-installed can break a python installation if multiple
conflicting versions of a package are installed
* refactor: Add descriptions to all tox environments
* fix: Remove ineffective platform config
platform = linux: linux
macos: darwin
windows: win32
Setting the above in the base environment has no effect:
* None of the environments have {linux,macos,windows} in the
name, so the platform config is always empty.
* chore: Bump `tox` to `>4.4.11` and no longer install `tox-monorepo`
tox4 natively solves what `tox-monorepo` was written to solve:
`--root` lets you set `toxinidir` independently of the config
file in use.
4.4.11 was chosen as the cutoff since it was the first release to
include a fix to `--root` that prevented `{work_dir}` from being
changed when `{toxinidir}` was changed by `--root`.
`tox-monorepo` should no longer be needed
* docs: Update CONTRIBUTING.md to use tox 4 commands
* References to tox-monorepo were removed
* Replaced discussion of `tox -l` with tox4's `tox list`
* Updated examples to use `--root`
* docs: Make `azure-media-{videoanalyzer,analytics}-edge`'s DevTips.md point to CONTRIBUTING.md for tox
* docs: Remove remaining references to tox-monorepo
* doc: Use `--root` throughout documentation
* engsys: Make tox_harness set `{toxinidir}` when invoking tox
* engsys: Set required tox version in config
* docs: Use the new url for tox docs
* docs: Add a callout that TOX_* environment variables control defaults
* TOX_CONFIG_DIR can be used to permanently set --conf
* TOX_ROOT_DIR can be used to permanently set --root
* update packaging requirement everywhere it is used.
* fix cspell
* docs: Use `tox run` syntax throughout documentation
* engsys: Use `tox run` syntax in tox_harness.py
* docs: Replace – (U+2013 : EN DASH) with - (ASCII code 45)
* update pytest requirements + coverage.
* update virtualenv
* more conflicts resolved
* bringing in urllib3 restriction, aligning ci_tools.txt with test_tools.txt
* check_call requires a list of type string. if we append ['blah', 'blah'] we are actually inserting another array. just use regular array concatenation
* engsys: Tox uses `run` for serial execution and `run-parallel` for parallel`
* azure-core: Use `@pytest_asyncio.fixture` to mark an asyncio fixture
See https://pytest-asyncio.readthedocs.io/en/latest/concepts.html#strict-mode
* pin coverage to newer version with no conflicts
* identify issue with missing code coverage being caused by wrongly named namespace (#30344)
* remove fix of dotfile!
* apply black formatting to resolve failing analyze step
* tox installing tox. probably not a good thing :)
* engsys: Remove `tox` from `azure-media-analytics-edge` dev_requirements
* align regression version of ci/test_tools.txt
* new coverage format requires that the package sources be present when combining .coverage files. amend our tox tree cleanup to exclude the whl directory when running coverage.
* ensure that missing hidden folder can't break cleanup operation
* ensure coverage isn't generated on whl_no_aio
* ensure that coverage is generated without wonky package paths
---------
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
* update to only apply the is_check_enabled when we are in CI, all tox environments will run regardless if target package is configured to skip them in CI or not.
* now skipping an entire analyze entire step if possible
* now marking skipped package / environment combinations
* don't install azure-core via editable install from dev reqs for verifytypes
* make same change for pyright so we aren't mixing whl/sdist install with editable install when running locally
* create whls locally
* create whls for deps for pyright/verifytypes
* move shared code to ci_tools.functions and share between tox harness and script
* cspell
* inactive packages are still assembled, but will not be automatically tested unless included using ENABLE_PACKAGE_NAME=true
* checks can now be individually disabled through the pyproject.toml
* added package classifiers to ParsedSetup properties
* add new function is_check_enabled which combines environment_exclusions, pyproject.toml, and environment overrides into one slick package
* all run_check scripts now honor is_check_enabled to enable/disable atomically
* update eng_sys_checks for new functionality
* add pyproject.toml for all core/ packages
* enable black on only track 2 core packages
* remove package build exclusion list in favor of honoring classifier
* update a couple packages with Inactive classifier so they are no longer built. azure-mgmt-scheduler, azure-mgmt-documentdb, azure-mgmt-regionmove
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* move exclusion list to a file within sdk-tools that we can import from random places in the CI. clean up tox_harness as the serial invocation is unused, untested, and worthless.
* type and doccomment all touched functions
* replacing in_ci in favor of ci_tools common version that also honors github actions
* enable filtering based on the opt out lists from the unified exclusion set
* rename tox environment from lint -> pylint. update single reference to said environment. update documentation to reflect new location of exclusion list
* re-add azure-ai-textanalytics and azure-ai-metricsadvisor to opt_out lists
* adding further cspell exclusions
* add updated logic to resolve assets.json and pass along to record|playback/start
* ensure that parallel tox environments don't accidentally stomp on each other by updating where the local proxy assets are stored during CI runs
* add assets cleanup to ensure we don't overfill the CI agents disks
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
* SDK-Tools refactor
* Refactors scripts present under eng/versioning to namespace ci_tools.versioning within the azure-sdk-tools package. All Integration points in CI are flipped over to consume entrypoints defined in the setup.py.
* Refactors build_packages.py and build_package.py into single module ci_tools.build. This function create_package within this module is used wherever packages are built.
* Moves anything related to setup.py parsing into ci_tools.parsing. More specifically, allows users access to ParsedSetup which has a classmethod from_path that can be aimed at any setup.py or package root.
* Instead of a crazy tuple that we can get lost in the sauce in, it's just strongly typed members of a class now.
* Moves anything related to processing of requirements into ci_tools.functions.
* Type and doc comment all the moved common functionality
* update build_packages to output to a deeper subfolder
* extend find_whl and find_sdst in common_tasks.py/tox_helper_tasks.py to search recursive in preparation for updated artifacts
* remove 'artifacts' artifact. update 'packages' artifact so that package artifacts are stored within folders aligning with each ci.yml artifact
* update archetype-sdk-release to leverage new artifact storage. no more staging.
* update test_regression.py to install package we need to test AFTER we install the dependent package. this ensures that `alpha` versions are always present
* adjustments to allow an argument w/ additional packages in it
* adding a wrapper ci.yml that can be invoked against the entire repo
* passing around a BuildDocs param. this means that the test-all can pass 'no docs' but the default will still generate them
* deleting the nightly build ymls. with some clever parametering we don't need them anymore!
* take advantage of improvements to tox-monorepo, plumb through tox environments in parallel
* update tox.ini to only use `envtmpdir` instead of `distdir` for temp work.
* disabling coverage in sdist environment
* cogservices textanalytics uses core
* changing the whl to use envtmpdir
* replacing dev_requirements during CI build. this should mitigate CI failures due to parallelization of install during tox runs
* ignoring the .tox directory to stop pytest discovery errors during collection
Scott Beddall
Jacob Lauzon
kdestin
Scott Beddall
Krista Pratico
praveenkuttappan
Sean Kane