Common-sense `tox` environment skipping (#27487)
* move exclusion list to a file within sdk-tools that we can import from random places in the CI. clean up tox_harness as the serial invocation is unused, untested, and worthless. * type and doccomment all touched functions * replacing in_ci in favor of ci_tools common version that also honors github actions * enable filtering based on the opt out lists from the unified exclusion set * rename tox environment from lint -> pylint. update single reference to said environment. update documentation to reflect new location of exclusion list * re-add azure-ai-textanalytics and azure-ai-metricsadvisor to opt_out lists * adding further cspell exclusions
This commit is contained in:
Scott Beddall
GitHub
Родитель
42b3884dc8
Коммит
3b1c811785
|
|
@ -97,6 +97,8 @@
|
|||
"tools/azure-sdk-tools/setup.py"
|
||||
],
|
||||
"words": [
|
||||
"qnamaker",
|
||||
"mindependency",
|
||||
"automl",
|
||||
"pyyaml",
|
||||
"CONLL",
|
||||
|
|
|
|||
|
|
@ -43,7 +43,7 @@ sdist
|
|||
|
||||
```
|
||||
|
||||
Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `lint` and `mypy` environments.
|
||||
Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `pylint` and `mypy` environments.
|
||||
|
||||
### Example Usage of the common Azure SDK For Python `tox.ini`
|
||||
|
||||
|
|
@ -91,11 +91,11 @@ Used for the local dev loop.
|
|||
|
||||
```
|
||||
|
||||
#### `lint` environment
|
||||
#### `pylint` environment
|
||||
Pylint install and run.
|
||||
|
||||
```
|
||||
\> tox -e lint -c <path to tox.ini>
|
||||
\> tox -e pylint -c <path to tox.ini>
|
||||
```
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -298,7 +298,7 @@ All client libraries in the Python SDK repo are automatically opted in to runnin
|
|||
reason why a particular library should not run type checking, it is possible to add that library to a block list to prevent mypy/pyright
|
||||
from running checks.
|
||||
|
||||
1) Place the package name on the appropriate block list: [eng/tox/environment_exclusion_list.py](https://github.com/Azure/azure-sdk-for-python/blob/main/eng/tox/environment_exclusion_list.py).
|
||||
1) Place the package name on the appropriate block list: [tools/azure-sdk-tools/ci_tools/environment_exclusions.py](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/ci_tools/environment_exclusions.py).
|
||||
2) Open an issue tracking that "library-name" should be opted in to running type checking
|
||||
|
||||
> Note: Blocking your library from type checking is a *temporary* state. It is expected that checks are re-enabled as soon as possible.
|
||||
|
|
|
|||
|
|
@ -5,23 +5,36 @@ testing infrastructure, and demonstrates how to write and run tests for a servic
|
|||
|
||||
### Table of contents
|
||||
|
||||
- [Set up your development environment](#set-up-your-development-environment)
|
||||
- [Integrate with pytest](#integrate-with-the-pytest-test-framework)
|
||||
- [Use Tox](#tox)
|
||||
- [The `devtools_testutils` package](#the-devtools_testutils-package)
|
||||
- [Write or run tests](#write-or-run-tests)
|
||||
- [Set up the test proxy](#perform-one-time-test-proxy-setup)
|
||||
- [Set up test resources](#set-up-test-resources)
|
||||
- [Configure credentials](#configure-credentials)
|
||||
- [Start the test proxy server](#start-the-test-proxy-server)
|
||||
- [Deliver environment variables to tests](#deliver-environment-variables-to-tests)
|
||||
- [Write your tests](#write-your-tests)
|
||||
- [Configure live or playback testing mode](#configure-live-or-playback-testing-mode)
|
||||
- [Run and record tests](#run-and-record-tests)
|
||||
- [Sanitize secrets](#sanitize-secrets)
|
||||
- [Functional vs. unit tests](#functional-vs-unit-tests)
|
||||
- [Further reading](#further-reading)
|
||||
- [Deprecated testing instructions](#deprecated-testing-instructions)
|
||||
- [Python SDK testing guide](#python-sdk-testing-guide)
|
||||
- [Table of contents](#table-of-contents)
|
||||
- [Set up your development environment](#set-up-your-development-environment)
|
||||
- [SDK root directory](#sdk-root-directory)
|
||||
- [Dependency installation](#dependency-installation)
|
||||
- [Open code in IDE](#open-code-in-ide)
|
||||
- [Integrate with the pytest test framework](#integrate-with-the-pytest-test-framework)
|
||||
- [Tox](#tox)
|
||||
- [The `devtools_testutils` package](#the-devtools_testutils-package)
|
||||
- [Write or run tests](#write-or-run-tests)
|
||||
- [Perform one-time test proxy setup](#perform-one-time-test-proxy-setup)
|
||||
- [Set up test resources](#set-up-test-resources)
|
||||
- [Configure credentials](#configure-credentials)
|
||||
- [Start the test proxy server](#start-the-test-proxy-server)
|
||||
- [Deliver environment variables to tests](#deliver-environment-variables-to-tests)
|
||||
- [Write your tests](#write-your-tests)
|
||||
- [Configure live or playback testing mode](#configure-live-or-playback-testing-mode)
|
||||
- [Run and record tests](#run-and-record-tests)
|
||||
- [Sanitize secrets](#sanitize-secrets)
|
||||
- [Special case: SAS tokens](#special-case-sas-tokens)
|
||||
- [Functional vs. unit tests](#functional-vs-unit-tests)
|
||||
- [Further reading](#further-reading)
|
||||
- [Deprecated testing instructions](#deprecated-testing-instructions)
|
||||
- [Define credentials (deprecated)](#define-credentials-deprecated)
|
||||
- [Create live test resources (deprecated)](#create-live-test-resources-deprecated)
|
||||
- [Write your tests (deprecated)](#write-your-tests-deprecated)
|
||||
- [An example test (deprecated)](#an-example-test-deprecated)
|
||||
- [Run and record the test (deprecated)](#run-and-record-the-test-deprecated)
|
||||
- [Purging secrets (deprecated)](#purging-secrets-deprecated)
|
||||
- [Special case: Shared Access Signature (deprecated)](#special-case-shared-access-signature-deprecated)
|
||||
|
||||
## Set up your development environment
|
||||
|
||||
|
|
@ -115,7 +128,7 @@ The Python SDK uses the [tox project](https://tox.readthedocs.io/en/latest/) to
|
|||
To run a tox command from your directory use the following commands:
|
||||
```cmd
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sphinx
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e lint
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pylint
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e mypy
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pyright
|
||||
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e verifytypes
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ steps:
|
|||
"$(TargetingString)"
|
||||
--mark_arg="${{ parameters.TestMarkArgument }}"
|
||||
--service="${{ parameters.ServiceDirectory }}"
|
||||
--toxenv="lint"
|
||||
--toxenv="pylint"
|
||||
--disablecov
|
||||
--filter-type="Omit_management"
|
||||
env: ${{ parameters.EnvVars }}
|
||||
|
|
|
|||
|
|
@ -1,382 +0,0 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
import logging
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
|
||||
|
||||
PYLINT_ACCEPTABLE_FAILURES = [
|
||||
"azure-applicationinsights",
|
||||
"azure-batch",
|
||||
"azure-cognitiveservices-anomalydetector",
|
||||
"azure-cognitiveservices-formrecognizer",
|
||||
"azure-cognitiveservices-knowledge-nspkg",
|
||||
"azure-cognitiveservices-knowledge-qnamaker",
|
||||
"azure-cognitiveservices-language-luis",
|
||||
"azure-cognitiveservices-language-nspkg",
|
||||
"azure-cognitiveservices-language-spellcheck",
|
||||
"azure-cognitiveservices-language-textanalytics",
|
||||
"azure-cognitiveservices-nspkg",
|
||||
"azure-cognitiveservices-personalizer",
|
||||
"azure-cognitiveservices-search-autosuggest",
|
||||
"azure-cognitiveservices-search-customimagesearch",
|
||||
"azure-cognitiveservices-search-customsearch",
|
||||
"azure-cognitiveservices-search-entitysearch",
|
||||
"azure-cognitiveservices-search-imagesearch",
|
||||
"azure-cognitiveservices-search-newssearch",
|
||||
"azure-cognitiveservices-search-nspkg",
|
||||
"azure-cognitiveservices-search-videosearch",
|
||||
"azure-cognitiveservices-search-visualsearch",
|
||||
"azure-cognitiveservices-search-websearch",
|
||||
"azure-cognitiveservices-vision-computervision",
|
||||
"azure-cognitiveservices-vision-contentmoderator",
|
||||
"azure-cognitiveservices-vision-customvision",
|
||||
"azure-cognitiveservices-vision-face",
|
||||
"azure-cognitiveservices-vision-nspkg",
|
||||
"azure-common",
|
||||
"azure-nspkg",
|
||||
"azure-servicemanagement-legacy",
|
||||
"azure-graphrbac",
|
||||
"azure-loganalytics",
|
||||
"azure-servicefabric",
|
||||
"azure-template",
|
||||
"azure-keyvault",
|
||||
"azure-synapse",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-spark",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-nspkg",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-security-attestation",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-purview-nspkg",
|
||||
"azure-purview-scanning",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-account",
|
||||
"azure-purview-administration",
|
||||
"azure-messaging-nspkg",
|
||||
"azure-agrifood-farming",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-developer-devcenter"
|
||||
]
|
||||
|
||||
# omit package from running mypy checks
|
||||
MYPY_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
# omit package from running pyright checks
|
||||
PYRIGHT_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-identity",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-ai-metricsadvisor",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-ai-personalizer",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-data-tables",
|
||||
"azure-ai-textanalytics",
|
||||
"azure-ai-translation-document",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
# omit package from running verifytypes checks
|
||||
VERIFYTYPES_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-identity",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-ai-metricsadvisor",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-ai-personalizer",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-ai-textanalytics",
|
||||
"azure-data-tables",
|
||||
"azure-messaging-webpubsubservice",
|
||||
|
||||
]
|
||||
|
||||
# omit package from running type checkers on samples
|
||||
# note: if removed from this list, you must enable one or both of mypy or pyright checks.
|
||||
TYPE_CHECK_SAMPLES_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-ai-translation-document",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------------------------
|
||||
# DO NOT add packages to the below lists. They are used to omit packages that will never run type checking.
|
||||
|
||||
IGNORE_FILTER = ["nspkg", "mgmt", "cognitiveservices"]
|
||||
FILTER_EXCLUSIONS = ["azure-mgmt-core"]
|
||||
IGNORE_PACKAGES = [
|
||||
"azure-applicationinsights",
|
||||
"azure-servicemanagement-legacy",
|
||||
"azure",
|
||||
"azure-storage",
|
||||
"azure-monitor",
|
||||
"azure-servicefabric",
|
||||
"azure-keyvault",
|
||||
"azure-synapse",
|
||||
"azure-common",
|
||||
"conda-recipe",
|
||||
"azure-graphrbac",
|
||||
"azure-loganalytics",
|
||||
"azure-media-analytics-edge",
|
||||
"azure-media-videoanalyzer-edge",
|
||||
"azure-template",
|
||||
]
|
||||
|
||||
|
||||
def is_ignored_package(package_name):
|
||||
if package_name in IGNORE_PACKAGES:
|
||||
return True
|
||||
if package_name not in FILTER_EXCLUSIONS and any([identifier in package_name for identifier in IGNORE_FILTER]):
|
||||
return True
|
||||
return False
|
||||
|
|
@ -13,7 +13,7 @@ import os
|
|||
import logging
|
||||
import sys
|
||||
|
||||
from environment_exclusion_list import (
|
||||
from ci_tools.environment_exclusions import (
|
||||
is_ignored_package,
|
||||
MYPY_OPT_OUT,
|
||||
TYPE_CHECK_SAMPLES_OPT_OUT,
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ import os
|
|||
import logging
|
||||
import sys
|
||||
|
||||
from environment_exclusion_list import PYLINT_ACCEPTABLE_FAILURES
|
||||
from ci_tools.environment_exclusions import PYLINT_OPT_OUT
|
||||
from ci_tools.parsing import ParsedSetup
|
||||
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
|
|
@ -43,7 +43,7 @@ if __name__ == "__main__":
|
|||
|
||||
top_level_module = pkg_details.namespace.split('.')[0]
|
||||
|
||||
if pkg_details.name not in PYLINT_ACCEPTABLE_FAILURES:
|
||||
if pkg_details.name not in PYLINT_OPT_OUT:
|
||||
try:
|
||||
check_call(
|
||||
[
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ import os
|
|||
import logging
|
||||
import sys
|
||||
|
||||
from environment_exclusion_list import (
|
||||
from ci_tools.environment_exclusions import (
|
||||
is_ignored_package,
|
||||
PYRIGHT_OPT_OUT,
|
||||
TYPE_CHECK_SAMPLES_OPT_OUT,
|
||||
|
|
|
|||
|
|
@ -15,7 +15,8 @@ import argparse
|
|||
import os
|
||||
import logging
|
||||
import sys
|
||||
from environment_exclusion_list import is_ignored_package, VERIFYTYPES_OPT_OUT
|
||||
|
||||
from ci_tools.environment_exclusions import is_ignored_package, VERIFYTYPES_OPT_OUT
|
||||
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ commands =
|
|||
{toxinidir}
|
||||
|
||||
|
||||
[testenv:lint]
|
||||
[testenv:pylint]
|
||||
skipsdist = true
|
||||
skip_install = true
|
||||
usedevelop = false
|
||||
|
|
|
|||
|
|
@ -219,13 +219,6 @@ if __name__ == "__main__":
|
|||
"--disablecov", help=("Flag that disables code coverage."), action="store_true"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--tparallel",
|
||||
default=False,
|
||||
help=("Flag that enables parallel tox invocation."),
|
||||
action="store_true",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--tenvparallel",
|
||||
default=False,
|
||||
|
|
|
|||
|
|
@ -1,31 +1,25 @@
|
|||
import sys
|
||||
import os
|
||||
import errno
|
||||
import shutil
|
||||
import re
|
||||
import multiprocessing
|
||||
import glob
|
||||
|
||||
if sys.version_info < (3, 0):
|
||||
from Queue import Queue
|
||||
else:
|
||||
from queue import Queue
|
||||
from threading import Thread
|
||||
from typing import List
|
||||
from argparse import Namespace
|
||||
|
||||
from subprocess import Popen, PIPE, STDOUT
|
||||
from common_tasks import (
|
||||
run_check_call,
|
||||
clean_coverage,
|
||||
log_file,
|
||||
read_file,
|
||||
is_error_code_5_allowed,
|
||||
create_code_coverage_params,
|
||||
find_whl
|
||||
find_whl,
|
||||
)
|
||||
|
||||
from ci_tools.functions import discover_targeted_packages
|
||||
from ci_tools.parsing import ParsedSetup
|
||||
from ci_tools.build import create_package
|
||||
from ci_tools.variables import in_ci
|
||||
from ci_tools.environment_exclusions import filter_tox_environment_string
|
||||
|
||||
from pkg_resources import parse_requirements, RequirementParseError
|
||||
import logging
|
||||
|
|
@ -40,51 +34,6 @@ IGNORED_TOX_INIS = ["azure-cosmos"]
|
|||
test_tools_path = os.path.join(root_dir, "eng", "test_tools.txt")
|
||||
dependency_tools_path = os.path.join(root_dir, "eng", "dependency_tools.txt")
|
||||
|
||||
class ToxWorkItem:
|
||||
def __init__(self, target_package_path, tox_env, options_array):
|
||||
self.target_package_path = target_package_path
|
||||
self.tox_env = tox_env
|
||||
self.options_array = options_array
|
||||
|
||||
|
||||
class Worker(Thread):
|
||||
def __init__(self, tasks):
|
||||
Thread.__init__(self)
|
||||
self.tasks = tasks
|
||||
self.daemon = True
|
||||
self.start()
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
func, args, kargs = self.tasks.get()
|
||||
try:
|
||||
func(*args, **kargs)
|
||||
except Exception as e:
|
||||
logging.error(e)
|
||||
finally:
|
||||
self.tasks.task_done()
|
||||
|
||||
|
||||
def in_ci():
|
||||
return os.getenv("TF_BUILD", False)
|
||||
|
||||
|
||||
class ThreadPool:
|
||||
def __init__(self, num_threads):
|
||||
self.tasks = Queue(num_threads)
|
||||
for _ in range(num_threads):
|
||||
Worker(self.tasks)
|
||||
|
||||
def add_task(self, func, *args, **kargs):
|
||||
self.tasks.put((func, args, kargs))
|
||||
|
||||
def map(self, func, args_list):
|
||||
for args in args_list:
|
||||
self.add_task(func, args)
|
||||
|
||||
def wait_completion(self):
|
||||
self.tasks.join()
|
||||
|
||||
|
||||
def combine_coverage_files(targeted_packages):
|
||||
# find tox.ini file. tox.ini is used to combine coverage paths to generate formatted report
|
||||
|
|
@ -117,78 +66,13 @@ def collect_tox_coverage_files(targeted_packages):
|
|||
for package_dir in [package for package in targeted_packages]:
|
||||
coverage_file = os.path.join(package_dir, ".coverage")
|
||||
if os.path.isfile(coverage_file):
|
||||
destination_file = os.path.join(
|
||||
root_coverage_dir, ".coverage_{}".format(os.path.basename(package_dir))
|
||||
)
|
||||
destination_file = os.path.join(root_coverage_dir, ".coverage_{}".format(os.path.basename(package_dir)))
|
||||
shutil.copyfile(coverage_file, destination_file)
|
||||
coverage_files.append(destination_file)
|
||||
|
||||
logging.info("Uploading .coverage files: {}".format(coverage_files))
|
||||
|
||||
|
||||
|
||||
def individual_workload(tox_command_tuple, workload_results):
|
||||
pkg = os.path.basename(tox_command_tuple[1])
|
||||
stdout = os.path.join(tox_command_tuple[1], "stdout.txt")
|
||||
stderr = os.path.join(tox_command_tuple[1], "stderr.txt")
|
||||
tox_dir = os.path.join(tox_command_tuple[1], "./.tox/")
|
||||
|
||||
with open(stdout, "w") as f_stdout, open(stderr, "w") as f_stderr:
|
||||
proc = Popen(
|
||||
tox_command_tuple[0],
|
||||
stdout=f_stdout,
|
||||
stderr=f_stderr,
|
||||
cwd=tox_command_tuple[1],
|
||||
env=os.environ.copy(),
|
||||
)
|
||||
|
||||
logging.info("POpened task for for {}".format(pkg))
|
||||
proc.wait()
|
||||
|
||||
return_code = proc.returncode
|
||||
|
||||
if proc.returncode != 0:
|
||||
logging.error("{} returned with code {}".format(pkg, proc.returncode))
|
||||
else:
|
||||
logging.info(
|
||||
"{} returned with code 0, output will be printed after the test run completes.".format(
|
||||
pkg
|
||||
)
|
||||
)
|
||||
|
||||
if read_file(stderr):
|
||||
logging.error("Package {} had stderror output. Logging.".format(pkg))
|
||||
return_code = "StdErr output detected"
|
||||
|
||||
workload_results[tox_command_tuple[1]] = (return_code, stdout, stderr)
|
||||
|
||||
if in_ci():
|
||||
shutil.rmtree(tox_dir)
|
||||
|
||||
def execute_tox_parallel(tox_command_tuples):
|
||||
pool = ThreadPool(pool_size)
|
||||
workload_results = {}
|
||||
run_result = 0
|
||||
|
||||
for index, cmd_tuple in enumerate(tox_command_tuples):
|
||||
pool.add_task(individual_workload, cmd_tuple, workload_results)
|
||||
|
||||
pool.wait_completion()
|
||||
|
||||
for key in workload_results.keys():
|
||||
log_file(workload_results[key][1])
|
||||
|
||||
if workload_results[key][0] != 0:
|
||||
logging.error(
|
||||
"{} tox invocation exited with returncode {}".format(
|
||||
os.path.basename(key), workload_results[key][0]
|
||||
)
|
||||
)
|
||||
run_result = 1
|
||||
|
||||
return run_result
|
||||
|
||||
|
||||
def compare_req_to_injected_reqs(parsed_req, injected_packages):
|
||||
if parsed_req is None:
|
||||
return False
|
||||
|
|
@ -201,9 +85,7 @@ def inject_custom_reqs(file, injected_packages, package_dir):
|
|||
injected_packages = [p for p in re.split("[\s,]", injected_packages) if p]
|
||||
|
||||
if injected_packages:
|
||||
logging.info(
|
||||
"Adding custom packages to requirements for {}".format(package_dir)
|
||||
)
|
||||
logging.info("Adding custom packages to requirements for {}".format(package_dir))
|
||||
with open(file, "r") as f:
|
||||
for line in f:
|
||||
logging.info("Attempting to parse {}".format(line))
|
||||
|
|
@ -218,10 +100,7 @@ def inject_custom_reqs(file, injected_packages, package_dir):
|
|||
all_adjustments = injected_packages + [
|
||||
line_tuple[0].strip()
|
||||
for line_tuple in req_lines
|
||||
if line_tuple[0].strip()
|
||||
and not compare_req_to_injected_reqs(
|
||||
line_tuple[1][0], injected_packages
|
||||
)
|
||||
if line_tuple[0].strip() and not compare_req_to_injected_reqs(line_tuple[1][0], injected_packages)
|
||||
]
|
||||
else:
|
||||
all_adjustments = injected_packages
|
||||
|
|
@ -246,7 +125,7 @@ def build_whl_for_req(req, package_path):
|
|||
parsed = ParsedSetup.from_path(req_pkg_path)
|
||||
|
||||
logging.info("Building wheel for package {}".format(parsed.name))
|
||||
create_package(req_pkg_path, temp_dir, enable_sdist = False)
|
||||
create_package(req_pkg_path, temp_dir, enable_sdist=False)
|
||||
|
||||
whl_path = os.path.join(temp_dir, find_whl(parsed.name, parsed.version, temp_dir))
|
||||
logging.info("Wheel for package {0} is {1}".format(parsed.name, whl_path))
|
||||
|
|
@ -261,16 +140,12 @@ def replace_dev_reqs(file, pkg_root):
|
|||
|
||||
with open(file, "r") as f:
|
||||
for line in f:
|
||||
args = [
|
||||
part.strip()
|
||||
for part in line.split()
|
||||
if part and not part.strip() == "-e"
|
||||
]
|
||||
args = [part.strip() for part in line.split() if part and not part.strip() == "-e"]
|
||||
amended_line = " ".join(args)
|
||||
|
||||
if amended_line.endswith("]"):
|
||||
trim_amount = amended_line[::-1].index("[") + 1
|
||||
amended_line = amended_line[0:(len(amended_line) - trim_amount)]
|
||||
amended_line = amended_line[0 : (len(amended_line) - trim_amount)]
|
||||
|
||||
adjusted_req_lines.append(amended_line)
|
||||
|
||||
|
|
@ -289,12 +164,10 @@ def replace_dev_reqs(file, pkg_root):
|
|||
|
||||
def collect_log_files(working_dir):
|
||||
logging.info("Collecting log files from {}".format(working_dir))
|
||||
package = working_dir.split('/')[-1]
|
||||
package = working_dir.split("/")[-1]
|
||||
# collect all the log files into one place for publishing in case of tox failure
|
||||
|
||||
log_directory = os.path.join(
|
||||
root_dir, "_tox_logs"
|
||||
)
|
||||
log_directory = os.path.join(root_dir, "_tox_logs")
|
||||
|
||||
try:
|
||||
os.mkdir(log_directory)
|
||||
|
|
@ -302,9 +175,7 @@ def collect_log_files(working_dir):
|
|||
except OSError:
|
||||
logging.info("'{}' directory already exists".format(log_directory))
|
||||
|
||||
log_directory = os.path.join(
|
||||
log_directory, package
|
||||
)
|
||||
log_directory = os.path.join(log_directory, package)
|
||||
|
||||
try:
|
||||
os.mkdir(log_directory)
|
||||
|
|
@ -312,9 +183,7 @@ def collect_log_files(working_dir):
|
|||
except OSError:
|
||||
logging.info("'{}' directory already exists".format(log_directory))
|
||||
|
||||
log_directory = os.path.join(
|
||||
log_directory, sys.version.split()[0]
|
||||
)
|
||||
log_directory = os.path.join(log_directory, sys.version.split()[0])
|
||||
|
||||
try:
|
||||
os.mkdir(log_directory)
|
||||
|
|
@ -344,10 +213,7 @@ def collect_log_files(working_dir):
|
|||
logging.info("LOG FILE: {}".format(filename))
|
||||
|
||||
file_location = os.path.join(log_files, filename)
|
||||
shutil.move(
|
||||
file_location,
|
||||
os.path.join(temp_dir, filename)
|
||||
)
|
||||
shutil.move(file_location, os.path.join(temp_dir, filename))
|
||||
logging.info("Moved file to {}".format(os.path.join(temp_dir, filename)))
|
||||
else:
|
||||
logging.info("Could not find {} directory".format(log_files))
|
||||
|
|
@ -355,6 +221,7 @@ def collect_log_files(working_dir):
|
|||
for f in glob.glob(os.path.join(root_dir, "_tox_logs", "*")):
|
||||
logging.info("Log file: {}".format(f))
|
||||
|
||||
|
||||
def execute_tox_serial(tox_command_tuples):
|
||||
return_code = 0
|
||||
|
||||
|
|
@ -364,9 +231,7 @@ def execute_tox_serial(tox_command_tuples):
|
|||
logging.info("tox_dir: {}".format(tox_dir))
|
||||
|
||||
logging.info(
|
||||
"Running tox for {}. {} of {}.".format(
|
||||
os.path.basename(cmd_tuple[1]), index + 1, len(tox_command_tuples)
|
||||
)
|
||||
"Running tox for {}. {} of {}.".format(os.path.basename(cmd_tuple[1]), index + 1, len(tox_command_tuples))
|
||||
)
|
||||
|
||||
result = run_check_call(cmd_tuple[0], cmd_tuple[1], always_exit=False)
|
||||
|
|
@ -391,7 +256,17 @@ def execute_tox_serial(tox_command_tuples):
|
|||
return return_code
|
||||
|
||||
|
||||
def prep_and_run_tox(targeted_packages, parsed_args, options_array=[]):
|
||||
def prep_and_run_tox(targeted_packages: List[str], parsed_args: Namespace, options_array: List[str] = []) -> None:
|
||||
"""
|
||||
Primary entry point for tox invocations during CI runs.
|
||||
|
||||
:param targeted_packages: The set of targeted packages. These are not just package names, and are instead the full absolute path to the package root directory.
|
||||
:param parsed_args: An argparse namespace object from setup_execute_tests.py. Not including it will effectively disable "customizations"
|
||||
of the tox invocation.
|
||||
:param options_array: When invoking tox, these additional options will be passed to the underlying tox invocations as arguments.
|
||||
When invoking of "tox -e whl -c ../../../eng/tox/tox.ini -- --suppress-no-test-exit-code", "--suppress-no-test-exit-code" the "--" will be
|
||||
passed directly to the pytest invocation.
|
||||
"""
|
||||
if parsed_args.wheel_dir:
|
||||
os.environ["PREBUILT_WHEEL_DIR"] = parsed_args.wheel_dir
|
||||
|
||||
|
|
@ -424,8 +299,7 @@ def prep_and_run_tox(targeted_packages, parsed_args, options_array=[]):
|
|||
|
||||
# if not present, re-use base
|
||||
if not os.path.exists(destination_tox_ini) or (
|
||||
os.path.exists(destination_tox_ini)
|
||||
and os.path.basename(package_dir) in IGNORED_TOX_INIS
|
||||
os.path.exists(destination_tox_ini) and os.path.basename(package_dir) in IGNORED_TOX_INIS
|
||||
):
|
||||
logging.info(
|
||||
"No customized tox.ini present, using common eng/tox/tox.ini for {}".format(
|
||||
|
|
@ -446,12 +320,19 @@ def prep_and_run_tox(targeted_packages, parsed_args, options_array=[]):
|
|||
replace_dev_reqs(dependency_tools_path, package_dir)
|
||||
os.environ["TOX_PARALLEL_NO_SPINNER"] = "1"
|
||||
|
||||
inject_custom_reqs(
|
||||
destination_dev_req, parsed_args.injected_packages, package_dir
|
||||
)
|
||||
inject_custom_reqs(destination_dev_req, parsed_args.injected_packages, package_dir)
|
||||
|
||||
if parsed_args.tox_env:
|
||||
tox_execution_array.extend(["-e", parsed_args.tox_env])
|
||||
filtered_tox_environment_set = filter_tox_environment_string(parsed_args.tox_env, package_name)
|
||||
|
||||
if not filtered_tox_environment_set:
|
||||
logging.info(
|
||||
f"All requested tox environments for package {package_name} have been excluded by the environment exclusion list."
|
||||
+ " Check file /tools/azure-sdk-tools/ci_tools/environment_exclusions.py"
|
||||
)
|
||||
continue
|
||||
|
||||
tox_execution_array.extend(["-e", filtered_tox_environment_set])
|
||||
|
||||
if parsed_args.tenvparallel:
|
||||
tox_execution_array.extend(["-p", "all"])
|
||||
|
|
@ -466,12 +347,9 @@ def prep_and_run_tox(targeted_packages, parsed_args, options_array=[]):
|
|||
|
||||
tox_command_tuples.append((tox_execution_array, package_dir))
|
||||
|
||||
if parsed_args.tparallel:
|
||||
return_code = execute_tox_parallel(tox_command_tuples)
|
||||
else:
|
||||
return_code = execute_tox_serial(tox_command_tuples)
|
||||
return_code = execute_tox_serial(tox_command_tuples)
|
||||
|
||||
if not parsed_args.disablecov:
|
||||
collect_tox_coverage_files(targeted_packages)
|
||||
|
||||
sys.exit(return_code)
|
||||
sys.exit(return_code)
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ Check that you are running pylint version >=2.5.2 and astroid version >=2.4.1.
|
|||
```
|
||||
4. Run pylint at the package level using tox and it will find the pylintrc file:
|
||||
```bash
|
||||
C:\azure-sdk-for-python\sdk\storage\azure-storage-blob>tox -c ../../../eng/tox/tox.ini -e lint
|
||||
C:\azure-sdk-for-python\sdk\storage\azure-storage-blob>tox -c ../../../eng/tox/tox.ini -e pylint
|
||||
```
|
||||
5. If you use the pylint extension for VS code or Pycharm it *should* find the pylintrc automatically.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ tox -c eng/tox/tox.ini
|
|||
To run a specific tox command from your directory use the following commands:
|
||||
```bash
|
||||
> tox -c ../../../eng/tox/tox.ini -e sphinx
|
||||
> tox -c ../../../eng/tox/tox.ini -e lint
|
||||
> tox -c ../../../eng/tox/tox.ini -e pylint
|
||||
> tox -c ../../../eng/tox/tox.ini -e mypy
|
||||
> tox -c ../../../eng/tox/tox.ini -e whl
|
||||
> tox -c ../../../eng/tox/tox.ini -e sdist
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ tox -c eng/tox/tox.ini
|
|||
To run a specific tox command from your directory use the following commands:
|
||||
```bash
|
||||
> tox -c ../../../eng/tox/tox.ini -e sphinx
|
||||
> tox -c ../../../eng/tox/tox.ini -e lint
|
||||
> tox -c ../../../eng/tox/tox.ini -e pylint
|
||||
> tox -c ../../../eng/tox/tox.ini -e mypy
|
||||
> tox -c ../../../eng/tox/tox.ini -e whl
|
||||
> tox -c ../../../eng/tox/tox.ini -e sdist
|
||||
|
|
|
|||
|
|
@ -0,0 +1,410 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
import logging
|
||||
|
||||
PYLINT_OPT_OUT = [
|
||||
"azure-applicationinsights",
|
||||
"azure-batch",
|
||||
"azure-cognitiveservices-anomalydetector",
|
||||
"azure-cognitiveservices-formrecognizer",
|
||||
"azure-cognitiveservices-knowledge-nspkg",
|
||||
"azure-cognitiveservices-knowledge-qnamaker",
|
||||
"azure-cognitiveservices-language-luis",
|
||||
"azure-cognitiveservices-language-nspkg",
|
||||
"azure-cognitiveservices-language-spellcheck",
|
||||
"azure-cognitiveservices-language-textanalytics",
|
||||
"azure-cognitiveservices-nspkg",
|
||||
"azure-cognitiveservices-personalizer",
|
||||
"azure-cognitiveservices-search-autosuggest",
|
||||
"azure-cognitiveservices-search-customimagesearch",
|
||||
"azure-cognitiveservices-search-customsearch",
|
||||
"azure-cognitiveservices-search-entitysearch",
|
||||
"azure-cognitiveservices-search-imagesearch",
|
||||
"azure-cognitiveservices-search-newssearch",
|
||||
"azure-cognitiveservices-search-nspkg",
|
||||
"azure-cognitiveservices-search-videosearch",
|
||||
"azure-cognitiveservices-search-visualsearch",
|
||||
"azure-cognitiveservices-search-websearch",
|
||||
"azure-cognitiveservices-vision-computervision",
|
||||
"azure-cognitiveservices-vision-contentmoderator",
|
||||
"azure-cognitiveservices-vision-customvision",
|
||||
"azure-cognitiveservices-vision-face",
|
||||
"azure-cognitiveservices-vision-nspkg",
|
||||
"azure-common",
|
||||
"azure-nspkg",
|
||||
"azure-servicemanagement-legacy",
|
||||
"azure-graphrbac",
|
||||
"azure-loganalytics",
|
||||
"azure-servicefabric",
|
||||
"azure-template",
|
||||
"azure-keyvault",
|
||||
"azure-synapse",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-spark",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-nspkg",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-security-attestation",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-purview-nspkg",
|
||||
"azure-purview-scanning",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-account",
|
||||
"azure-purview-administration",
|
||||
"azure-messaging-nspkg",
|
||||
"azure-agrifood-farming",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-developer-devcenter",
|
||||
]
|
||||
|
||||
# omit package from running mypy checks
|
||||
MYPY_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
# omit package from running pyright checks
|
||||
PYRIGHT_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-identity",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-ai-metricsadvisor",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-ai-personalizer",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-data-tables",
|
||||
"azure-ai-textanalytics",
|
||||
"azure-ai-translation-document",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
# omit package from running verifytypes checks
|
||||
VERIFYTYPES_OPT_OUT = [
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-identity",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-ai-metricsadvisor",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-ai-personalizer",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-data-tables",
|
||||
"azure-messaging-webpubsubservice",
|
||||
"azure-ai-textanalytics",
|
||||
]
|
||||
|
||||
# omit package from running type checkers on samples
|
||||
# note: if removed from this list, you must enable one or both of mypy or pyright checks.
|
||||
TYPE_CHECK_SAMPLES_OPT_OUT = [
|
||||
"azure-ai-metricsadvisor",
|
||||
"azure-agrifood-farming",
|
||||
"azure-ai-anomalydetector",
|
||||
"azure-appconfiguration",
|
||||
"azure-appconfiguration-provider",
|
||||
"azure-security-attestation",
|
||||
"azure-batch",
|
||||
"azure-ai-language-conversations",
|
||||
"azure-ai-language-questionanswering",
|
||||
"azure-communication-chat",
|
||||
"azure-communication-email",
|
||||
"azure-communication-identity",
|
||||
"azure-communication-jobrouter",
|
||||
"azure-communication-networktraversal",
|
||||
"azure-communication-phonenumbers",
|
||||
"azure-communication-rooms",
|
||||
"azure-communication-sms",
|
||||
"azure-confidentialledger",
|
||||
"azure-containerregistry",
|
||||
"azure-core",
|
||||
"azure-mgmt-core",
|
||||
"azure-core-experimental",
|
||||
"azure-core-tracing-opencensus",
|
||||
"azure-core-tracing-opentelemetry",
|
||||
"azure-cosmos",
|
||||
"azure-developer-devcenter",
|
||||
"azure-iot-deviceupdate",
|
||||
"azure-digitaltwins-core",
|
||||
"azure-eventgrid",
|
||||
"azure-eventhub",
|
||||
"azure-eventhub-checkpointstoreblob",
|
||||
"azure-eventhub-checkpointstoreblob-aio",
|
||||
"azure-eventhub-checkpointstoretable",
|
||||
"azure-ai-formrecognizer",
|
||||
"azure-keyvault-administration",
|
||||
"azure-keyvault-certificates",
|
||||
"azure-keyvault-keys",
|
||||
"azure-keyvault-secrets",
|
||||
"azure-developer-loadtesting",
|
||||
"azure-maps-geolocation",
|
||||
"azure-maps-render",
|
||||
"azure-maps-route",
|
||||
"azure-maps-search",
|
||||
"azure-mixedreality-authentication",
|
||||
"azure-ai-ml",
|
||||
"azure-iot-modelsrepository",
|
||||
"azure-monitor-ingestion",
|
||||
"azure-monitor-opentelemetry-exporter",
|
||||
"azure-monitor-query",
|
||||
"azure-purview-administration",
|
||||
"azure-purview-catalog",
|
||||
"azure-purview-scanning",
|
||||
"azure-mixedreality-remoterendering",
|
||||
"azure-schemaregistry",
|
||||
"azure-schemaregistry-avroencoder",
|
||||
"azure-search-documents",
|
||||
"azure-servicebus",
|
||||
"azure-storage-blob",
|
||||
"azure-storage-blob-changefeed",
|
||||
"azure-storage-file-datalake",
|
||||
"azure-storage-file-share",
|
||||
"azure-storage-queue",
|
||||
"azure-synapse-accesscontrol",
|
||||
"azure-synapse-artifacts",
|
||||
"azure-synapse-managedprivateendpoints",
|
||||
"azure-synapse-monitoring",
|
||||
"azure-synapse-spark",
|
||||
"azure-ai-translation-document",
|
||||
"azure-messaging-webpubsubservice",
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------------------------
|
||||
# DO NOT add packages to the below lists. They are used to omit packages that will never run type checking.
|
||||
IGNORE_FILTER = ["nspkg", "mgmt", "cognitiveservices"]
|
||||
FILTER_EXCLUSIONS = ["azure-mgmt-core"]
|
||||
IGNORE_PACKAGES = [
|
||||
"azure-applicationinsights",
|
||||
"azure-servicemanagement-legacy",
|
||||
"azure",
|
||||
"azure-storage",
|
||||
"azure-monitor",
|
||||
"azure-servicefabric",
|
||||
"azure-keyvault",
|
||||
"azure-synapse",
|
||||
"azure-common",
|
||||
"conda-recipe",
|
||||
"azure-graphrbac",
|
||||
"azure-loganalytics",
|
||||
"azure-media-analytics-edge",
|
||||
"azure-media-videoanalyzer-edge",
|
||||
"azure-template",
|
||||
]
|
||||
|
||||
|
||||
def filter_tox_environment_string(namespace_argument: str, package_name: str) -> str:
|
||||
"""
|
||||
Takes an incoming comma separated list of tox environments and package name. Resolves whether or not
|
||||
each given tox environment should run, given comparison to single unified exclusion file in `environment_exclusions`.
|
||||
|
||||
:param namespace_argument: A namespace argument.
|
||||
:param package_name: The name of the package. This takes the form of a comma separated list: "whl,sdist,mindependency". "whl". "lint,pyright,sphinx".
|
||||
"""
|
||||
if namespace_argument:
|
||||
tox_envs = namespace_argument.strip().split(",")
|
||||
filtered_set = []
|
||||
|
||||
for tox_env in tox_envs:
|
||||
exclusions_for_env = []
|
||||
try:
|
||||
exclusions_for_env = globals()[f"{tox_env.strip().upper()}_OPT_OUT"]
|
||||
except Exception as e:
|
||||
pass
|
||||
|
||||
if exclusions_for_env:
|
||||
if package_name in exclusions_for_env:
|
||||
continue
|
||||
|
||||
filtered_set.append(tox_env)
|
||||
return ",".join(filtered_set)
|
||||
|
||||
return namespace_argument
|
||||
|
||||
|
||||
def is_ignored_package(package_name: str) -> bool:
|
||||
"""
|
||||
Evaluates a package name and evaluates whether or not tox environments should run against it.
|
||||
"""
|
||||
if package_name in IGNORE_PACKAGES:
|
||||
return True
|
||||
if package_name not in FILTER_EXCLUSIONS and any([identifier in package_name for identifier in IGNORE_FILTER]):
|
||||
return True
|
||||
return False
|
||||
Загрузка…
Ссылка в новой задаче