Clean up advanced testing guide (#36818)
This commit is contained in:
McCoy Patiño
GitHub
Родитель
715f4179c8
Коммит
d718fc4cc5
|
|
@ -1,25 +1,24 @@
|
|||
# Setup Python Development Environment - Advanced
|
||||
In this document we will provide additional information about the test environments:
|
||||
# Python SDK advanced testing guide
|
||||
This guide covers advanced testing scenarios for Azure SDK for Python libraries.
|
||||
|
||||
- [Setup Python Development Environment - Advanced](#setup-python-development-environment---advanced)
|
||||
- [Test Mixin Classes](#test-mixin-classes)
|
||||
- [Preparers](#preparers)
|
||||
- [Examples with Preparers](#examples-with-preparers)
|
||||
- [Example 2: Basic Preparer Usage with Storage](#example-2-basic-preparer-usage-with-storage)
|
||||
- [Example 3: Cached Preparer Usage](#example-3-cached-preparer-usage)
|
||||
- [mgmt\_settings\_real file](#mgmt_settings_real-file)
|
||||
## Table of contents
|
||||
|
||||
## Test Mixin Classes
|
||||
Many of our test suites use a mixin class to reduce re-writing code in multiple test files. For example, in the Tables test suite there is a `_shared` directory containing two of these mixin classes, a [sync one](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/testcase.py) and an [async version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/asynctestcase.py). These classes will often have ways to create connection strings from an account name and key, formulate the account url, configure logging, or validate service responses. In order for these mixin classes to be used by both the functional and unit tests they should inherit from `object`. For example:
|
||||
- [Mixin classes](#test-mixin-classes)
|
||||
- [Pre-test setup](#pre-test-setup)
|
||||
- [xunit-style setup](#xunit-style-setup)
|
||||
- [Fixture setup](#fixture-setup)
|
||||
|
||||
## Mixin classes
|
||||
Many of our test suites use a base/mixin class to consolidate shared test logic. Mixin classes can define instance attributes to handle environment variables, make complex assertions, and more. By inheriting from these mixins, test classes can then share this logic throughout multiple files.
|
||||
|
||||
For example, in the Tables test suite there is a `_shared` directory containing two of these mixin classes: a [sync version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/testcase.py) and an [async version](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/_shared/asynctestcase.py).
|
||||
|
||||
```python
|
||||
|
||||
class TablesTestMixin(object):
|
||||
def connection_string(self, account, key):
|
||||
return "DefaultEndpointsProtocol=https;AccountName=" + account + ";AccountKey=" + str(key) + ";EndpointSuffix=core.windows.net"
|
||||
class TableTestCase(object):
|
||||
|
||||
def account_url(self, account, endpoint_type):
|
||||
"""Return an url of storage account.
|
||||
|
||||
:param str storage_account: Storage account name
|
||||
:param str storage_type: The Storage type part of the URL. Should be "table", or "cosmos", etc.
|
||||
"""
|
||||
|
|
@ -27,44 +26,49 @@ class TablesTestMixin(object):
|
|||
if endpoint_type == "table":
|
||||
return account.primary_endpoints.table.rstrip("/")
|
||||
if endpoint_type == "cosmos":
|
||||
return "https://{}.table.cosmos.azure.com".format(account.name)
|
||||
else:
|
||||
raise ValueError("Unknown storage type {}".format(storage_type))
|
||||
except AttributeError: # Didn't find "primary_endpoints"
|
||||
cosmos_suffix = os.getenv("TABLES_COSMOS_ENDPOINT_SUFFIX", DEFAULT_COSMOS_ENDPOINT_SUFFIX)
|
||||
return f"https://{account.name}.table.{cosmos_suffix}"
|
||||
except AttributeError: # Didn't find "account.primary_endpoints"
|
||||
if endpoint_type == "table":
|
||||
return 'https://{}.{}.core.windows.net'.format(account, endpoint_type)
|
||||
storage_suffix = os.getenv("TABLES_STORAGE_ENDPOINT_SUFFIX", DEFAULT_STORAGE_ENDPOINT_SUFFIX)
|
||||
return f"https://{account}.table.{storage_suffix}"
|
||||
if endpoint_type == "cosmos":
|
||||
return "https://{}.table.cosmos.azure.com".format(account)
|
||||
cosmos_suffix = os.getenv("TABLES_COSMOS_ENDPOINT_SUFFIX", DEFAULT_COSMOS_ENDPOINT_SUFFIX)
|
||||
return f"https://{account}.table.{cosmos_suffix}"
|
||||
|
||||
def enable_logging(self):
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
|
||||
self.logger.handlers = [handler]
|
||||
self.logger.setLevel(logging.INFO)
|
||||
self.logger.propagate = True
|
||||
self.logger.disabled = False
|
||||
...
|
||||
|
||||
def _assert_delete_retention_policy_equal(self, policy1, policy2):
|
||||
"""Assert that two deletion retention policies are equal."""
|
||||
if policy1 is None or policy2 is None:
|
||||
assert policy1 == policy2
|
||||
return
|
||||
|
||||
assert policy1.enabled == policy2.enabled
|
||||
assert policy1.days == policy2.days
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
In action this class can be used in functional tests:
|
||||
|
||||
```python
|
||||
class TestTablesFunctional(AzureTestCase, TablesTestMixin):
|
||||
...
|
||||
def test_with_mixin(self, account, key):
|
||||
conn_str = self.connection_string(account, key)
|
||||
client = TableClient.from_connection_string(conn_str)
|
||||
client.create_table('first')
|
||||
client.create_table('second')
|
||||
tables = 0
|
||||
for table in client.list_tables():
|
||||
tables += 1
|
||||
|
||||
assert tables == 2
|
||||
class TestTable(AzureRecordedTestCase, TableTestCase):
|
||||
@tables_decorator
|
||||
@recorded_by_proxy
|
||||
def test_create_properties(self, tables_storage_account_name, tables_primary_storage_account_key):
|
||||
# # Arrange
|
||||
account_url = self.account_url(tables_storage_account_name, "table")
|
||||
ts = TableServiceClient(credential=tables_primary_storage_account_key, endpoint=account_url)
|
||||
table_name = self._get_table_reference()
|
||||
# Act
|
||||
created = ts.create_table(table_name)
|
||||
...
|
||||
```
|
||||
|
||||
Or can be used in a unit test:
|
||||
```python
|
||||
class TestTablesUnit(TablesTestMixin):
|
||||
class TestTablesUnit(TableTestCase):
|
||||
...
|
||||
def test_valid_url(self):
|
||||
account = "fake_tables_account"
|
||||
|
|
@ -74,114 +78,77 @@ class TestTablesUnit(TablesTestMixin):
|
|||
client = TableClient(account_url=url, credential=credential)
|
||||
|
||||
assert client is not None
|
||||
assert client.account_url == "https://{}.tables.core.windows.net/".format(account)
|
||||
assert client.account_url == f"https://{account}.tables.core.windows.net/"
|
||||
```
|
||||
|
||||
## Preparers
|
||||
## Pre-test setup
|
||||
Tests will often use shared resources that make sense to set up before tests execute. There are two recommended
|
||||
approaches for this kind of setup, with each having benefits and drawbacks.
|
||||
|
||||
The Azure SDK team has created some in house tools to help with easier testing. These additional tools are located in the `devtools_testutils` package that was installed with your `dev_requirements.txt`. In this package are the preparers that will be commonly used throughout the repository to test various resources. A preparer is a way to programmatically create fresh resources to run our tests against and then deleting them after running a test suite. These help guarantee standardized behavior by starting each test group from a fresh resource and account.
|
||||
|
||||
If this situation is a requirement for your tests, you can opt to create a new preparer for your service from the management plane library for a service. There are already a few preparers built in the [devtools_testutils](https://github.com/Azure/azure-sdk-for-python/tree/main/tools/azure-sdk-tools/devtools_testutils). Most prepares will start with the [`ResourceGroupPreparer`](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L29-L99) to first create a resource group for your service.
|
||||
|
||||
To build your own preparer you will need to use the management plane library to create a service and pass the credentials you need into your tests. The two important methods for a preparer are the `create_resource` and `remove_resource` methods. In the `create_resource` method you will use the management client to create the resource and return a dictionary of key-value pairs. The keys will be matched with the test method parameters and passed in as positional arguments to the test. The `remove_resource` method will clean up and remove the resource to prevent a backlog of unused resources in your subscription. For examples of each of these methods, check out these examples:
|
||||
|
||||
| Preparer | `create_resource` | `remove_resource` |
|
||||
|-|-|-|
|
||||
| Resource Group | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L57-L85) | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py#L87-L99) |
|
||||
| Storage Account | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L53-L102) | [link](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L104-L107) |
|
||||
|
||||
## Examples with Preparers
|
||||
|
||||
### Example 2: Basic Preparer Usage with Storage
|
||||
### xunit-style setup
|
||||
Pytest has documentation describing this setup style: https://docs.pytest.org/en/latest/how-to/xunit_setup.html. For
|
||||
example:
|
||||
|
||||
```python
|
||||
import os
|
||||
import pytest
|
||||
from devtools_testutils.azure_recorded_testcase import get_credential
|
||||
|
||||
from azure.data.tables import TableServiceClient
|
||||
from devtools_testutils import (
|
||||
AzureTestCase,
|
||||
ResourceGroupPreparer,
|
||||
StorageAccountPreparer
|
||||
)
|
||||
class TestService(AzureRecordedTestCase):
|
||||
def setup_method(self, method):
|
||||
"""This method is called before each test in the class executes."""
|
||||
credential = self.get_credential(ServiceClient) # utility from parent class
|
||||
self.client = ServiceClient("...", credential)
|
||||
|
||||
class ExampleStorageTestCase(AzureTestCase):
|
||||
|
||||
@ResourceGroupPreparer()
|
||||
@StorageAccountPreparer()
|
||||
def test_create_table(self, resource_group, location, storage_account, storage_account_key):
|
||||
account_url = self.account_url(storage_account, "table")
|
||||
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
|
||||
|
||||
valid_table_name = "validtablename"
|
||||
table = client.create_table(valid_table_name)
|
||||
|
||||
assert valid_table_name == table.table_name
|
||||
@classmethod
|
||||
def setup_class(cls):
|
||||
"""This method is called only once, before any tests execute."""
|
||||
credential = get_credential() # only module-level and classmethod utilities are available
|
||||
cls.client = ServiceClient("...", credential)
|
||||
```
|
||||
|
||||
This test uses preparers to create resources, then creates a table, and finally verifies the name is correct.
|
||||
The primary benefit of using `setup_method` is retaining access to the utilities provided your test class. You could
|
||||
use `self.get_credential`, for example, to pick up our core utility for selecting a client credential based on your
|
||||
environment. A drawback is that `setup_method` runs before each test method in the class, so your setup needs to be
|
||||
idempotent to avoid issues caused by repeated invocations.
|
||||
|
||||
Notes:
|
||||
1. This test is aiming to create a new Table, which requires a storage account, which in hand requires a resource group. The first decorator (`@ResourceGroupPreparer()`) creates a new resource group, and passes the parameters of this resource group into the `@StorageAccountPreparer()` which creates the storage account. The parameters from the storage account creation is passed into the signature of `test_create_table` .
|
||||
2. The `create_client_from_credential` is used again but this time with `storage_account_key` instead of getting a credential from the `self.get_credential` method showed in the previous section. The storage account preparer returns the key for the account which is a valid credential.
|
||||
Alternatively, the class-level `setup_class` method runs once before all tests, but doesn't give you access to all
|
||||
instance attributes on the class. You can still set attributes on the test class to reference from tests, and
|
||||
module-level utilities can be used in place of instance attributes, as shown in the example above.
|
||||
|
||||
### Fixture setup
|
||||
Pytest has documentation explaining how to implement and use fixtures:
|
||||
https://docs.pytest.org/en/latest/how-to/fixtures.html. For example, in a library's `conftest.py`:
|
||||
|
||||
### Example 3: Cached Preparer Usage
|
||||
```python
|
||||
import os
|
||||
import pytest
|
||||
from devtools_testutils.azure_recorded_testcase import get_credential
|
||||
|
||||
from azure.core.exceptions import ResourceExistsError
|
||||
from azure.data.tables import TableServiceClient
|
||||
from devtools_testutils import (
|
||||
AzureTestCase,
|
||||
CachedResourceGroupPreparer,
|
||||
CachedStorageAccountPreparer
|
||||
)
|
||||
|
||||
class ExampleStorageTestCase(AzureTestCase):
|
||||
|
||||
@CachedResourceGroupPreparer(name_prefix="storagetest")
|
||||
@CachedStorageAccountPreparer(name_prefix="storagetest")
|
||||
def test_create_table(self, resource_group, location, storage_account, storage_account_key):
|
||||
account_url = self.account_url(storage_account, "table")
|
||||
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
|
||||
|
||||
valid_table_name = "validtablename"
|
||||
table = client.create_table(valid_table_name)
|
||||
|
||||
assert valid_table_name == table.table_name
|
||||
|
||||
@CachedResourceGroupPreparer(name_prefix="storagetest")
|
||||
@CachedStorageAccountPreparer(name_prefix="storagetest")
|
||||
def test_create_table_if_exists (self, resource_group, location, storage_account, storage_account_key):
|
||||
account_url = self.account_url(storage_account, "table")
|
||||
client = self.create_client_from_credential(TableServiceClient, storage_account_key, account_url=account_url)
|
||||
|
||||
valid_table_name = "validtablename"
|
||||
with pytest.raises(ResourceExistsError):
|
||||
table = client.create_table(valid_table_name)
|
||||
@pytest.fixture(scope="session")
|
||||
def setup_teardown_fixture():
|
||||
# Note that we can't reference AzureRecordedTestCase.get_credential but can use the module-level function
|
||||
client = ServiceClient("...", get_credential())
|
||||
client.set_up_resource()
|
||||
yield # <-- Tests run here, and execution resumes after they finish
|
||||
client.tear_down_resources()
|
||||
```
|
||||
|
||||
The first test is the same as above, the second test tries to create a table that already exists and asserts that the correct type of error is raised in response. These tests use cached preparers unlike the previous example.
|
||||
We can then request the fixture from a test class:
|
||||
|
||||
Notes:
|
||||
1. The cached preparers here will first look to see if an existing resource group or storage account exists with the given parameters, in this case the `name_prefix`. For more information on what parameters differentiate a new resource group or storage account look for the `self.set_cache()` method in the preparer source code [here](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/storage_testcase.py#L49). The advantage to using a cached preparer is the time saver to re-using the same resource instead of creating a new resource for each test. However, this can increase the possibility that you have to be more exact about cleaning up the entities created in between test runs.
|
||||
|
||||
## mgmt_settings_real file
|
||||
|
||||
A `mgmt_settings_real.py` can be used in place of a `.env` file by copying `sdk/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py` to `sdk/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_real.py` and providing real credentials to it. The following changes need to be made to the `mgmt_settings_real.py` file:
|
||||
|
||||
1. Change the value of the `SUBSCRIPTION_ID` variable to your organizations subscription ID, which can be found in the "Overview" section of the "Subscriptions" blade in the [Azure portal](https://portal.azure.com/).
|
||||
2. Define `TENANT_ID`, `CLIENT_ID`, and `CLIENT_SECRET`, which are available after creating a Service Principal or can be retrieved from the Azure Portal after creating a Service Principal. Check out the [Azure docs](https://docs.microsoft.com/cli/azure/ad/sp?view=azure-cli-latest#az_ad_sp_create_for_rbac) to create a Service Principal with a simple one line command to create one. The recommended practice is to include your alias or name in the Service Principal name.
|
||||
3. Change the [`get_azure_core_credentials(**kwargs):`](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py#L39-L53) function in the `mgmt_settings_real.py` file to construct and return a `ClientSecretCredential` object. Pass in the `CLIENT_ID`, `CLIENT_SECRET`, and `TENANT_ID` values to the `ClientSecretCredential` object. This method should look like this:
|
||||
```python
|
||||
def get_azure_core_credentials(**kwargs):
|
||||
from azure.identity import ClientSecretCredential
|
||||
import os
|
||||
return ClientSecretCredential(
|
||||
client_id = CLIENT_ID,
|
||||
client_secret = CLIENT_SECRET,
|
||||
tenant_id = TENANT_ID
|
||||
)
|
||||
```
|
||||
@pytest.mark.usefixtures("setup_teardown_fixture")
|
||||
class TestService(AzureRecordedTestCase):
|
||||
...
|
||||
```
|
||||
|
||||
By requesting a fixture from the test class, the fixture will execute before any tests in the class do. Fixtures are the
|
||||
preferred solution from pytest's perspective and offer a great deal of modular functionality.
|
||||
|
||||
As shown in the example above, the
|
||||
[`yield`](https://docs.pytest.org/latest/how-to/fixtures.html#yield-fixtures-recommended) command will defer to test
|
||||
execution -- after tests finish running, the fixture code after `yield` will execute. This enables the use of a fixture
|
||||
for both setup and teardown.
|
||||
|
||||
However, fixtures in this context have similar drawbacks to the `setup_class` method described in
|
||||
[xunit-style setup](#xunit-style-setup). Since their scope is outside of the test class, test class instance utilities
|
||||
can't be accessed and class state can't be modified.
|
||||
|
||||
By convention, fixtures should be defined in a library's `tests/conftest.py` file. This will provide access to the
|
||||
fixture across test files, and the fixture can be requested without having to manually import it.
|
||||
|
|
|
|||
|
|
@ -340,6 +340,9 @@ AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy`.
|
|||
If you need logging functionality for your testing, pytest also offers [logging][pytest_logging] capabilities either
|
||||
inline through the `caplog` fixture or with command line flags.
|
||||
|
||||
If your tests use shared resources or configuration that needs to be set up at test-running time, you can refer to the
|
||||
[Pre-test setup][setup] section of our advanced testing guide for recommended practices.
|
||||
|
||||
### Configure live or playback testing mode
|
||||
|
||||
"Live" tests refer to tests that make requests to actual Azure resources. "Playback" tests require a recording for each
|
||||
|
|
@ -516,10 +519,13 @@ Some sanitizers IDs that are often opted out of are:
|
|||
|
||||
## Functional vs. unit tests
|
||||
|
||||
The tests written above are functional tests: they generate HTTP traffic and send data to the service. For tests that
|
||||
don't need to make HTTP requests -- i.e. unit tests -- the best practice is to have a separate test class from the one
|
||||
containing functional tests. For example, the `azure-data-tables` package has client-side validation for the table name
|
||||
and properties of the entity; below is an example of how these could be tested:
|
||||
The tests written above are functional tests: they generate HTTP traffic and send data to the service. Client operations
|
||||
that interact with the service should be tested by functional tests wherever possible.
|
||||
|
||||
Tests that don't make HTTP requests -- e.g. tests for model serialization or mocked service interactions for complex
|
||||
scenarios -- can be referred to as "unit tests". For unit tests, the best practice is to have a separate test class
|
||||
from the one containing functional tests. For example, the `azure-data-tables` package has client-side validation for
|
||||
the table name and properties of the entity; below is an example of how these could be tested:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
|
@ -600,6 +606,7 @@ For information about more advanced testing scenarios, refer to the [advanced te
|
|||
[pytest_logging]: https://docs.pytest.org/en/stable/logging.html
|
||||
[python-dotenv_readme]: https://github.com/theskumar/python-dotenv
|
||||
[recording_move]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/recording_migration_guide.md
|
||||
[setup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests-advanced.md#pre-test-setup
|
||||
[test_proxy_sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/57382d5dc00b10a2f9cfd597293eeee0c2dbd8fd/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/SanitizerDictionary.cs#L65
|
||||
[test_proxy_startup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md#start-the-proxy-server
|
||||
[test_resources]: https://github.com/Azure/azure-sdk-for-python/tree/main/eng/common/TestResources#readme
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче