Remove `docker` integration with test framework (#28941)
This commit is contained in:
Родитель
cce052ff73
Коммит
f1a39c98f5
|
@ -53,6 +53,7 @@ _azci_*.log
|
|||
.artifacts
|
||||
.tmp_whl_dir
|
||||
.logs
|
||||
.proxy
|
||||
|
||||
# tox environment folders
|
||||
.tox/
|
||||
|
|
|
@ -392,6 +392,25 @@
|
|||
"whls"
|
||||
],
|
||||
"overrides": [
|
||||
{
|
||||
"filename": "doc/dev/test_proxy_migration_guide.md",
|
||||
"words": [
|
||||
"pytestmarkparametrize"
|
||||
]
|
||||
},
|
||||
{
|
||||
"filename": "tools/azure-sdk-tools/devtools_testutils/proxy_startup.py",
|
||||
"words": [
|
||||
"certifi",
|
||||
"passenv"
|
||||
]
|
||||
},
|
||||
{
|
||||
"filename": "tools/azure-sdk-tools/tests/integration/test_proxy_startup.py",
|
||||
"words": [
|
||||
"spinup"
|
||||
]
|
||||
},
|
||||
{
|
||||
"filename": "sdk/remoterendering/**",
|
||||
"words": [
|
||||
|
|
|
@ -24,7 +24,7 @@ Please refer to the [troubleshooting guide][troubleshooting] if you have any iss
|
|||
- [Advanced details](#advanced-details)
|
||||
- [What does the test proxy do?](#what-does-the-test-proxy-do)
|
||||
- [How does the test proxy know when and what to record or play back?](#how-does-the-test-proxy-know-when-and-what-to-record-or-play-back)
|
||||
- [Start the proxy manually](#start-the-proxy-manually)
|
||||
- [Use pytest.mark.parametrize with migrated tests](#use-pytestmarkparametrize-with-migrated-tests)
|
||||
|
||||
## Update existing tests
|
||||
|
||||
|
@ -96,10 +96,7 @@ Resource preparers need a management client to function, so test classes that us
|
|||
|
||||
### Perform one-time setup
|
||||
|
||||
1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
|
||||
2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
|
||||
3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
|
||||
order to communicate with the test proxy over a secure connection.
|
||||
The test proxy uses a self-signed certificate to communicate with HTTPS. Follow the general setup instructions [here][proxy_cert_docs] to trust this certificate locally.
|
||||
|
||||
### Start the proxy server
|
||||
|
||||
|
@ -120,8 +117,8 @@ def start_proxy(test_proxy):
|
|||
return
|
||||
```
|
||||
|
||||
The `test_proxy` fixture will fetch the test proxy Docker image and create a new container called
|
||||
`ambitious_azsdk_test_proxy`, which will be deleted after test execution unless interrupted.
|
||||
The `test_proxy` fixture will download a test proxy executable if one isn't available locally, start the tool, and stop
|
||||
it after tests complete.
|
||||
|
||||
If your tests already use an `autouse`d, session-level fixture for tests, you can accept the `test_proxy` parameter in
|
||||
that existing fixture instead of adding a new one. For an example, see the [Register sanitizers](#register-sanitizers)
|
||||
|
@ -141,6 +138,9 @@ Recordings for a given package will end up in that package's `/tests/recordings`
|
|||
do. Recordings that use the test proxy are `.json` files instead of `.yml` files, so migrated test suites no longer
|
||||
need old `.yml` recordings.
|
||||
|
||||
After migrating to use the test proxy, libraries can and are encouraged to use out-of-repo recordings. For more
|
||||
information, refer to the [recording migration guide][recording_migration].
|
||||
|
||||
> **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been
|
||||
> deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
|
||||
|
||||
|
@ -153,7 +153,7 @@ Instead, sanitizers (as well as matchers and transforms) can be registered on th
|
|||
`add_general_string_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at
|
||||
[devtools_testutils/sanitizers.py][py_sanitizers].
|
||||
|
||||
Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that
|
||||
Sanitizers, matchers, and transforms remain registered until the proxy tool is stopped, so for any sanitizers that
|
||||
are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to
|
||||
[pytest's scoped fixture documentation][pytest_fixtures] for more details.
|
||||
|
||||
|
@ -403,36 +403,6 @@ Running tests in playback follows the same pattern, except that requests will be
|
|||
The `recorded_by_proxy` and `recorded_by_proxy_async` decorators send the appropriate requests at the start and end of
|
||||
each test case.
|
||||
|
||||
### Start the proxy manually
|
||||
|
||||
There are two options for manually starting and stopping the test proxy: one uses a PowerShell command, and one uses
|
||||
methods from `devtools_testutils`.
|
||||
|
||||
#### PowerShell
|
||||
|
||||
There is a [PowerShell script][docker_start_proxy] in `eng/common/testproxy` that will fetch the proxy Docker image if
|
||||
you don't already have it, and will start or stop a container running the image for you. You can run the following
|
||||
command from the root of the `azure-sdk-for-python` directory to start the container whenever you want to make the test
|
||||
proxy available for running tests:
|
||||
|
||||
```powershell
|
||||
.\eng\common\testproxy\docker-start-proxy.ps1 "start"
|
||||
```
|
||||
|
||||
Note that the proxy is available as long as the container is running. In other words, you don't need to start and
|
||||
stop the container for each test run or between tests for different SDKs. You can run the above command in the morning
|
||||
and just stop the container whenever you'd like. To stop the container, run the same command but with `"stop"` in place
|
||||
of `"start"`.
|
||||
|
||||
#### Python
|
||||
|
||||
There are two methods in `devtools_testutils`, [start_test_proxy][start_test_proxy] and
|
||||
[stop_test_proxy][stop_test_proxy], that can be used to manually start and stop the test proxy. Like
|
||||
`docker-start-proxy.ps1`, `start_test_proxy` will automatically fetch the proxy Docker image for you and start the
|
||||
container if it's not already running.
|
||||
|
||||
For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].
|
||||
|
||||
### Use `pytest.mark.parametrize` with migrated tests
|
||||
|
||||
Migrating tests to use basic `pytest` tools allows us to take advantage of helpful features such as
|
||||
|
@ -493,12 +463,10 @@ client to the test.
|
|||
|
||||
|
||||
[detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
|
||||
[docker_install]: https://docs.docker.com/get-docker/
|
||||
[docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
|
||||
|
||||
[env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
|
||||
|
||||
[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
|
||||
[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
|
||||
|
||||
[mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py
|
||||
|
||||
|
@ -507,7 +475,6 @@ client to the test.
|
|||
[parametrize_class]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/_test_case.py#L59
|
||||
[pipelines_ci]: https://github.com/Azure/azure-sdk-for-python/blob/5ba894966ed6b0e1ee8d854871f8c2da36a73d79/sdk/eventgrid/ci.yml#L30
|
||||
[pipelines_live]: https://github.com/Azure/azure-sdk-for-python/blob/e2b5852deaef04752c1323d2ab0958f83b98858f/sdk/textanalytics/tests.yml#L26-L27
|
||||
[podman]: https://podman.io/
|
||||
[proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md
|
||||
[py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
|
||||
[pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
|
||||
|
|
|
@ -10,23 +10,15 @@ GitHub repository, and documentation of how to set up and use the proxy can be f
|
|||
## Table of contents
|
||||
- [Guide for test proxy troubleshooting](#guide-for-test-proxy-troubleshooting)
|
||||
- [Table of contents](#table-of-contents)
|
||||
- [General troubleshooting tip](#general-troubleshooting-tip)
|
||||
- [Test collection failure](#test-collection-failure)
|
||||
- [Errors in tests using resource preparers](#errors-in-tests-using-resource-preparers)
|
||||
- [Playback failures from body matching errors](#playback-failures-from-body-matching-errors)
|
||||
- [Recordings not being produced](#recordings-not-being-produced)
|
||||
- [KeyError during container startup](#keyerror-during-container-startup)
|
||||
- [ConnectionError during test startup](#connectionerror-during-test-startup)
|
||||
- [ConnectionError during tests](#connectionerror-during-tests)
|
||||
- [Different error than expected when using proxy](#different-error-than-expected-when-using-proxy)
|
||||
- [Test setup failure in test pipeline](#test-setup-failure-in-test-pipeline)
|
||||
- [Fixture not found error](#fixture-not-found-error)
|
||||
|
||||
## General troubleshooting tip
|
||||
|
||||
For any issue that may come up, it's generally a good idea to first try deleting any existing proxy container (which
|
||||
will be called `ambitious_azsdk_test_proxy`) and creating a new one by running tests. This will fetch the latest tag of
|
||||
the test proxy Docker container, meaning the latest version of the proxy tool will be used.
|
||||
|
||||
## Test collection failure
|
||||
|
||||
Because tests are now using pure `pytest` conventions without `unittest.TestCase` components, discovering tests with
|
||||
|
@ -56,22 +48,14 @@ matching enabled by default.
|
|||
|
||||
## Recordings not being produced
|
||||
|
||||
First, make sure that the environment variable `AZURE_SKIP_LIVE_RECORDING` isn't set to "true". If it's not and live
|
||||
tests still aren't producing recordings, try deleting the `ambitious_azsdk_test_proxy` Docker container and re-running
|
||||
tests. The recording storage location is determined when the test proxy Docker container is created. If there are
|
||||
multiple local copies of the `azure-sdk-for-python` repo on your machine, the container could be storing recordings in
|
||||
the wrong repo.
|
||||
Ensure the environment variable `AZURE_SKIP_LIVE_RECORDING` **isn't** set to "true", and that `AZURE_TEST_RUN_LIVE`
|
||||
**is** set to "true".
|
||||
|
||||
## KeyError during container startup
|
||||
## ConnectionError during tests
|
||||
|
||||
Try updating your machine's version of Docker. Older versions of Docker may not return a status to indicate whether or
|
||||
not the proxy container is running, which the [proxy_startup.py][proxy_startup] script needs to determine.
|
||||
|
||||
## ConnectionError during test startup
|
||||
|
||||
For example, you may see a `requests.exceptions.ConnectionError` when trying to contact URL `/Info/Available`. This
|
||||
means that the test proxy tool wasn't started up properly, so requests to the tool are failing. Make sure Docker is
|
||||
installed and is up to date, and ensure that Linux containers are being used.
|
||||
For example, you may see a `requests.exceptions.ConnectionError` when trying to make service or sanitizer setup
|
||||
requests. This means that the test proxy tool never started correctly; ensure the `test_proxy` fixture is being invoked
|
||||
during test startup so that the tool is available during tests.
|
||||
|
||||
## Different error than expected when using proxy
|
||||
|
||||
|
|
|
@ -169,10 +169,7 @@ To migrate an existing test suite to use the test proxy, or to learn more about
|
|||
|
||||
### Perform one-time test proxy setup
|
||||
|
||||
1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
|
||||
2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
|
||||
3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
|
||||
order to communicate with the test proxy over a secure connection.
|
||||
The test proxy uses a self-signed certificate to communicate with HTTPS. Follow the general setup instructions [here][proxy_cert_docs] to trust this certificate locally.
|
||||
|
||||
### Set up test resources
|
||||
|
||||
|
@ -383,11 +380,8 @@ to the process of updating recordings.
|
|||
|
||||
- The targeted library is already migrated to use the test proxy.
|
||||
- Git version > 2.25.0 is to on the machine and in the path. Git is used by the script and test proxy.
|
||||
- [Docker][docker_install] or [Podman][podman] is installed.
|
||||
- Global [git config settings][git_setup] are configured for `user.name` and `user.email`.
|
||||
- These settings are also set with environment variables `GIT_COMMIT_OWNER` and `GIT_COMMIT_EMAIL`, respectively (in your environment or your local `.env` file).
|
||||
- The environment variable `GIT_TOKEN` is set to a valid [personal access token][git_token] for your user (in your environment or your local `.env` file).
|
||||
- This token is necessary for authenticating git requests made in a Docker/Podman container.
|
||||
- Membership in the `azure-sdk-write` GitHub group.
|
||||
|
||||
Test recordings will be updated if tests are run while `AZURE_TEST_RUN_LIVE` is set to "true" and
|
||||
|
@ -409,15 +403,16 @@ The recording directory in this case is `2Km2Z8755`, the string between the two
|
|||
|
||||
After verifying that your recording updates look correct, you can use the [`manage_recordings.py`][manage_recordings]
|
||||
script from `azure-sdk-for-python/scripts` to push these recordings to the `azure-sdk-assets` repo. This script accepts
|
||||
a verb and a **relative** path to your package's `assets.json` file. For example, from the root of the
|
||||
`azure-sdk-for-python` repo:
|
||||
a verb and a **relative** path to your package's `assets.json` file (this path is optional, and simply `assets.json`
|
||||
by default). For example, from the root of the `azure-sdk-for-python` repo:
|
||||
```
|
||||
python scripts/manage_recordings.py push sdk/{service}/{package}/assets.json
|
||||
python scripts/manage_recordings.py push -p sdk/{service}/{package}/assets.json
|
||||
```
|
||||
|
||||
The verbs that can be provided to this script are "push", "restore", and "reset":
|
||||
- **push**: pushes recording updates to a new assets repo tag and updates the tag pointer in `assets.json`.
|
||||
- **restore**: fetches recordings from the assets repo, based on the tag pointer in `assets.json`.
|
||||
- **reset**: discards any pending changes to recordings, based on the tag pointer in `assets.json`.
|
||||
|
||||
After pushing your recordings, the `assets.json` file for your package will be updated to point to a new `Tag` that
|
||||
contains the updates. Include this `assets.json` update in any pull request to update the recordings pointer in the
|
||||
|
@ -714,8 +709,6 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
|
|||
[azure_portal]: https://portal.azure.com/
|
||||
[azure_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/7e66e3877519a15c1d4304eb69abf0a2281773/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L44
|
||||
|
||||
[docker_install]: https://docs.docker.com/get-docker/
|
||||
|
||||
[engsys_wiki]: https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/48/Create-a-new-Live-Test-pipeline?anchor=test-resources.json
|
||||
[env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
|
||||
|
||||
|
@ -732,7 +725,6 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
|
|||
[mgmt_settings_fake]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py
|
||||
|
||||
[packaging]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md
|
||||
[podman]: https://podman.io/
|
||||
[proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md
|
||||
[proxy_general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
|
||||
[proxy_migration_guide]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md
|
||||
|
|
|
@ -7,142 +7,103 @@
|
|||
import argparse
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
except:
|
||||
pass
|
||||
|
||||
import subprocess
|
||||
from devtools_testutils.config import PROXY_URL
|
||||
from devtools_testutils.proxy_startup import (
|
||||
ascend_to_root,
|
||||
check_proxy_availability,
|
||||
prepare_local_tool,
|
||||
stop_test_proxy,
|
||||
)
|
||||
|
||||
|
||||
# This file contains a script for managing test recordings in the azure-sdk-assets repository with Docker.
|
||||
TOOL_ENV_VAR = "PROXY_PID"
|
||||
|
||||
|
||||
# This file contains a script for managing test recordings in the azure-sdk-assets repository.
|
||||
#
|
||||
# INSTRUCTIONS FOR USE:
|
||||
#
|
||||
# - Set GIT_TOKEN, GIT_COMMIT_OWNER, and GIT_COMMIT_EMAIL environment variables to authenticate git requests.
|
||||
# These can be set in-process or added to a .env file at the root of or directory above your local copy of the
|
||||
# azure-sdk-for-python repository.
|
||||
# - Set your working directory to be inside your local copy of the azure-sdk-for-python repository.
|
||||
# - Run the following command:
|
||||
#
|
||||
# `python {path to script}/manage_recordings.py {verb} {relative path to package's assets.json file}`
|
||||
# `python {path to script}/manage_recordings.py {verb} [-p {relative path to package's assets.json file}]`
|
||||
#
|
||||
# For example, with the root of the azure-sdk-for-python repo as the working directory, you can push modified
|
||||
# azure-keyvault-keys recordings to the assets repo with:
|
||||
#
|
||||
# `python scripts/manage_recordings.py push sdk/keyvault/azure-keyvault-keys/assets.json`
|
||||
# `python scripts/manage_recordings.py push -p sdk/keyvault/azure-keyvault-keys/assets.json`
|
||||
#
|
||||
# If this script is run from the directory containing an assets.json file, no path needs to be provided. For example,
|
||||
# with a working directory at the azure-keyvault-keys package root:
|
||||
#
|
||||
# `python ../../../scripts/manage_recordings.py push`
|
||||
#
|
||||
# - In addition to "push", you can also use the "restore" or "reset" verbs in the same command format.
|
||||
#
|
||||
# * push: pushes recording updates to a new assets repo tag and updates the tag pointer in `assets.json`.
|
||||
# * restore: fetches recordings from the assets repo, based on the tag pointer in `assets.json`.
|
||||
# * reset: discards any pending changes to recordings, based on the tag pointer in `assets.json`.
|
||||
#
|
||||
# For more information about how recording asset synchronization, please refer to
|
||||
# https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/asset-sync/README.md.
|
||||
|
||||
# Load environment variables from user's .env file
|
||||
|
||||
CONTAINER_NAME = "azsdkengsys.azurecr.io/engsys/test-proxy"
|
||||
GIT_TOKEN = os.getenv("GIT_TOKEN", "")
|
||||
GIT_OWNER = os.getenv("GIT_COMMIT_OWNER", "")
|
||||
GIT_EMAIL = os.getenv("GIT_COMMIT_EMAIL", "")
|
||||
def start_test_proxy() -> str:
|
||||
"""Starts the test proxy and returns when the tool is ready to receive requests, returning the tool's local name."""
|
||||
repo_root = ascend_to_root(os.getcwd())
|
||||
tool_name = prepare_local_tool(repo_root)
|
||||
|
||||
# always start the proxy with these two defaults set
|
||||
passenv = {
|
||||
"ASPNETCORE_Kestrel__Certificates__Default__Path": os.path.join(
|
||||
repo_root, "eng", "common", "testproxy", "dotnet-devcert.pfx"
|
||||
),
|
||||
"ASPNETCORE_Kestrel__Certificates__Default__Password": "password",
|
||||
}
|
||||
# if they are already set, override with what is in os.environ
|
||||
passenv.update(os.environ)
|
||||
|
||||
# ----- HELPERS ----- #
|
||||
|
||||
|
||||
discovered_roots = []
|
||||
|
||||
|
||||
def ascend_to_root(start_dir_or_file: str) -> str:
|
||||
"""
|
||||
Given a path, ascend until encountering a folder with a `.git` folder present within it. Return that directory.
|
||||
|
||||
:param str start_dir_or_file: The starting directory or file. Either is acceptable.
|
||||
"""
|
||||
if os.path.isfile(start_dir_or_file):
|
||||
current_dir = os.path.dirname(start_dir_or_file)
|
||||
else:
|
||||
current_dir = start_dir_or_file
|
||||
|
||||
while current_dir is not None and not (os.path.dirname(current_dir) == current_dir):
|
||||
possible_root = os.path.join(current_dir, ".git")
|
||||
|
||||
# we need the git check to prevent ascending out of the repo
|
||||
if os.path.exists(possible_root):
|
||||
if current_dir not in discovered_roots:
|
||||
discovered_roots.append(current_dir)
|
||||
return current_dir
|
||||
else:
|
||||
current_dir = os.path.dirname(current_dir)
|
||||
|
||||
raise Exception(f'Requested target "{start_dir_or_file}" does not exist within a git repo.')
|
||||
|
||||
|
||||
def delete_container() -> None:
|
||||
"""Delete container if it remained"""
|
||||
proc = subprocess.Popen(
|
||||
shlex.split(f"docker rm -f {CONTAINER_NAME}"),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
stdin=subprocess.DEVNULL,
|
||||
shlex.split(f'{tool_name} start --storage-location="{repo_root}" -- --urls "{PROXY_URL}"'),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.STDOUT,
|
||||
env=passenv,
|
||||
)
|
||||
output, stderr = proc.communicate(timeout=10)
|
||||
return None
|
||||
os.environ[TOOL_ENV_VAR] = str(proc.pid)
|
||||
check_proxy_availability()
|
||||
return tool_name
|
||||
|
||||
|
||||
def get_image_tag(repo_root: str) -> str:
|
||||
"""Gets the test proxy Docker image tag from the target_version.txt file in /eng/common/testproxy"""
|
||||
version_file_location = os.path.relpath("eng/common/testproxy/target_version.txt")
|
||||
version_file_location_from_root = os.path.abspath(os.path.join(repo_root, version_file_location))
|
||||
|
||||
with open(version_file_location_from_root, "r") as f:
|
||||
image_tag = f.read().strip()
|
||||
|
||||
return image_tag
|
||||
|
||||
|
||||
# ----- CORE LOGIC ----- #
|
||||
|
||||
|
||||
if not (GIT_TOKEN and GIT_OWNER and GIT_EMAIL):
|
||||
raise ValueError(
|
||||
"GIT_TOKEN, GIT_COMMIT_OWNER, and GIT_COMMIT_EMAIL environment variables must be set, "
|
||||
"either in-process or in a .env file"
|
||||
)
|
||||
|
||||
# Prepare command arguments
|
||||
parser = argparse.ArgumentParser(description="Script for managing recording assets with Docker.")
|
||||
parser.add_argument("verb", help='The action verb for managing recordings: "push" or "restore".')
|
||||
parser.add_argument("verb", help='The action verb for managing recordings: "push", "restore", or "reset".')
|
||||
parser.add_argument(
|
||||
"path",
|
||||
"-p",
|
||||
"--path",
|
||||
default="assets.json",
|
||||
help='The *relative* path to your package\'s `assets.json` file. Default is "assets.json".',
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verb and args.path:
|
||||
try:
|
||||
normalized_path = args.path.replace("\\", "/")
|
||||
|
||||
current_directory = os.getcwd()
|
||||
repo_root = ascend_to_root(current_directory)
|
||||
image_tag = get_image_tag(repo_root)
|
||||
print("\nStarting the test proxy...")
|
||||
tool_name = start_test_proxy()
|
||||
|
||||
root_path = os.path.abspath(repo_root)
|
||||
cwd_relpath = os.path.relpath(current_directory, root_path)
|
||||
assets_path = os.path.join(cwd_relpath, args.path).replace("\\", "/")
|
||||
print(f"\nUpdating recordings with {args.verb.lower()} operation...")
|
||||
subprocess.run(
|
||||
shlex.split(
|
||||
f"{tool_name} {args.verb.lower()} -a {normalized_path}"
|
||||
),
|
||||
stdout=sys.stdout,
|
||||
stderr=sys.stderr,
|
||||
)
|
||||
|
||||
delete_container() # Delete any lingering container so a new one can be created with necessary environment variables
|
||||
|
||||
subprocess.run(
|
||||
shlex.split(
|
||||
f'docker run --rm -v "{repo_root}:/srv/testproxy" '
|
||||
f'-e "GIT_TOKEN={GIT_TOKEN}" -e "GIT_COMMIT_OWNER={GIT_OWNER}" -e "GIT_COMMIT_EMAIL={GIT_EMAIL}" '
|
||||
f"{CONTAINER_NAME}:{image_tag} test-proxy {args.verb.lower()} -a {assets_path}"
|
||||
),
|
||||
stdout=sys.stdout,
|
||||
stderr=sys.stderr,
|
||||
)
|
||||
finally:
|
||||
print("\nStopping the test proxy...")
|
||||
stop_test_proxy()
|
||||
|
|
|
@ -7,14 +7,21 @@
|
|||
import os
|
||||
import logging
|
||||
import shlex
|
||||
import sys
|
||||
import time
|
||||
import signal
|
||||
import platform
|
||||
import shutil
|
||||
import tarfile
|
||||
from typing import Optional
|
||||
import zipfile
|
||||
|
||||
import certifi
|
||||
import pytest
|
||||
import subprocess
|
||||
from urllib3 import PoolManager, Retry
|
||||
from urllib3.exceptions import SSLError
|
||||
|
||||
import urllib3
|
||||
from ci_tools.variables import in_ci
|
||||
|
||||
from .config import PROXY_URL
|
||||
from .helpers import is_live_and_not_recording
|
||||
|
@ -23,19 +30,54 @@ from .sanitizers import add_remove_header_sanitizer, set_custom_default_matcher
|
|||
|
||||
_LOGGER = logging.getLogger()
|
||||
|
||||
CONTAINER_NAME = "ambitious_azsdk_test_proxy"
|
||||
LINUX_IMAGE_SOURCE_PREFIX = "azsdkengsys.azurecr.io/engsys/testproxy-lin"
|
||||
WINDOWS_IMAGE_SOURCE_PREFIX = "azsdkengsys.azurecr.io/engsys/testproxy-win"
|
||||
CONTAINER_STARTUP_TIMEOUT = 60
|
||||
PROXY_MANUALLY_STARTED = os.getenv("PROXY_MANUAL_START", False)
|
||||
|
||||
PROXY_CHECK_URL = PROXY_URL + "/Info/Available"
|
||||
TOOL_ENV_VAR = "PROXY_PID"
|
||||
|
||||
discovered_roots = []
|
||||
AVAILABLE_TEST_PROXY_BINARIES = {
|
||||
"Windows": {
|
||||
"AMD64": {
|
||||
"system": "Windows",
|
||||
"machine": "AMD64",
|
||||
"file_name": "test-proxy-standalone-win-x64.zip",
|
||||
"executable": "Azure.Sdk.Tools.TestProxy.exe",
|
||||
},
|
||||
},
|
||||
"Linux": {
|
||||
"X86_64": {
|
||||
"system": "Linux",
|
||||
"machine": "X86_64",
|
||||
"file_name": "test-proxy-standalone-linux-x64.tar.gz",
|
||||
"executable": "Azure.Sdk.Tools.TestProxy",
|
||||
},
|
||||
"ARM64": {
|
||||
"system": "Linux",
|
||||
"machine": "ARM64",
|
||||
"file_name": "test-proxy-standalone-linux-arm64.tar.gz",
|
||||
"executable": "Azure.Sdk.Tools.TestProxy",
|
||||
},
|
||||
},
|
||||
"Darwin": {
|
||||
"X86_64": {
|
||||
"system": "Darwin",
|
||||
"machine": "X86_64",
|
||||
"file_name": "test-proxy-standalone-osx-x64.zip",
|
||||
"executable": "Azure.Sdk.Tools.TestProxy",
|
||||
},
|
||||
"ARM64": {
|
||||
"system": "Darwin",
|
||||
"machine": "ARM64",
|
||||
"file_name": "test-proxy-standalone-osx-arm64.zip",
|
||||
"executable": "Azure.Sdk.Tools.TestProxy",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
from urllib3 import PoolManager, Retry
|
||||
from urllib3.exceptions import HTTPError
|
||||
PROXY_DOWNLOAD_URL = "https://github.com/Azure/azure-sdk-tools/releases/download/Azure.Sdk.Tools.TestProxy_{}/{}"
|
||||
|
||||
discovered_roots = []
|
||||
|
||||
if os.getenv("REQUESTS_CA_BUNDLE"):
|
||||
http_client = PoolManager(
|
||||
|
@ -47,20 +89,32 @@ else:
|
|||
http_client = PoolManager(retries=Retry(total=1, raise_on_status=False))
|
||||
|
||||
|
||||
def get_image_tag(repo_root: str) -> str:
|
||||
"""Gets the test proxy Docker image tag from the target_version.txt file in /eng/common/testproxy"""
|
||||
def get_target_version(repo_root: str) -> str:
|
||||
"""Gets the target test-proxy version from the target_version.txt file in /eng/common/testproxy"""
|
||||
version_file_location = os.path.relpath("eng/common/testproxy/target_version.txt")
|
||||
version_file_location_from_root = os.path.abspath(os.path.join(repo_root, version_file_location))
|
||||
|
||||
with open(version_file_location_from_root, "r") as f:
|
||||
image_tag = f.read().strip()
|
||||
target_version = f.read().strip()
|
||||
|
||||
return image_tag
|
||||
return target_version
|
||||
|
||||
|
||||
def get_downloaded_version(repo_root: str) -> Optional[str]:
|
||||
"""Gets version from downloaded_version.txt within the local download folder"""
|
||||
|
||||
downloaded_version_file = os.path.abspath(os.path.join(repo_root, ".proxy", "downloaded_version.txt"))
|
||||
|
||||
if os.path.exists(downloaded_version_file):
|
||||
with open(downloaded_version_file, "r") as f:
|
||||
version = f.read().strip()
|
||||
return version
|
||||
else:
|
||||
return None
|
||||
|
||||
|
||||
def ascend_to_root(start_dir_or_file: str) -> str:
|
||||
"""
|
||||
Given a path, ascend until encountering a folder with a `.git` folder present within it. Return that directory.
|
||||
"""Given a path, ascend until encountering a folder with a `.git` folder present within it. Return that directory.
|
||||
|
||||
:param str start_dir_or_file: The starting directory or file. Either is acceptable.
|
||||
"""
|
||||
|
@ -83,25 +137,13 @@ def ascend_to_root(start_dir_or_file: str) -> str:
|
|||
raise Exception(f'Requested target "{start_dir_or_file}" does not exist within a git repo.')
|
||||
|
||||
|
||||
def delete_container() -> None:
|
||||
"""Delete container if it remained"""
|
||||
proc = subprocess.Popen(
|
||||
shlex.split(f"docker rm -f {CONTAINER_NAME}"),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
stdin=subprocess.DEVNULL,
|
||||
)
|
||||
output, stderr = proc.communicate(timeout=10)
|
||||
return None
|
||||
|
||||
|
||||
def check_availability() -> None:
|
||||
"""Attempts request to /Info/Available. If a test-proxy instance is responding, we should get a response."""
|
||||
try:
|
||||
response = http_client.request(method="GET", url=PROXY_CHECK_URL, timeout=10)
|
||||
return response.status
|
||||
# We get an SSLError if the container is started but the endpoint isn't available yet
|
||||
except urllib3.exceptions.SSLError as sslError:
|
||||
except SSLError as sslError:
|
||||
_LOGGER.debug(sslError)
|
||||
return 404
|
||||
except Exception as e:
|
||||
|
@ -109,15 +151,49 @@ def check_availability() -> None:
|
|||
return 404
|
||||
|
||||
|
||||
def check_system_proxy_availability() -> None:
|
||||
"""Checks for SSL_CERT_DIR and REQUESTS_CA_BUNDLE environment variables."""
|
||||
ssl_cert = "SSL_CERT_DIR"
|
||||
ca_bundle = "REQUESTS_CA_BUNDLE"
|
||||
def check_certificate_location(repo_root: str) -> None:
|
||||
"""Checks for SSL_CERT_DIR and REQUESTS_CA_BUNDLE environment variables.
|
||||
|
||||
if PROXY_URL.startswith("https") and not os.environ.get(ssl_cert):
|
||||
_LOGGER.error(f"Please ensure the '{ssl_cert}' environment variable is correctly set in your test environment")
|
||||
if PROXY_URL.startswith("https") and not os.environ.get(ca_bundle):
|
||||
_LOGGER.error(f"Please ensure the '{ca_bundle}' environment variable is correctly set in your test environment")
|
||||
If both variables aren't set, this function configures the certificate bundle and sets these environment variables
|
||||
for the duration of the process.
|
||||
"""
|
||||
ssl_cert_dir = "SSL_CERT_DIR"
|
||||
requests_ca_bundle = "REQUESTS_CA_BUNDLE"
|
||||
|
||||
if PROXY_URL.startswith("https") and not (os.environ.get(ssl_cert_dir) and os.environ.get(requests_ca_bundle)):
|
||||
_LOGGER.info(
|
||||
"Missing SSL_CERT_DIR and/or REQUESTS_CA_BUNDLE environment variables. "
|
||||
"Setting these for the current session."
|
||||
)
|
||||
|
||||
existing_root_pem = certifi.where()
|
||||
local_dev_cert = os.path.abspath(os.path.join(repo_root, 'eng', 'common', 'testproxy', 'dotnet-devcert.crt'))
|
||||
combined_filename = os.path.basename(local_dev_cert).split(".")[0] + ".pem"
|
||||
combined_folder = os.path.join(repo_root, '.certificate')
|
||||
combined_location = os.path.join(combined_folder, combined_filename)
|
||||
|
||||
# If no local certificate folder exists, create one
|
||||
if not os.path.exists(combined_folder):
|
||||
_LOGGER.info("Missing a test proxy certificate under azure-sdk-for-python/.certificate. Creating one now.")
|
||||
os.mkdir(combined_folder)
|
||||
|
||||
if not os.path.exists(combined_location):
|
||||
# Copy the dev cert's content into the new certificate bundle
|
||||
with open(local_dev_cert, "r") as f:
|
||||
data = f.read()
|
||||
with open(combined_location, "w") as f:
|
||||
f.write(data)
|
||||
|
||||
# Copy the existing CA bundle contents into the repository's certificate bundle
|
||||
with open(existing_root_pem, "r") as f:
|
||||
content = f.readlines()
|
||||
with open(combined_location, "a") as f:
|
||||
f.writelines(content)
|
||||
|
||||
if not os.environ.get(ssl_cert_dir):
|
||||
os.environ[ssl_cert_dir] = combined_folder
|
||||
if not os.environ.get(requests_ca_bundle):
|
||||
os.environ[requests_ca_bundle] = combined_location
|
||||
|
||||
|
||||
def check_proxy_availability() -> None:
|
||||
|
@ -130,67 +206,119 @@ def check_proxy_availability() -> None:
|
|||
now = time.time()
|
||||
|
||||
|
||||
def create_container(repo_root: str) -> None:
|
||||
"""Creates the test proxy Docker container"""
|
||||
# Most of the time, running this script on a Windows machine will work just fine, as Docker defaults to Linux
|
||||
# containers. However, in CI, Windows images default to _Windows_ containers. We cannot swap them. We can tell
|
||||
# if we're in a CI build by checking for the environment variable TF_BUILD.
|
||||
delete_container()
|
||||
def prepare_local_tool(repo_root: str) -> str:
|
||||
"""Returns the path to a downloaded executable."""
|
||||
|
||||
if sys.platform.startswith("win") and os.environ.get("TF_BUILD"):
|
||||
image_prefix = WINDOWS_IMAGE_SOURCE_PREFIX
|
||||
path_prefix = "C:"
|
||||
linux_container_args = ""
|
||||
target_proxy_version = get_target_version(repo_root)
|
||||
|
||||
download_folder = os.path.join(repo_root, ".proxy")
|
||||
|
||||
system = platform.system() # Darwin, Linux, Windows
|
||||
machine = platform.machine().upper() # arm64, x86_64, AMD64
|
||||
|
||||
if system in AVAILABLE_TEST_PROXY_BINARIES:
|
||||
available_for_system = AVAILABLE_TEST_PROXY_BINARIES[system]
|
||||
|
||||
if machine in available_for_system:
|
||||
target_info = available_for_system[machine]
|
||||
|
||||
downloaded_version = get_downloaded_version(repo_root)
|
||||
download_necessary = not downloaded_version == target_proxy_version
|
||||
|
||||
if download_necessary:
|
||||
if os.path.exists(download_folder):
|
||||
# cleanup the directory for re-download
|
||||
shutil.rmtree(download_folder)
|
||||
os.makedirs(download_folder)
|
||||
|
||||
download_url = PROXY_DOWNLOAD_URL.format(target_proxy_version, target_info["file_name"])
|
||||
download_file = os.path.join(download_folder, target_info["file_name"])
|
||||
|
||||
http_client = PoolManager()
|
||||
with open(download_file, "wb") as out:
|
||||
r = http_client.request("GET", download_url, preload_content=False)
|
||||
shutil.copyfileobj(r, out)
|
||||
|
||||
if download_file.endswith(".zip"):
|
||||
with zipfile.ZipFile(download_file, "r") as zip_ref:
|
||||
zip_ref.extractall(download_folder)
|
||||
|
||||
if download_file.endswith(".tar.gz"):
|
||||
with tarfile.open(download_file) as tar_ref:
|
||||
tar_ref.extractall(download_folder)
|
||||
|
||||
os.remove(download_file) # Remove downloaded file after contents are extracted
|
||||
|
||||
# Record downloaded version for later comparison with target version in repo
|
||||
with open(os.path.join(download_folder, "downloaded_version.txt"), "w") as f:
|
||||
f.writelines([target_proxy_version])
|
||||
|
||||
return os.path.abspath(os.path.join(download_folder, target_info["executable"])).replace("\\", "/")
|
||||
else:
|
||||
_LOGGER.error(f'There are no available standalone proxy binaries for platform "{machine}".')
|
||||
raise Exception(
|
||||
"Unable to download a compatible standalone proxy for the current platform. File an issue against "
|
||||
"Azure/azure-sdk-tools with this error."
|
||||
)
|
||||
else:
|
||||
image_prefix = LINUX_IMAGE_SOURCE_PREFIX
|
||||
path_prefix = ""
|
||||
linux_container_args = "--add-host=host.docker.internal:host-gateway"
|
||||
|
||||
image_tag = get_image_tag(repo_root)
|
||||
subprocess.Popen(
|
||||
shlex.split(
|
||||
f"docker run --rm --name {CONTAINER_NAME} -v '{repo_root}:{path_prefix}/srv/testproxy' "
|
||||
f"{linux_container_args} -p 5001:5001 -p 5000:5000 {image_prefix}:{image_tag}"
|
||||
),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
stdin=subprocess.DEVNULL,
|
||||
)
|
||||
_LOGGER.error(f'There are no available standalone proxy binaries for system "{system}".')
|
||||
raise Exception(
|
||||
"Unable to download a compatible standalone proxy for the current system. File an issue against "
|
||||
"Azure/azure-sdk-tools with this error."
|
||||
)
|
||||
|
||||
|
||||
def start_test_proxy(request) -> None:
|
||||
"""Starts the test proxy and returns when the proxy server is ready to receive requests. In regular use
|
||||
cases, this will auto-start the test-proxy docker container. In CI, or when environment variable TF_BUILD is set, this
|
||||
function will start the test-proxy .NET tool."""
|
||||
"""Starts the test proxy and returns when the proxy server is ready to receive requests.
|
||||
|
||||
In regular use cases, this will auto-start the test-proxy docker container. In CI, or when environment variable
|
||||
TF_BUILD is set, this function will start the test-proxy .NET tool.
|
||||
"""
|
||||
|
||||
repo_root = ascend_to_root(request.node.items[0].module.__file__)
|
||||
check_system_proxy_availability()
|
||||
check_certificate_location(repo_root)
|
||||
|
||||
if not PROXY_MANUALLY_STARTED:
|
||||
if os.getenv("TF_BUILD"):
|
||||
_LOGGER.info("Starting the test proxy tool...")
|
||||
if check_availability() == 200:
|
||||
_LOGGER.debug("Tool is responding, exiting...")
|
||||
else:
|
||||
envname = os.getenv("TOX_ENV_NAME", "default")
|
||||
root = os.getenv("BUILD_SOURCESDIRECTORY", repo_root)
|
||||
log = open(os.path.join(root, "_proxy_log_{}.log".format(envname)), "a")
|
||||
if check_availability() == 200:
|
||||
_LOGGER.debug("Tool is responding, exiting...")
|
||||
else:
|
||||
root = os.getenv("BUILD_SOURCESDIRECTORY", repo_root)
|
||||
_LOGGER.info("{} is calculated repo root".format(root))
|
||||
|
||||
_LOGGER.info("{} is calculated repo root".format(root))
|
||||
# If we're in CI, allow for tox environment parallelization and write proxy output to a log file
|
||||
log = None
|
||||
if in_ci():
|
||||
envname = os.getenv("TOX_ENV_NAME", "default")
|
||||
log = open(os.path.join(root, "_proxy_log_{}.log".format(envname)), "a")
|
||||
|
||||
os.environ["PROXY_ASSETS_FOLDER"] = os.path.join(root, "l", envname)
|
||||
if not os.path.exists(os.environ["PROXY_ASSETS_FOLDER"]):
|
||||
os.makedirs(os.environ["PROXY_ASSETS_FOLDER"])
|
||||
|
||||
proc = subprocess.Popen(
|
||||
shlex.split('test-proxy start --storage-location="{}" -- --urls "{}"'.format(root, PROXY_URL)),
|
||||
stdout=log,
|
||||
stderr=log,
|
||||
)
|
||||
os.environ[TOOL_ENV_VAR] = str(proc.pid)
|
||||
else:
|
||||
_LOGGER.info("Starting the test proxy container...")
|
||||
create_container(repo_root)
|
||||
if os.getenv("TF_BUILD"):
|
||||
_LOGGER.info("Starting the test proxy tool from dotnet tool cache...")
|
||||
tool_name = "test-proxy"
|
||||
else:
|
||||
_LOGGER.info("Downloading and starting standalone proxy executable...")
|
||||
tool_name = prepare_local_tool(root)
|
||||
|
||||
# Always start the proxy with these two defaults set to allow SSL connection
|
||||
passenv = {
|
||||
"ASPNETCORE_Kestrel__Certificates__Default__Path": os.path.join(
|
||||
root, "eng", "common", "testproxy", "dotnet-devcert.pfx"
|
||||
),
|
||||
"ASPNETCORE_Kestrel__Certificates__Default__Password": "password",
|
||||
}
|
||||
# If they are already set, override what we give the proxy with what is in os.environ
|
||||
passenv.update(os.environ)
|
||||
|
||||
proc = subprocess.Popen(
|
||||
shlex.split(f'{tool_name} start --storage-location="{root}" -- --urls "{PROXY_URL}"'),
|
||||
stdout=log or subprocess.DEVNULL,
|
||||
stderr=log or subprocess.STDOUT,
|
||||
env=passenv,
|
||||
)
|
||||
os.environ[TOOL_ENV_VAR] = str(proc.pid)
|
||||
|
||||
# Wait for the proxy server to become available
|
||||
check_proxy_availability()
|
||||
|
@ -205,22 +333,12 @@ def stop_test_proxy() -> None:
|
|||
"""Stops any running instance of the test proxy"""
|
||||
|
||||
if not PROXY_MANUALLY_STARTED:
|
||||
if os.getenv("TF_BUILD"):
|
||||
_LOGGER.info("Stopping the test proxy tool...")
|
||||
_LOGGER.info("Stopping the test proxy tool...")
|
||||
|
||||
try:
|
||||
os.kill(int(os.getenv(TOOL_ENV_VAR)), signal.SIGTERM)
|
||||
except:
|
||||
_LOGGER.debug("Unable to kill running test-proxy process.")
|
||||
|
||||
else:
|
||||
_LOGGER.info("Stopping the test proxy container...")
|
||||
subprocess.Popen(
|
||||
shlex.split(f"docker stop {CONTAINER_NAME}"),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
stdin=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
os.kill(int(os.getenv(TOOL_ENV_VAR)), signal.SIGTERM)
|
||||
except:
|
||||
_LOGGER.debug("Unable to kill running test-proxy process.")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
|
|
|
@ -51,6 +51,6 @@ setup(
|
|||
},
|
||||
extras_require={
|
||||
":python_version>='3.5'": ["pytest-asyncio>=0.9.0"],
|
||||
"build": ["six", "setuptools", "pyparsing", "requests"],
|
||||
"build": ["six", "setuptools", "pyparsing", "certifi"],
|
||||
},
|
||||
)
|
||||
|
|
|
@ -0,0 +1,6 @@
|
|||
import pytest
|
||||
from devtools_testutils import test_proxy
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def start_proxy(test_proxy):
|
||||
return
|
|
@ -0,0 +1,26 @@
|
|||
# -------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for
|
||||
# license information.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
from urllib3 import PoolManager, Retry
|
||||
import os
|
||||
from devtools_testutils.proxy_startup import PROXY_CHECK_URL
|
||||
|
||||
if os.getenv("REQUESTS_CA_BUNDLE"):
|
||||
http_client = PoolManager(
|
||||
retries=Retry(total=3, raise_on_status=False),
|
||||
cert_reqs="CERT_REQUIRED",
|
||||
ca_certs=os.getenv("REQUESTS_CA_BUNDLE"),
|
||||
)
|
||||
else:
|
||||
http_client = PoolManager(retries=Retry(total=1, raise_on_status=False))
|
||||
|
||||
|
||||
class TestProxyIntegration:
|
||||
# These tests are checking spinup of the proxy, not automatic redirect.
|
||||
# Therefore we are not using recorded_by_proxy decorator or recorded_test fixture
|
||||
def test_tool_spinup_http(self):
|
||||
result = http_client.request("GET", PROXY_CHECK_URL)
|
||||
assert(result.status == 200)
|
Загрузка…
Ссылка в новой задаче