Setting up dev as V4 branch for PyWorker (#936)

* Setting up dev as V4 branch for PyWorker
* Installing Dotnet 6 for AzurePipelines and GH Actions
* Disabling SharedMem tests from Mac, and disabling shm when disabled.
* Logging stdout errors, and adding proc kill for lingering processes
* Updating extension version in csproj
* Enable SharedMemory for Tests
* Adding extensionBundle info to host.json template
* Adding extension.csproj file for correctly loading extensions.dll
* Tweaking csproj for e2e tests
* Reintroducing Trace Logging
* Adding some sleep time
* Solving the timestamp timezone awareness issue
* Fixing UTs and E2E tests, setup.py changes, and v4 pipeline changes
* Formatting and Typo changes
* Refactoring azure.pipelines.yml file.
This commit is contained in:
Varad Meru 2022-02-05 04:05:36 +05:30 коммит произвёл GitHub
Родитель 3755e68eb8
Коммит 5e522fc610
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
27 изменённых файлов: 554 добавлений и 251 удалений

Просмотреть файл

@ -8,6 +8,8 @@ ignore = W503,E402,E731
exclude = .git, __pycache__, build, dist, .eggs, .github, .local, docs/,
Samples, azure_functions_worker/protos/,
azure_functions_worker/_thirdparty/typing_inspect.py,
tests/unittests/test_typing_inspect.py, .venv*, .env*, .vscode, venv*
tests/unittests/test_typing_inspect.py,
tests/unittests/broken_functions/syntax_error/main.py,
.env*, .vscode, venv*, *.venv*,
max-line-length = 80
max-line-length = 80

4
.github/workflows/ci_e2e_workflow.yml поставляемый
Просмотреть файл

@ -38,6 +38,10 @@ jobs:
uses: actions/setup-dotnet@v1
with:
dotnet-version: '5.0.x'
- name: Set up Dotnet 6.x
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.x'
- name: Install dependencies and the worker
run: |
retry() {

4
.github/workflows/ut_ci_workflow.yml поставляемый
Просмотреть файл

@ -35,6 +35,10 @@ jobs:
uses: actions/setup-dotnet@v1
with:
dotnet-version: '5.0.x'
- name: Set up Dotnet 6.x
uses: actions/setup-dotnet@v1
with:
dotnet-version: '6.x'
- name: Install dependencies and the worker
run: |
retry() {

7
.gitignore поставляемый
Просмотреть файл

@ -125,4 +125,9 @@ prof/
# E2E Integration Test Core Tools
.ci/e2e_integration_test/Azure.Functions.Cli/
.ci/e2e_integration_test/Azure.Functions.Cli*
.ci/e2e_integration_test/Azure.Functions.Cli*
# Lingering test files
tests/**/host.json
tests/**/bin
tests/**/extensions.csproj

Просмотреть файл

@ -5,7 +5,7 @@
|master|[![Build Status](https://azfunc.visualstudio.com/Azure%20Functions/_apis/build/status/Azure.azure-functions-python-worker?branchName=master)](https://azfunc.visualstudio.com/Azure%20Functions/_build/latest?definitionId=57&branchName=master)|[![codecov](https://codecov.io/gh/Azure/azure-functions-python-worker/branch/master/graph/badge.svg)](https://codecov.io/gh/Azure/azure-functions-python-worker)|![CI Unit tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20Unit%20tests/badge.svg?branch=master)|![CI E2E tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20E2E%20tests/badge.svg?branch=master)
|dev|[![Build Status](https://azfunc.visualstudio.com/Azure%20Functions/_apis/build/status/Azure.azure-functions-python-worker?branchName=dev)](https://azfunc.visualstudio.com/Azure%20Functions/_build/latest?definitionId=57&branchName=dev)|[![codecov](https://codecov.io/gh/Azure/azure-functions-python-worker/branch/dev/graph/badge.svg)](https://codecov.io/gh/Azure/azure-functions-python-worker)|![CI Unit tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20Unit%20tests/badge.svg?branch=dev)|![CI E2E tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20E2E%20tests/badge.svg?branch=dev)
Python support for Azure Functions is based on Python 3.6, Python 3.7, and Python 3.8, serverless hosting on Linux and the Functions 2.0 and 3.0 runtime.
Python support for Azure Functions is based on Python 3.6, Python 3.7, Python 3.8, and Python 3.9 serverless hosting on Linux and the Functions 2.0, 3.0 and 4.0 runtime.
Here is the current status of Python in Azure Functions:
@ -14,9 +14,12 @@ What are the supported Python versions?
|Azure Functions Runtime|Python 3.6|Python 3.7|Python 3.8|Python 3.9|
|---|---|---|---|---|
|Azure Functions 2.0|✔|✔|-|-|
|Azure Functions 3.0|✔|✔|✔|(preview)|
|Azure Functions 3.0|✔|✔|✔|✔|
|Azure Functions 4.0|-|✔|✔|✔|
What's available?
For information about Azure Functions Runtime, please refer to [Azure Functions runtime versions overview](https://docs.microsoft.com/en-us/azure/azure-functions/functions-versions) page.
### What's available?
- Build, test, debug and publish using Azure Functions Core Tools (CLI) or Visual Studio Code
- Deploy Python Function project onto consumption, dedicated, or elastic premium plan.
@ -28,7 +31,7 @@ What's coming?
- [Durable Functions For Python](https://github.com/Azure/azure-functions-durable-python)
# Get Started
###Get Started
- [Create your first Python function](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-function-python)
- [Developer guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python)

Просмотреть файл

@ -1,14 +1,17 @@
name: $(Date:yyyyMMdd).$(Rev:r)
trigger:
- release/2.*
- release/3.*
- release/4.*
- dev
variables:
DOTNET_VERSION: '3.1.405'
DOTNET_VERSION_5: '5.0.x'
DOTNET_VERSION_6: '6.x'
patchBuildNumberForDev: $(Build.BuildNumber)
PROD_V3_WORKER_PY : 'python/prodV3/worker.py'
PROD_V4_WORKER_PY : 'python/prodV4/worker.py'
jobs:
- job: Tests
@ -18,14 +21,14 @@ jobs:
- ImageOverride -equals MMSUbuntu20.04TLS
strategy:
matrix:
Python36:
pythonVersion: '3.6'
Python37:
pythonVersion: '3.7'
Python38:
pythonVersion: '3.8'
Python39:
pythonVersion: '3.9'
Python310:
pythonVersion: '3.10'
steps:
- task: UsePythonVersion@0
inputs:
@ -41,6 +44,11 @@ jobs:
inputs:
packageType: 'sdk'
version: $(DOTNET_VERSION_5)
- task: UseDotNet@2
displayName: 'Install DotNet 6.x'
inputs:
packageType: 'sdk'
version: $(DOTNET_VERSION_6)
- task: ShellScript@2
inputs:
disableAutoCwd: true
@ -55,24 +63,27 @@ jobs:
- ImageOverride -equals MMS2019TLS
strategy:
matrix:
Python36V2:
pythonVersion: '3.6'
workerPath: 'python/prodV2/worker.py'
Python37V2:
pythonVersion: '3.7'
workerPath: 'python/prodV2/worker.py'
Python36V3:
pythonVersion: '3.6'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V3:
pythonVersion: '3.7'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python38V3:
pythonVersion: '3.8'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python39V3:
pythonVersion: '3.9'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V4:
pythonVersion: '3.7'
workerPath: $(PROD_V4_WORKER_PY)
Python38V4:
pythonVersion: '3.8'
workerPath: $(PROD_V4_WORKER_PY)
Python39V4:
pythonVersion: '3.9'
workerPath: $(PROD_V4_WORKER_PY)
steps:
- template: pack/templates/win_env_gen.yml
parameters:
@ -88,18 +99,24 @@ jobs:
- ImageOverride -equals MMS2019TLS
strategy:
matrix:
Python37V2:
pythonVersion: '3.7'
workerPath: 'python/prodV2/worker.py'
Python37V3:
pythonVersion: '3.7'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python38V3:
pythonVersion: '3.8'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python39V3:
pythonVersion: '3.9'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V4:
pythonVersion: '3.7'
workerPath: $(PROD_V4_WORKER_PY)
Python38V4:
pythonVersion: '3.8'
workerPath: $(PROD_V4_WORKER_PY)
Python39V4:
pythonVersion: '3.9'
workerPath: $(PROD_V4_WORKER_PY)
steps:
- template: pack/templates/win_env_gen.yml
parameters:
@ -115,24 +132,27 @@ jobs:
- ImageOverride -equals MMSUbuntu20.04TLS
strategy:
matrix:
Python36V2:
pythonVersion: '3.6'
workerPath: 'python/prodV2/worker.py'
Python37V2:
pythonVersion: '3.7'
workerPath: 'python/prodV2/worker.py'
Python36V3:
pythonVersion: '3.6'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V3:
pythonVersion: '3.7'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python38V3:
pythonVersion: '3.8'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python39V3:
pythonVersion: '3.9'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V4:
pythonVersion: '3.7'
workerPath: $(PROD_V4_WORKER_PY)
Python38V4:
pythonVersion: '3.8'
workerPath: $(PROD_V4_WORKER_PY)
Python39V4:
pythonVersion: '3.9'
workerPath: $(PROD_V4_WORKER_PY)
steps:
- template: pack/templates/nix_env_gen.yml
parameters:
@ -145,24 +165,27 @@ jobs:
vmImage: 'macOS-10.15'
strategy:
matrix:
Python36V2:
pythonVersion: '3.6'
workerPath: 'python/prodV2/worker.py'
Python37V2:
pythonVersion: '3.7'
workerPath: 'python/prodV2/worker.py'
Python36V3:
pythonVersion: '3.6'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V3:
pythonVersion: '3.7'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python38V3:
pythonVersion: '3.8'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python39V3:
pythonVersion: '3.9'
workerPath: 'python/prodV3/worker.py'
workerPath: $(PROD_V3_WORKER_PY)
Python37V4:
pythonVersion: '3.7'
workerPath: $(PROD_V4_WORKER_PY)
Python38V4:
pythonVersion: '3.8'
workerPath: $(PROD_V4_WORKER_PY)
Python39V4:
pythonVersion: '3.9'
workerPath: $(PROD_V4_WORKER_PY)
steps:
- template: pack/templates/nix_env_gen.yml
parameters:
@ -174,8 +197,8 @@ jobs:
dependsOn: ['Build_WINDOWS_X64', 'Build_WINDOWS_X86', 'Build_LINUX_X64', 'Build_OSX_X64']
condition: or(startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'), eq(variables['Build.SourceBranch'], 'refs/heads/dev'))
pool:
#vmImage: 'vs2017-win2016'
name: '1ES-Hosted-AzFunc' #MMS2019TLS for Windows2019 or MMSUbuntu20.04TLS for ubuntu
# vmImage: 'vs2017-win2016'
name: '1ES-Hosted-AzFunc' # MMS2019TLS for Windows2019 or MMSUbuntu20.04TLS for ubuntu
demands:
- ImageOverride -equals MMS2019TLS
steps:
@ -183,23 +206,22 @@ jobs:
echo "Releasing from $BUILD_SOURCEBRANCHNAME"
apt install jq
if [[ $BUILD_SOURCEBRANCHNAME = 2\.* ]]
then
echo "Generating V2 Release Package for $BUILD_SOURCEBRANCHNAME"
NUSPEC="pack\Microsoft.Azure.Functions.V2.PythonWorker.nuspec"
WKVERSION="$BUILD_SOURCEBRANCHNAME"
elif [[ $BUILD_SOURCEBRANCHNAME = 3\.* ]]
if [[ $BUILD_SOURCEBRANCHNAME = 3\.* ]]
then
echo "Generating V3 Release Package for $BUILD_SOURCEBRANCHNAME"
NUSPEC="pack\Microsoft.Azure.Functions.V3.PythonWorker.nuspec"
WKVERSION="$BUILD_SOURCEBRANCHNAME"
elif [[ $BUILD_SOURCEBRANCHNAME = 4\.* ]]
then
echo "Generating V4 Release Package for $BUILD_SOURCEBRANCHNAME"
NUSPEC="pack\Microsoft.Azure.Functions.V4.PythonWorker.nuspec"
WKVERSION="$BUILD_SOURCEBRANCHNAME"
elif [[ $BUILD_SOURCEBRANCHNAME = dev ]]
then
echo "Generating V3 Integration Test Package for $BUILD_SOURCEBRANCHNAME"
LATEST_TAG=$(curl https://api.github.com/repos/Azure/azure-functions-python-worker/tags -s | jq '.[0].name' | sed 's/\"//g' | cut -d'.' -f-2)
NUSPEC="pack\Microsoft.Azure.Functions.V3.PythonWorker.nuspec"
# Only required for Integration Test. Version number contains date (e.g. 3.1.2.20211028-dev)
WKVERSION="3.$LATEST_TAG-$(patchBuildNumberForDev)"
echo "Generating V4 Integration Test Package for $BUILD_SOURCEBRANCHNAME"
VERSION=$(cat azure_functions_worker/version.py | tail -1 | cut -d' ' -f3 | sed "s/'//g")
NUSPEC="pack\Microsoft.Azure.Functions.V4.PythonWorker.nuspec"
WKVERSION="$VERSION-$(patchBuildNumberForDev)"
else
echo "No Matching Release Tag For $BUILD_SOURCEBRANCH"
fi
@ -236,4 +258,4 @@ jobs:
nuGetFeedType: 'internal'
publishVstsFeed: 'e6a70c92-4128-439f-8012-382fe78d6396/f37f760c-aebd-443e-9714-ce725cd427df'
allowPackageConflicts: true
displayName: '[Integration Test] Push NuGet package to the AzureFunctionsPreRelease feed'
displayName: '[Integration Test] Push NuGet package to the AzureFunctionsPreRelease feed'

Просмотреть файл

@ -1,4 +1,2 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
__version__ = '1.2.6'

Просмотреть файл

@ -82,3 +82,16 @@ class FileAccessor(metaclass=ABCMeta):
mem_map.write(consts.HeaderFlags.Initialized)
# Seek back the memory map to the original position
mem_map.seek(original_pos)
class DummyFileAccessor(FileAccessor):
def open_mem_map(self, mem_map_name: str, mem_map_size: int,
access: int = mmap.ACCESS_READ) -> Optional[mmap.mmap]:
pass
def create_mem_map(self, mem_map_name: str,
mem_map_size: int) -> Optional[mmap.mmap]:
pass
def delete_mem_map(self, mem_map_name: str, mem_map: mmap.mmap) -> bool:
pass

Просмотреть файл

@ -2,8 +2,13 @@
# Licensed under the MIT License.
import os
import sys
from .file_accessor import DummyFileAccessor
from .file_accessor_unix import FileAccessorUnix
from .file_accessor_windows import FileAccessorWindows
from ...constants import FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED
from ...utils.common import is_envvar_true
class FileAccessorFactory:
@ -13,6 +18,9 @@ class FileAccessorFactory:
"""
@staticmethod
def create_file_accessor():
if os.name == 'nt':
if sys.platform == "darwin" and not is_envvar_true(
FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED):
return DummyFileAccessor()
elif os.name == 'nt':
return FileAccessorWindows()
return FileAccessorUnix()

Просмотреть файл

@ -18,7 +18,6 @@ from typing import List, Optional
import grpc
from . import __version__
from . import bindings
from . import constants
from . import functions
@ -39,6 +38,8 @@ from .utils.common import get_app_setting, is_envvar_true
from .utils.dependency import DependencyManager
from .utils.tracing import marshall_exception_trace
from .utils.wrappers import disable_feature_by
from .version import VERSION
_TRUE = "true"
@ -262,7 +263,7 @@ class Dispatcher(metaclass=DispatcherMeta):
async def _handle__worker_init_request(self, req):
logger.info('Received WorkerInitRequest, '
'python version %s, worker version %s, request ID %s',
sys.version, __version__, self.request_id)
sys.version, VERSION, self.request_id)
enable_debug_logging_recommendation()
worker_init_request = req.worker_init_request

Просмотреть файл

@ -71,26 +71,39 @@ LOCALHOST = "127.0.0.1"
HOST_JSON_TEMPLATE = """\
{
"version": "2.0",
"logging": {
"logLevel": {
"default": "Trace"
}
},
"http": {
"routePrefix": "api"
},
"swagger": {
"enabled": true
},
"eventHub": {
"maxBatchSize": 1000,
"prefetchCount": 1000,
"batchCheckpointFrequency": 1
},
"functionTimeout": "00:05:00"
"logging": {"logLevel": {"default": "Trace"}}
}
"""
EXTENSION_CSPROJ_TEMPLATE = """\
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<WarningsAsErrors></WarningsAsErrors>
<DefaultItemExcludes>**</DefaultItemExcludes>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
Version="5.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
Version="3.1.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
Version="3.0.10" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage"
Version="4.0.5" />
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.Storage.Blobs"
Version="5.0.0" />
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues"
Version="5.0.0" />
<PackageReference
Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator"
Version="1.1.3" />
</ItemGroup>
</Project>
"""
SECRETS_TEMPLATE = """\
{
"masterKey": {
@ -246,7 +259,8 @@ class WebHostTestCase(unittest.TestCase, metaclass=WebHostTestCaseMeta):
self.host_stdout.seek(last_pos)
self.host_out = self.host_stdout.read()
self.host_stdout_logger.error(
f'Captured WebHost stdout:\n{self.host_out}')
f'Captured WebHost stdout from {self.host_stdout.name} '
f':\n{self.host_out}')
finally:
if test_exception is not None:
raise test_exception
@ -731,7 +745,10 @@ class _WebHostProxy:
self._proc.stderr.close()
self._proc.terminate()
self._proc.wait()
try:
self._proc.wait(20)
except subprocess.TimeoutExpired:
self._proc.kill()
def _find_open_port():
@ -882,7 +899,8 @@ def start_webhost(*, script_dir=None, stdout=None):
addr = f'http://{LOCALHOST}:{port}'
health_check_endpoint = f'{addr}/api/ping'
for _ in range(10):
host_out = stdout.readlines(100)
for _ in range(5):
try:
r = requests.get(health_check_endpoint,
params={'code': 'testFunctionKey'})
@ -891,13 +909,14 @@ def start_webhost(*, script_dir=None, stdout=None):
if 200 <= r.status_code < 300:
# Give the host a bit more time to settle
time.sleep(2)
time.sleep(1)
break
else:
print(f'Failed to ping {health_check_endpoint}', flush=True)
print(f'Failed to ping {health_check_endpoint}, status code: '
f'{r.status_code}', flush=True)
except requests.exceptions.ConnectionError:
pass
time.sleep(2)
time.sleep(1)
else:
proc.terminate()
try:
@ -905,7 +924,8 @@ def start_webhost(*, script_dir=None, stdout=None):
except subprocess.TimeoutExpired:
proc.kill()
raise RuntimeError('could not start the webworker in time. Please'
f' check the log file for details: {stdout.name} ')
f' check the log file for details: {stdout.name} \n'
f' Captured WebHost stdout:\n{host_out}')
return _WebHostProxy(proc, addr)
@ -963,11 +983,16 @@ def _setup_func_app(app_root):
extensions = app_root / 'bin'
ping_func = app_root / 'ping'
host_json = app_root / 'host.json'
extensions_csproj_file = app_root / 'extensions.csproj'
if not os.path.isfile(host_json):
with open(host_json, 'w') as f:
f.write(HOST_JSON_TEMPLATE)
if not os.path.isfile(extensions_csproj_file):
with open(extensions_csproj_file, 'w') as f:
f.write(EXTENSION_CSPROJ_TEMPLATE)
_symlink_dir(TESTS_ROOT / 'common' / 'ping', ping_func)
_symlink_dir(EXTENSIONS_PATH, extensions)
@ -976,8 +1001,11 @@ def _teardown_func_app(app_root):
extensions = app_root / 'bin'
ping_func = app_root / 'ping'
host_json = app_root / 'host.json'
extensions_csproj_file = app_root / 'extensions.csproj'
extensions_obj_file = app_root / 'obj'
for path in (extensions, ping_func, host_json):
for path in (extensions, ping_func, host_json, extensions_csproj_file,
extensions_obj_file):
remove_path(path)

Просмотреть файл

@ -0,0 +1,4 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
VERSION = '4.0.0'

Просмотреть файл

@ -0,0 +1,28 @@
<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2012/06/nuspec.xsd">
<metadata>
<id>Microsoft.Azure.Functions.PythonWorker</id>
<version>1.1.0</version>
<authors>Microsoft</authors>
<owners>Microsoft</owners>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<description>Microsoft Azure Functions Python Worker</description>
<copyright>© .NET Foundation. All rights reserved.</copyright>
</metadata>
<files>
<file src="..\3.7_WINDOWS_X64\**" target="tools\3.7\WINDOWS\X64" />
<file src="..\3.7_WINDOWS_X86\**" target="tools\3.7\WINDOWS\X86" />
<file src="..\3.7_LINUX_X64\**" target="tools\3.7\LINUX\X64" />
<file src="..\3.7_OSX_X64\**" target="tools\3.7\OSX\X64" />
<file src="..\3.8_WINDOWS_X64\**" target="tools\3.8\WINDOWS\X64" />
<file src="..\3.8_WINDOWS_X86\**" target="tools\3.8\WINDOWS\X86" />
<file src="..\3.8_LINUX_X64\**" target="tools\3.8\LINUX\X64" />
<file src="..\3.8_OSX_X64\**" target="tools\3.8\OSX\X64" />
<file src="..\3.9_WINDOWS_X64\**" target="tools\3.9\WINDOWS\X64" />
<file src="..\3.9_WINDOWS_X86\**" target="tools\3.9\WINDOWS\X86" />
<file src="..\3.9_LINUX_X64\**" target="tools\3.9\LINUX\X64" />
<file src="..\3.9_OSX_X64\**" target="tools\3.9\OSX\X64" />
<file src="..\python\prodV3\worker.config.json" target="tools" />
<file src="Microsoft.Azure.Functions.PythonWorker.targets" target="build" />
</files>
</package>

Просмотреть файл

@ -0,0 +1,12 @@
{
"description":{
"language":"python",
"defaultRuntimeVersion":"3.9",
"supportedOperatingSystems":["LINUX", "OSX", "WINDOWS"],
"supportedRuntimeVersions":["3.7", "3.8", "3.9"],
"supportedArchitectures":["X64", "X86"],
"extensions":[".py"],
"defaultExecutablePath":"python",
"defaultWorkerPath":"%FUNCTIONS_WORKER_RUNTIME_VERSION%/{os}/{architecture}/worker.py"
}
}

74
python/prodV4/worker.py Normal file
Просмотреть файл

@ -0,0 +1,74 @@
import os
import sys
from pathlib import Path
# User packages
PKGS_PATH = "site/wwwroot/.python_packages"
VENV_PKGS_PATH = "site/wwwroot/worker_venv"
PKGS = "lib/site-packages"
# Azure environment variables
AZURE_WEBSITE_INSTANCE_ID = "WEBSITE_INSTANCE_ID"
AZURE_CONTAINER_NAME = "CONTAINER_NAME"
AZURE_WEBJOBS_SCRIPT_ROOT = "AzureWebJobsScriptRoot"
def is_azure_environment():
"""Check if the function app is running on the cloud"""
return (AZURE_CONTAINER_NAME in os.environ
or AZURE_WEBSITE_INSTANCE_ID in os.environ)
def add_script_root_to_sys_path():
"""Append function project root to module finding sys.path"""
functions_script_root = os.getenv(AZURE_WEBJOBS_SCRIPT_ROOT)
if functions_script_root is not None:
sys.path.append(functions_script_root)
def determine_user_pkg_paths():
"""This finds the user packages when function apps are running on the cloud
For Python 3.7+, we only accept:
/home/site/wwwroot/.python_packages/lib/site-packages
"""
minor_version = sys.version_info[1]
home = Path.home()
pkgs_path = os.path.join(home, PKGS_PATH)
user_pkg_paths = []
if minor_version in (7, 8, 9):
user_pkg_paths.append(os.path.join(pkgs_path, PKGS))
else:
raise RuntimeError(f'Unsupported Python version: 3.{minor_version}')
return user_pkg_paths
if __name__ == '__main__':
# worker.py lives in the same directory as azure_functions_worker
func_worker_dir = str(Path(__file__).absolute().parent)
env = os.environ
if is_azure_environment():
user_pkg_paths = determine_user_pkg_paths()
joined_pkg_paths = os.pathsep.join(user_pkg_paths)
# On cloud, we prioritize third-party user packages
# over worker packages in PYTHONPATH
env['PYTHONPATH'] = f'{joined_pkg_paths}:{func_worker_dir}'
os.execve(sys.executable,
[sys.executable, '-m', 'azure_functions_worker']
+ sys.argv[1:],
env)
else:
# On local development, we prioritize worker packages over
# third-party user packages (in .venv)
sys.path.insert(1, func_worker_dir)
add_script_root_to_sys_path()
from azure_functions_worker import main
main.main()

312
setup.py
Просмотреть файл

@ -1,84 +1,138 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
import distutils.cmd
import glob
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
import json
import tempfile
import urllib.request
import zipfile
import re
from distutils import dir_util
from distutils.command import build
from distutils.dist import Distribution
from setuptools import setup
from setuptools.command import develop
from azure_functions_worker import __version__
from azure_functions_worker.version import VERSION
# The GitHub repository of the Azure Functions Host
WEBHOST_GITHUB_API = "https://api.github.com/repos/Azure/azure-functions-host"
WEBHOST_TAG_PREFIX = "v3."
WEBHOST_TAG_PREFIX = "v4."
# Extensions necessary for non-core bindings.
AZURE_EXTENSIONS = """\
<?xml version="1.0" encoding="UTF-8"?>
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
<WarningsAsErrors></WarningsAsErrors>
<DefaultItemExcludes>**</DefaultItemExcludes>
</PropertyGroup>
<ItemGroup>
<PackageReference
Include="Microsoft.NET.Sdk.Functions"
Version="3.0.3"
/>
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
Version="3.0.5"
/>
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
Version="3.0.6"
/>
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
Version="2.1.0"
/>
<PackageReference
Include="Microsoft.Azure.WebJobs.Extensions.Storage"
Version="3.0.10"
/>
<PackageReference
Include="Microsoft.Azure.WebJobs.ServiceBus"
Version="3.0.0-beta8"
/>
</ItemGroup>
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v4</AzureFunctionsVersion>
<WarningsAsErrors />
<DefaultItemExcludes>**</DefaultItemExcludes>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Sdk.Functions"
Version="4.0.1" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
Version="3.0.10" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
Version="5.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
Version="3.1.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage"
Version="4.0.5" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus"
Version="4.2.1" />
<PackageReference
Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator"
Version="1.1.3" />
</ItemGroup>
</Project>
"""
NUGET_CONFIG = """\
<?xml version="1.0" encoding="utf-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<packageSources>
<add key="nuget.org" value="https://www.nuget.org/api/v2/" />
<add key="azure_app_service"
value="https://www.myget.org/F/azure-appservice/api/v2" />
<add key="azure_app_service_staging"
value="https://www.myget.org/F/azure-appservice-staging/api/v2" />
<add key="buildTools"
value="https://www.myget.org/F/30de4ee06dd54956a82013fa17a3accb/" />
<add key="AspNetVNext"
value="https://www.myget.org/F/aspnetcore-dev/api/v3/index.json" />
</packageSources>
<packageSources>
<add key="nuget.org"
value="https://www.nuget.org/api/v2/" />
<add key="azure_app_service"
value="https://www.myget.org/F/azure-appservice/api/v2" />
<add key="azure_app_service_staging"
value="https://www.myget.org/F/azure-appservice-staging/api/v2" />
<add key="buildTools"
value="https://www.myget.org/F/30de4ee06dd54956a82013fa17a3accb/" />
<add key="AspNetVNext"
value="https://www.myget.org/F/aspnetcore-dev/api/v3/index.json" />
</packageSources>
</configuration>
"""
CLASSIFIERS = [
"Development Status :: 5 - Production/Stable",
'Programming Language :: Python',
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Operating System :: Microsoft :: Windows",
"Operating System :: POSIX",
"Operating System :: MacOS :: MacOS X",
"Environment :: Web Environment",
"License :: OSI Approved :: MIT License",
"Intended Audience :: Developers",
]
PACKAGES = [
"azure_functions_worker",
"azure_functions_worker.protos",
"azure_functions_worker.protos.identity",
"azure_functions_worker.protos.shared",
"azure_functions_worker.bindings",
"azure_functions_worker.bindings.shared_memory_data_transfer",
"azure_functions_worker.utils",
"azure_functions_worker._thirdparty"
]
INSTALL_REQUIRES = [
"grpcio~=1.43.0",
"grpcio-tools~=1.43.0",
"protobuf~=3.19.3",
"azure-functions==1.8.0"
]
EXTRA_REQUIRES = {
"dev": [
"azure-eventhub~=5.1.0",
"python-dateutil~=2.8.1",
"pycryptodome~=3.10.1",
"flake8~=3.7.9",
"mypy",
"pytest",
"requests==2.*",
"coverage",
"pytest-sugar",
"pytest-cov",
"pytest-xdist",
"pytest-randomly",
"pytest-instafail",
"pytest-rerunfailures",
"ptvsd"
]
}
class BuildGRPC:
"""Generate gRPC bindings."""
def _gen_grpc(self):
@ -128,9 +182,10 @@ class BuildGRPC:
# https://github.com/protocolbuffers/protobuf/issues/1491
self.make_absolute_imports(compiled_files)
dir_util.copy_tree(built_protos_dir, str(proto_root_dir))
dir_util.copy_tree(str(built_protos_dir), str(proto_root_dir))
def make_absolute_imports(self, compiled_files):
@staticmethod
def make_absolute_imports(compiled_files):
for compiled in compiled_files:
with open(compiled, 'r+') as f:
content = f.read()
@ -153,19 +208,19 @@ class BuildGRPC:
f.truncate()
class build(build.build, BuildGRPC):
class BuildProtos(build.build, BuildGRPC):
def run(self, *args, **kwargs):
self._gen_grpc()
super().run(*args, **kwargs)
super().run()
class develop(develop.develop, BuildGRPC):
class Development(develop.develop, BuildGRPC):
def run(self, *args, **kwargs):
self._gen_grpc()
super().run(*args, **kwargs)
super().run()
class extension(distutils.cmd.Command):
class Extension(distutils.cmd.Command):
description = (
'Resolve WebJobs Extensions from AZURE_EXTENSIONS and NUGET_CONFIG.'
)
@ -174,9 +229,13 @@ class extension(distutils.cmd.Command):
'A path to the directory where extension should be installed')
]
def initialize_options(self):
def __init__(self, dist: Distribution):
super().__init__(dist)
self.extensions_dir = None
def initialize_options(self):
pass
def finalize_options(self):
if self.extensions_dir is None:
self.extensions_dir = \
@ -204,7 +263,7 @@ class extension(distutils.cmd.Command):
args=['dotnet', 'build', '-o', '.'], check=True,
cwd=str(self.extensions_dir),
stdout=sys.stdout, stderr=sys.stderr, env=env)
except Exception:
except Exception: # NoQA
print(".NET Core SDK is required to build the extensions. "
"Please visit https://aka.ms/dotnet-download")
sys.exit(1)
@ -213,7 +272,7 @@ class extension(distutils.cmd.Command):
self._install_extensions()
class webhost(distutils.cmd.Command):
class Webhost(distutils.cmd.Command):
description = 'Download and setup Azure Functions Web Host.'
user_options = [
('webhost-version', None,
@ -222,9 +281,13 @@ class webhost(distutils.cmd.Command):
'A path to the directory where Azure Web Host will be installed.'),
]
def initialize_options(self):
self.webhost_version = None
def __init__(self, dist: Distribution):
super().__init__(dist)
self.webhost_dir = None
self.webhost_version = None
def initialize_options(self):
pass
def finalize_options(self):
if self.webhost_version is None:
@ -234,7 +297,8 @@ class webhost(distutils.cmd.Command):
self.webhost_dir = \
pathlib.Path(__file__).parent / 'build' / 'webhost'
def _get_webhost_version(self) -> str:
@staticmethod
def _get_webhost_version() -> str:
# Return the latest matched version (e.g. 3.0.15278)
github_api_url = f'{WEBHOST_GITHUB_API}/tags?page=1&per_page=10'
print(f'Checking latest webhost version from {github_api_url}')
@ -243,12 +307,13 @@ class webhost(distutils.cmd.Command):
# As tags are placed in time desending order, the latest v3
# tag should be the first occurance starts with 'v3.' string
latest_v3 = [
latest = [
gt for gt in tags if gt['name'].startswith(WEBHOST_TAG_PREFIX)
]
return latest_v3[0]['name'].replace('v', '')
return latest[0]['name'].replace('v', '')
def _download_webhost_zip(self, version: str) -> str:
@staticmethod
def _download_webhost_zip(version: str) -> str:
# Return the path of the downloaded host
temporary_file = tempfile.NamedTemporaryFile()
zip_url = (
@ -269,13 +334,15 @@ class webhost(distutils.cmd.Command):
print(f'Functions Host is downloaded into {temporary_file.name}')
return temporary_file.name
def _create_webhost_folder(self, dest_folder: pathlib.Path):
@staticmethod
def _create_webhost_folder(dest_folder: pathlib.Path):
if dest_folder.exists():
shutil.rmtree(dest_folder)
os.makedirs(dest_folder, exist_ok=True)
print(f'Functions Host folder is created in {dest_folder}')
def _extract_webhost_zip(self, version: str, src_zip: str, dest: str):
@staticmethod
def _extract_webhost_zip(version: str, src_zip: str, dest: str):
print(f'Extracting Functions Host from {src_zip}')
with zipfile.ZipFile(src_zip) as archive:
@ -311,7 +378,8 @@ class webhost(distutils.cmd.Command):
print(f'Functions Host is extracted into {dest}')
def _chmod_protobuf_generation_script(self, webhost_dir: pathlib.Path):
@staticmethod
def _chmod_protobuf_generation_script(webhost_dir: pathlib.Path):
# This script is needed to set to executable in order to build the
# WebJobs.Script.Grpc project in Linux and MacOS
script_path = (
@ -321,7 +389,8 @@ class webhost(distutils.cmd.Command):
print('Change generate_protos.sh script permission')
os.chmod(script_path, 0o555)
def _compile_webhost(self, webhost_dir: pathlib.Path):
@staticmethod
def _compile_webhost(webhost_dir: pathlib.Path):
print(f'Compiling Functions Host from {webhost_dir}')
try:
@ -330,7 +399,7 @@ class webhost(distutils.cmd.Command):
check=True,
cwd=str(webhost_dir),
stdout=sys.stdout, stderr=sys.stderr)
except Exception:
except Exception: # NoQA
print(f"Failed to compile webhost in {webhost_dir}. "
".NET Core SDK is required to build the solution. "
"Please visit https://aka.ms/dotnet-download",
@ -350,73 +419,54 @@ class webhost(distutils.cmd.Command):
self._compile_webhost(self.webhost_dir)
with open("README.md") as readme:
long_description = readme.read()
class Clean(distutils.cmd.Command):
description = 'Clean up build generated files'
user_options = []
def __init__(self, dist: Distribution):
super().__init__(dist)
self.dir_list_to_delete = [
"build"
]
def initialize_options(self) -> None:
pass
def finalize_options(self) -> None:
pass
def run(self) -> None:
for dir_to_delete in self.dir_list_to_delete:
dir_delete = pathlib.Path(dir_to_delete)
if dir_delete.exists():
dir_util.remove_tree(str(dir_delete))
COMMAND_CLASS = {
'develop': Development,
'build': BuildProtos,
'webhost': Webhost,
'extension': Extension,
'clean': Clean
}
setup(
name='azure-functions-worker',
version=__version__,
description='Python Language Worker for Azure Functions Host',
author="Microsoft Corp.",
name="azure-functions-worker",
version=VERSION,
description="Python Language Worker for Azure Functions Host",
author="Azure Functions team at Microsoft Corp.",
author_email="azurefunctions@microsoft.com",
keywords="azure azurefunctions python",
keywords="azure functions azurefunctions python serverless",
url="https://github.com/Azure/azure-functions-python-worker",
long_description=long_description,
long_description_content_type='text/markdown',
classifiers=[
'Development Status :: 5 - Production/Stable',
'License :: OSI Approved :: MIT License',
'Intended Audience :: Developers',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Operating System :: Microsoft :: Windows',
'Operating System :: POSIX',
'Operating System :: MacOS :: MacOS X',
'Environment :: Web Environment',
],
license='MIT',
packages=['azure_functions_worker',
'azure_functions_worker.protos',
'azure_functions_worker.protos.identity',
'azure_functions_worker.protos.shared',
'azure_functions_worker.bindings',
'azure_functions_worker.bindings.shared_memory_data_transfer',
'azure_functions_worker.utils',
'azure_functions_worker._thirdparty'],
install_requires=[
'grpcio~=1.33.2',
'grpcio-tools~=1.33.2',
],
extras_require={
'dev': [
'azure-functions==1.8.0',
'azure-eventhub~=5.1.0',
'python-dateutil~=2.8.1',
'pycryptodome~=3.10.1',
'flake8~=3.7.9',
'mypy',
'pytest',
'requests==2.*',
'coverage',
'pytest-sugar',
'pytest-cov',
'pytest-xdist',
'pytest-randomly',
'pytest-instafail',
'pytest-rerunfailures',
'ptvsd'
]
},
long_description=open("README.md").read(),
long_description_content_type="text/markdown",
classifiers=CLASSIFIERS,
license="MIT",
packages=PACKAGES,
install_requires=INSTALL_REQUIRES,
extras_require=EXTRA_REQUIRES,
include_package_data=True,
cmdclass={
'develop': develop,
'build': build,
'webhost': webhost,
'extension': extension
},
cmdclass=COMMAND_CLASS,
test_suite='tests'
)

Просмотреть файл

@ -9,7 +9,7 @@
{
"type": "blob",
"direction": "in",
"name": "input_file_1",
"name": "inputfile1",
"dataType": "binary",
"connection": "AzureWebJobsStorage",
"path": "python-worker-tests/shmem-test-bytes-1.txt"
@ -17,7 +17,7 @@
{
"type": "blob",
"direction": "in",
"name": "input_file_2",
"name": "inputfile2",
"dataType": "binary",
"connection": "AzureWebJobsStorage",
"path": "python-worker-tests/shmem-test-bytes-2.txt"
@ -25,7 +25,7 @@
{
"type": "blob",
"direction": "out",
"name": "output_file_1",
"name": "outputfile1",
"dataType": "binary",
"connection": "AzureWebJobsStorage",
"path": "python-worker-tests/shmem-test-bytes-out-1.txt"
@ -33,7 +33,7 @@
{
"type": "blob",
"direction": "out",
"name": "output_file_2",
"name": "outputfile2",
"dataType": "binary",
"connection": "AzureWebJobsStorage",
"path": "python-worker-tests/shmem-test-bytes-out-2.txt"

Просмотреть файл

@ -15,10 +15,10 @@ def _generate_content_and_digest(content_size):
def main(
req: azf.HttpRequest,
input_file_1: bytes,
input_file_2: bytes,
output_file_1: azf.Out[bytes],
output_file_2: azf.Out[bytes]) -> azf.HttpResponse:
inputfile1: bytes,
inputfile2: bytes,
outputfile1: azf.Out[bytes],
outputfile2: azf.Out[bytes]) -> azf.HttpResponse:
"""
Read two blobs (bytes) and respond back (in HTTP response) with the number
of bytes read from each blob and the MD5 digest of the content of each.
@ -26,11 +26,11 @@ def main(
bytes written in each blob and the MD5 digest of the content of each.
The number of bytes to write are specified in the input HTTP request.
"""
input_content_size_1 = len(input_file_1)
input_content_size_2 = len(input_file_2)
input_content_size_1 = len(inputfile1)
input_content_size_2 = len(inputfile2)
input_content_md5_1 = hashlib.md5(input_file_1).hexdigest()
input_content_md5_2 = hashlib.md5(input_file_2).hexdigest()
input_content_md5_1 = hashlib.md5(inputfile1).hexdigest()
input_content_md5_2 = hashlib.md5(inputfile2).hexdigest()
output_content_size_1 = int(req.params['output_content_size_1'])
output_content_size_2 = int(req.params['output_content_size_2'])
@ -40,8 +40,8 @@ def main(
output_content_2, output_content_md5_2 = \
_generate_content_and_digest(output_content_size_2)
output_file_1.set(output_content_1)
output_file_2.set(output_content_2)
outputfile1.set(output_content_1)
outputfile2.set(output_content_2)
response_dict = {
'input_content_size_1': input_content_size_1,

Просмотреть файл

@ -94,7 +94,7 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
r = self.webhost.request('GET', 'get_metadata_batch_triggered')
self.assertEqual(r.status_code, 200)
# Check metadata and events length, events should be batch processed
# Check metadata and events length, events should be batched processed
events = r.json()
self.assertIsInstance(events, list)
self.assertGreater(len(events), 1)
@ -104,7 +104,8 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
event = events[event_index]
# Check if the event is enqueued between start_time and end_time
enqueued_time = parser.isoparse(event['enqueued_time'])
enqueued_time = parser.isoparse(event['enqueued_time']).astimezone(
tz=tz.UTC)
self.assertTrue(start_time < enqueued_time < end_time)
# Check if event properties are properly set
@ -120,7 +121,8 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
enqueued_time = parser.isoparse(sys_props['EnqueuedTimeUtc'])
# Check event trigger time and other system properties
self.assertTrue(start_time < enqueued_time < end_time)
self.assertTrue(
start_time.timestamp() < enqueued_time.timestamp() < end_time.timestamp()) # NoQA
self.assertIsNone(sys_props['PartitionKey'])
self.assertGreaterEqual(sys_props['SequenceNumber'], 0)
self.assertIsNotNone(sys_props['Offset'])

Просмотреть файл

@ -89,8 +89,11 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
self.assertIsNotNone(event['metadata'])
metadata = event['metadata']
sys_props = metadata['SystemProperties']
enqueued_time = parser.isoparse(metadata['EnqueuedTimeUtc'])
self.assertTrue(start_time < enqueued_time < end_time)
enqueued_time = parser.isoparse(metadata['EnqueuedTimeUtc']).astimezone(
tz=tz.UTC)
self.assertTrue(
start_time.timestamp() < enqueued_time.timestamp() < end_time.timestamp()) # NoQA
self.assertIsNone(sys_props['PartitionKey'])
self.assertGreaterEqual(sys_props['SequenceNumber'], 0)
self.assertIsNotNone(sys_props['Offset'])

Просмотреть файл

@ -76,7 +76,9 @@ class TestQueueFunctions(testutils.WebHostTestCase):
def test_queue_return_multiple(self):
r = self.webhost.request('POST', 'put_queue_return_multiple',
data='foo')
self.assertTrue(200 <= r.status_code < 300)
self.assertTrue(200 <= r.status_code < 300,
f"Returned status code {r.status_code}, "
"not in the 200-300 range.")
# wait for queue_trigger to process the queue item
time.sleep(1)
@ -85,5 +87,7 @@ class TestQueueFunctions(testutils.WebHostTestCase):
def test_queue_return_multiple_outparam(self):
r = self.webhost.request('POST', 'put_queue_multiple_out',
data='foo')
self.assertTrue(200 <= r.status_code < 300)
self.assertTrue(200 <= r.status_code < 300,
f"Returned status code {r.status_code}, "
"not in the 200-300 range.")
self.assertEqual(r.text, 'HTTP response: foo')

Просмотреть файл

@ -2,12 +2,18 @@
# Licensed under the MIT License.
import os
import sys
import unittest
from unittest import skipIf
from azure_functions_worker import testutils
from azure_functions_worker.bindings.shared_memory_data_transfer \
import SharedMemoryException
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
'shared memory filesystems and thus skipping'
' these tests for the time being')
class TestFileAccessor(testutils.SharedMemoryTestCase):
"""
Tests for FileAccessor.

Просмотреть файл

@ -2,7 +2,10 @@
# Licensed under the MIT License.
import os
import sys
import unittest
from unittest.mock import patch
from azure_functions_worker.bindings.shared_memory_data_transfer \
import FileAccessorFactory
from azure_functions_worker.bindings.\
@ -15,6 +18,15 @@ class TestFileAccessorFactory(unittest.TestCase):
"""
Tests for FileAccessorFactory.
"""
def setUp(self):
env = os.environ.copy()
env['FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED'] = "true"
self.mock_environ = patch.dict('os.environ', env)
self.mock_environ.start()
def tearDown(self):
self.mock_environ.stop()
@unittest.skipIf(os.name != 'nt',
'FileAccessorWindows is only valid on Windows')
def test_file_accessor_windows_created(self):
@ -24,7 +36,7 @@ class TestFileAccessorFactory(unittest.TestCase):
file_accessor = FileAccessorFactory.create_file_accessor()
self.assertTrue(type(file_accessor) is FileAccessorWindows)
@unittest.skipIf(os.name == 'nt',
@unittest.skipIf(os.name == 'nt' or sys.platform == 'darwin',
'FileAccessorUnix is only valid on Unix')
def test_file_accessor_unix_created(self):
"""

Просмотреть файл

@ -3,6 +3,10 @@
import json
import hashlib
import time
from unittest import skipIf
import sys
from azure_functions_worker.bindings.shared_memory_data_transfer \
import SharedMemoryMap
from azure_functions_worker.bindings.shared_memory_data_transfer \
@ -11,6 +15,9 @@ from azure_functions_worker import protos
from azure_functions_worker import testutils
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
'shared memory filesystems and thus skipping'
' these tests for the time being')
class TestMockBlobSharedMemoryFunctions(testutils.SharedMemoryTestCase,
testutils.AsyncTestCase):
"""
@ -440,14 +447,15 @@ class TestMockBlobSharedMemoryFunctions(testutils.SharedMemoryTestCase,
method='GET',
query=http_params))),
protos.ParameterBinding(
name='input_file_1',
name='inputfile1',
rpc_shared_memory=input_value_1
),
protos.ParameterBinding(
name='input_file_2',
name='inputfile2',
rpc_shared_memory=input_value_2
)
])
time.sleep(1)
# Dispose the shared memory map since the function is done using it
input_shared_mem_map_1.dispose()

Просмотреть файл

@ -5,6 +5,7 @@ import math
import os
import json
import sys
from unittest import skipIf
from unittest.mock import patch
from azure_functions_worker.utils.common import is_envvar_true
from azure.functions import meta as bind_meta
@ -17,12 +18,17 @@ from azure_functions_worker.constants \
import FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
'shared memory filesystems and thus skipping'
' these tests for the time being')
class TestSharedMemoryManager(testutils.SharedMemoryTestCase):
"""
Tests for SharedMemoryManager.
"""
def setUp(self):
self.mock_environ = patch.dict('os.environ', os.environ.copy())
env = os.environ.copy()
env['FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED'] = "true"
self.mock_environ = patch.dict('os.environ', env)
self.mock_sys_module = patch.dict('sys.modules', sys.modules.copy())
self.mock_sys_path = patch('sys.path', sys.path.copy())
self.mock_environ.start()

Просмотреть файл

@ -2,7 +2,10 @@
# Licensed under the MIT License.
import os
import sys
import unittest
from unittest import skipIf
from azure_functions_worker import testutils
from azure_functions_worker.bindings.shared_memory_data_transfer \
import SharedMemoryMap
@ -12,6 +15,9 @@ from azure_functions_worker.bindings.shared_memory_data_transfer \
import SharedMemoryException
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
'shared memory filesystems and thus skipping'
' these tests for the time being')
class TestSharedMemoryMap(testutils.SharedMemoryTestCase):
"""
Tests for SharedMemoryMap.

Просмотреть файл

@ -98,7 +98,7 @@ class TestDependencyManager(unittest.TestCase):
def test_initialize_in_windows_core_tools(self):
os.environ['AzureWebJobsScriptRoot'] = 'C:\\FunctionApp'
sys.path.extend([
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
'C:\\Users\\user\\AppData\\Roaming\\npm\\'
'node_modules\\azure-functions-core-tools\\bin\\'
'workers\\python\\3.6\\WINDOWS\\X64',
'C:\\FunctionApp\\.venv38\\lib\\site-packages',
@ -115,7 +115,7 @@ class TestDependencyManager(unittest.TestCase):
)
self.assertEqual(
DependencyManager.worker_deps_path,
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\node_modules\\'
'C:\\Users\\user\\AppData\\Roaming\\npm\\node_modules\\'
'azure-functions-core-tools\\bin\\workers\\python\\3.6\\WINDOWS'
'\\X64'
)
@ -183,12 +183,12 @@ class TestDependencyManager(unittest.TestCase):
def test_get_worker_deps_path_from_windows_core_tools(self):
# Test for Windows Core Tools Environment
sys.path.append('C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
sys.path.append('C:\\Users\\user\\AppData\\Roaming\\npm\\'
'node_modules\\azure-functions-core-tools\\bin\\'
'workers\\python\\3.6\\WINDOWS\\X64')
result = DependencyManager._get_worker_deps_path()
self.assertEqual(result,
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
'C:\\Users\\user\\AppData\\Roaming\\npm\\'
'node_modules\\azure-functions-core-tools\\bin\\'
'workers\\python\\3.6\\WINDOWS\\X64')
@ -267,7 +267,7 @@ class TestDependencyManager(unittest.TestCase):
def test_add_to_sys_path_allow_resolution_from_import_statement(self):
"""The standard Python import mechanism allows deriving a specific
module in a import statement, e.g.
module in an import statement, e.g.
from azure import functions # OK
"""