Setting up dev as V4 branch for PyWorker (#936)
* Setting up dev as V4 branch for PyWorker * Installing Dotnet 6 for AzurePipelines and GH Actions * Disabling SharedMem tests from Mac, and disabling shm when disabled. * Logging stdout errors, and adding proc kill for lingering processes * Updating extension version in csproj * Enable SharedMemory for Tests * Adding extensionBundle info to host.json template * Adding extension.csproj file for correctly loading extensions.dll * Tweaking csproj for e2e tests * Reintroducing Trace Logging * Adding some sleep time * Solving the timestamp timezone awareness issue * Fixing UTs and E2E tests, setup.py changes, and v4 pipeline changes * Formatting and Typo changes * Refactoring azure.pipelines.yml file.
This commit is contained in:
Родитель
3755e68eb8
Коммит
5e522fc610
6
.flake8
6
.flake8
|
@ -8,6 +8,8 @@ ignore = W503,E402,E731
|
|||
exclude = .git, __pycache__, build, dist, .eggs, .github, .local, docs/,
|
||||
Samples, azure_functions_worker/protos/,
|
||||
azure_functions_worker/_thirdparty/typing_inspect.py,
|
||||
tests/unittests/test_typing_inspect.py, .venv*, .env*, .vscode, venv*
|
||||
tests/unittests/test_typing_inspect.py,
|
||||
tests/unittests/broken_functions/syntax_error/main.py,
|
||||
.env*, .vscode, venv*, *.venv*,
|
||||
|
||||
max-line-length = 80
|
||||
max-line-length = 80
|
||||
|
|
|
@ -38,6 +38,10 @@ jobs:
|
|||
uses: actions/setup-dotnet@v1
|
||||
with:
|
||||
dotnet-version: '5.0.x'
|
||||
- name: Set up Dotnet 6.x
|
||||
uses: actions/setup-dotnet@v1
|
||||
with:
|
||||
dotnet-version: '6.x'
|
||||
- name: Install dependencies and the worker
|
||||
run: |
|
||||
retry() {
|
||||
|
|
|
@ -35,6 +35,10 @@ jobs:
|
|||
uses: actions/setup-dotnet@v1
|
||||
with:
|
||||
dotnet-version: '5.0.x'
|
||||
- name: Set up Dotnet 6.x
|
||||
uses: actions/setup-dotnet@v1
|
||||
with:
|
||||
dotnet-version: '6.x'
|
||||
- name: Install dependencies and the worker
|
||||
run: |
|
||||
retry() {
|
||||
|
|
|
@ -125,4 +125,9 @@ prof/
|
|||
|
||||
# E2E Integration Test Core Tools
|
||||
.ci/e2e_integration_test/Azure.Functions.Cli/
|
||||
.ci/e2e_integration_test/Azure.Functions.Cli*
|
||||
.ci/e2e_integration_test/Azure.Functions.Cli*
|
||||
|
||||
# Lingering test files
|
||||
tests/**/host.json
|
||||
tests/**/bin
|
||||
tests/**/extensions.csproj
|
||||
|
|
11
README.md
11
README.md
|
@ -5,7 +5,7 @@
|
|||
|master|[![Build Status](https://azfunc.visualstudio.com/Azure%20Functions/_apis/build/status/Azure.azure-functions-python-worker?branchName=master)](https://azfunc.visualstudio.com/Azure%20Functions/_build/latest?definitionId=57&branchName=master)|[![codecov](https://codecov.io/gh/Azure/azure-functions-python-worker/branch/master/graph/badge.svg)](https://codecov.io/gh/Azure/azure-functions-python-worker)|![CI Unit tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20Unit%20tests/badge.svg?branch=master)|![CI E2E tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20E2E%20tests/badge.svg?branch=master)
|
||||
|dev|[![Build Status](https://azfunc.visualstudio.com/Azure%20Functions/_apis/build/status/Azure.azure-functions-python-worker?branchName=dev)](https://azfunc.visualstudio.com/Azure%20Functions/_build/latest?definitionId=57&branchName=dev)|[![codecov](https://codecov.io/gh/Azure/azure-functions-python-worker/branch/dev/graph/badge.svg)](https://codecov.io/gh/Azure/azure-functions-python-worker)|![CI Unit tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20Unit%20tests/badge.svg?branch=dev)|![CI E2E tests](https://github.com/Azure/azure-functions-python-worker/workflows/CI%20E2E%20tests/badge.svg?branch=dev)
|
||||
|
||||
Python support for Azure Functions is based on Python 3.6, Python 3.7, and Python 3.8, serverless hosting on Linux and the Functions 2.0 and 3.0 runtime.
|
||||
Python support for Azure Functions is based on Python 3.6, Python 3.7, Python 3.8, and Python 3.9 serverless hosting on Linux and the Functions 2.0, 3.0 and 4.0 runtime.
|
||||
|
||||
Here is the current status of Python in Azure Functions:
|
||||
|
||||
|
@ -14,9 +14,12 @@ What are the supported Python versions?
|
|||
|Azure Functions Runtime|Python 3.6|Python 3.7|Python 3.8|Python 3.9|
|
||||
|---|---|---|---|---|
|
||||
|Azure Functions 2.0|✔|✔|-|-|
|
||||
|Azure Functions 3.0|✔|✔|✔|(preview)|
|
||||
|Azure Functions 3.0|✔|✔|✔|✔|
|
||||
|Azure Functions 4.0|-|✔|✔|✔|
|
||||
|
||||
What's available?
|
||||
For information about Azure Functions Runtime, please refer to [Azure Functions runtime versions overview](https://docs.microsoft.com/en-us/azure/azure-functions/functions-versions) page.
|
||||
|
||||
### What's available?
|
||||
|
||||
- Build, test, debug and publish using Azure Functions Core Tools (CLI) or Visual Studio Code
|
||||
- Deploy Python Function project onto consumption, dedicated, or elastic premium plan.
|
||||
|
@ -28,7 +31,7 @@ What's coming?
|
|||
|
||||
- [Durable Functions For Python](https://github.com/Azure/azure-functions-durable-python)
|
||||
|
||||
# Get Started
|
||||
###Get Started
|
||||
|
||||
- [Create your first Python function](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-function-python)
|
||||
- [Developer guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python)
|
||||
|
|
|
@ -1,14 +1,17 @@
|
|||
name: $(Date:yyyyMMdd).$(Rev:r)
|
||||
|
||||
trigger:
|
||||
- release/2.*
|
||||
- release/3.*
|
||||
- release/4.*
|
||||
- dev
|
||||
|
||||
variables:
|
||||
DOTNET_VERSION: '3.1.405'
|
||||
DOTNET_VERSION_5: '5.0.x'
|
||||
DOTNET_VERSION_6: '6.x'
|
||||
patchBuildNumberForDev: $(Build.BuildNumber)
|
||||
PROD_V3_WORKER_PY : 'python/prodV3/worker.py'
|
||||
PROD_V4_WORKER_PY : 'python/prodV4/worker.py'
|
||||
|
||||
jobs:
|
||||
- job: Tests
|
||||
|
@ -18,14 +21,14 @@ jobs:
|
|||
- ImageOverride -equals MMSUbuntu20.04TLS
|
||||
strategy:
|
||||
matrix:
|
||||
Python36:
|
||||
pythonVersion: '3.6'
|
||||
Python37:
|
||||
pythonVersion: '3.7'
|
||||
Python38:
|
||||
pythonVersion: '3.8'
|
||||
Python39:
|
||||
pythonVersion: '3.9'
|
||||
Python310:
|
||||
pythonVersion: '3.10'
|
||||
steps:
|
||||
- task: UsePythonVersion@0
|
||||
inputs:
|
||||
|
@ -41,6 +44,11 @@ jobs:
|
|||
inputs:
|
||||
packageType: 'sdk'
|
||||
version: $(DOTNET_VERSION_5)
|
||||
- task: UseDotNet@2
|
||||
displayName: 'Install DotNet 6.x'
|
||||
inputs:
|
||||
packageType: 'sdk'
|
||||
version: $(DOTNET_VERSION_6)
|
||||
- task: ShellScript@2
|
||||
inputs:
|
||||
disableAutoCwd: true
|
||||
|
@ -55,24 +63,27 @@ jobs:
|
|||
- ImageOverride -equals MMS2019TLS
|
||||
strategy:
|
||||
matrix:
|
||||
Python36V2:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python37V2:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python36V3:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V3:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python38V3:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python39V3:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V4:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python38V4:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python39V4:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
steps:
|
||||
- template: pack/templates/win_env_gen.yml
|
||||
parameters:
|
||||
|
@ -88,18 +99,24 @@ jobs:
|
|||
- ImageOverride -equals MMS2019TLS
|
||||
strategy:
|
||||
matrix:
|
||||
Python37V2:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python37V3:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python38V3:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python39V3:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V4:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python38V4:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python39V4:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
steps:
|
||||
- template: pack/templates/win_env_gen.yml
|
||||
parameters:
|
||||
|
@ -115,24 +132,27 @@ jobs:
|
|||
- ImageOverride -equals MMSUbuntu20.04TLS
|
||||
strategy:
|
||||
matrix:
|
||||
Python36V2:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python37V2:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python36V3:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V3:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python38V3:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python39V3:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V4:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python38V4:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python39V4:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
steps:
|
||||
- template: pack/templates/nix_env_gen.yml
|
||||
parameters:
|
||||
|
@ -145,24 +165,27 @@ jobs:
|
|||
vmImage: 'macOS-10.15'
|
||||
strategy:
|
||||
matrix:
|
||||
Python36V2:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python37V2:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV2/worker.py'
|
||||
Python36V3:
|
||||
pythonVersion: '3.6'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V3:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python38V3:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python39V3:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: 'python/prodV3/worker.py'
|
||||
workerPath: $(PROD_V3_WORKER_PY)
|
||||
Python37V4:
|
||||
pythonVersion: '3.7'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python38V4:
|
||||
pythonVersion: '3.8'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
Python39V4:
|
||||
pythonVersion: '3.9'
|
||||
workerPath: $(PROD_V4_WORKER_PY)
|
||||
steps:
|
||||
- template: pack/templates/nix_env_gen.yml
|
||||
parameters:
|
||||
|
@ -174,8 +197,8 @@ jobs:
|
|||
dependsOn: ['Build_WINDOWS_X64', 'Build_WINDOWS_X86', 'Build_LINUX_X64', 'Build_OSX_X64']
|
||||
condition: or(startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'), eq(variables['Build.SourceBranch'], 'refs/heads/dev'))
|
||||
pool:
|
||||
#vmImage: 'vs2017-win2016'
|
||||
name: '1ES-Hosted-AzFunc' #MMS2019TLS for Windows2019 or MMSUbuntu20.04TLS for ubuntu
|
||||
# vmImage: 'vs2017-win2016'
|
||||
name: '1ES-Hosted-AzFunc' # MMS2019TLS for Windows2019 or MMSUbuntu20.04TLS for ubuntu
|
||||
demands:
|
||||
- ImageOverride -equals MMS2019TLS
|
||||
steps:
|
||||
|
@ -183,23 +206,22 @@ jobs:
|
|||
echo "Releasing from $BUILD_SOURCEBRANCHNAME"
|
||||
apt install jq
|
||||
|
||||
if [[ $BUILD_SOURCEBRANCHNAME = 2\.* ]]
|
||||
then
|
||||
echo "Generating V2 Release Package for $BUILD_SOURCEBRANCHNAME"
|
||||
NUSPEC="pack\Microsoft.Azure.Functions.V2.PythonWorker.nuspec"
|
||||
WKVERSION="$BUILD_SOURCEBRANCHNAME"
|
||||
elif [[ $BUILD_SOURCEBRANCHNAME = 3\.* ]]
|
||||
if [[ $BUILD_SOURCEBRANCHNAME = 3\.* ]]
|
||||
then
|
||||
echo "Generating V3 Release Package for $BUILD_SOURCEBRANCHNAME"
|
||||
NUSPEC="pack\Microsoft.Azure.Functions.V3.PythonWorker.nuspec"
|
||||
WKVERSION="$BUILD_SOURCEBRANCHNAME"
|
||||
elif [[ $BUILD_SOURCEBRANCHNAME = 4\.* ]]
|
||||
then
|
||||
echo "Generating V4 Release Package for $BUILD_SOURCEBRANCHNAME"
|
||||
NUSPEC="pack\Microsoft.Azure.Functions.V4.PythonWorker.nuspec"
|
||||
WKVERSION="$BUILD_SOURCEBRANCHNAME"
|
||||
elif [[ $BUILD_SOURCEBRANCHNAME = dev ]]
|
||||
then
|
||||
echo "Generating V3 Integration Test Package for $BUILD_SOURCEBRANCHNAME"
|
||||
LATEST_TAG=$(curl https://api.github.com/repos/Azure/azure-functions-python-worker/tags -s | jq '.[0].name' | sed 's/\"//g' | cut -d'.' -f-2)
|
||||
NUSPEC="pack\Microsoft.Azure.Functions.V3.PythonWorker.nuspec"
|
||||
# Only required for Integration Test. Version number contains date (e.g. 3.1.2.20211028-dev)
|
||||
WKVERSION="3.$LATEST_TAG-$(patchBuildNumberForDev)"
|
||||
echo "Generating V4 Integration Test Package for $BUILD_SOURCEBRANCHNAME"
|
||||
VERSION=$(cat azure_functions_worker/version.py | tail -1 | cut -d' ' -f3 | sed "s/'//g")
|
||||
NUSPEC="pack\Microsoft.Azure.Functions.V4.PythonWorker.nuspec"
|
||||
WKVERSION="$VERSION-$(patchBuildNumberForDev)"
|
||||
else
|
||||
echo "No Matching Release Tag For $BUILD_SOURCEBRANCH"
|
||||
fi
|
||||
|
@ -236,4 +258,4 @@ jobs:
|
|||
nuGetFeedType: 'internal'
|
||||
publishVstsFeed: 'e6a70c92-4128-439f-8012-382fe78d6396/f37f760c-aebd-443e-9714-ce725cd427df'
|
||||
allowPackageConflicts: true
|
||||
displayName: '[Integration Test] Push NuGet package to the AzureFunctionsPreRelease feed'
|
||||
displayName: '[Integration Test] Push NuGet package to the AzureFunctionsPreRelease feed'
|
||||
|
|
|
@ -1,4 +1,2 @@
|
|||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License.
|
||||
|
||||
__version__ = '1.2.6'
|
||||
|
|
|
@ -82,3 +82,16 @@ class FileAccessor(metaclass=ABCMeta):
|
|||
mem_map.write(consts.HeaderFlags.Initialized)
|
||||
# Seek back the memory map to the original position
|
||||
mem_map.seek(original_pos)
|
||||
|
||||
|
||||
class DummyFileAccessor(FileAccessor):
|
||||
def open_mem_map(self, mem_map_name: str, mem_map_size: int,
|
||||
access: int = mmap.ACCESS_READ) -> Optional[mmap.mmap]:
|
||||
pass
|
||||
|
||||
def create_mem_map(self, mem_map_name: str,
|
||||
mem_map_size: int) -> Optional[mmap.mmap]:
|
||||
pass
|
||||
|
||||
def delete_mem_map(self, mem_map_name: str, mem_map: mmap.mmap) -> bool:
|
||||
pass
|
||||
|
|
|
@ -2,8 +2,13 @@
|
|||
# Licensed under the MIT License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from .file_accessor import DummyFileAccessor
|
||||
from .file_accessor_unix import FileAccessorUnix
|
||||
from .file_accessor_windows import FileAccessorWindows
|
||||
from ...constants import FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED
|
||||
from ...utils.common import is_envvar_true
|
||||
|
||||
|
||||
class FileAccessorFactory:
|
||||
|
@ -13,6 +18,9 @@ class FileAccessorFactory:
|
|||
"""
|
||||
@staticmethod
|
||||
def create_file_accessor():
|
||||
if os.name == 'nt':
|
||||
if sys.platform == "darwin" and not is_envvar_true(
|
||||
FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED):
|
||||
return DummyFileAccessor()
|
||||
elif os.name == 'nt':
|
||||
return FileAccessorWindows()
|
||||
return FileAccessorUnix()
|
||||
|
|
|
@ -18,7 +18,6 @@ from typing import List, Optional
|
|||
|
||||
import grpc
|
||||
|
||||
from . import __version__
|
||||
from . import bindings
|
||||
from . import constants
|
||||
from . import functions
|
||||
|
@ -39,6 +38,8 @@ from .utils.common import get_app_setting, is_envvar_true
|
|||
from .utils.dependency import DependencyManager
|
||||
from .utils.tracing import marshall_exception_trace
|
||||
from .utils.wrappers import disable_feature_by
|
||||
from .version import VERSION
|
||||
|
||||
|
||||
_TRUE = "true"
|
||||
|
||||
|
@ -262,7 +263,7 @@ class Dispatcher(metaclass=DispatcherMeta):
|
|||
async def _handle__worker_init_request(self, req):
|
||||
logger.info('Received WorkerInitRequest, '
|
||||
'python version %s, worker version %s, request ID %s',
|
||||
sys.version, __version__, self.request_id)
|
||||
sys.version, VERSION, self.request_id)
|
||||
enable_debug_logging_recommendation()
|
||||
|
||||
worker_init_request = req.worker_init_request
|
||||
|
|
|
@ -71,26 +71,39 @@ LOCALHOST = "127.0.0.1"
|
|||
HOST_JSON_TEMPLATE = """\
|
||||
{
|
||||
"version": "2.0",
|
||||
"logging": {
|
||||
"logLevel": {
|
||||
"default": "Trace"
|
||||
}
|
||||
},
|
||||
"http": {
|
||||
"routePrefix": "api"
|
||||
},
|
||||
"swagger": {
|
||||
"enabled": true
|
||||
},
|
||||
"eventHub": {
|
||||
"maxBatchSize": 1000,
|
||||
"prefetchCount": 1000,
|
||||
"batchCheckpointFrequency": 1
|
||||
},
|
||||
"functionTimeout": "00:05:00"
|
||||
"logging": {"logLevel": {"default": "Trace"}}
|
||||
}
|
||||
"""
|
||||
|
||||
EXTENSION_CSPROJ_TEMPLATE = """\
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>netcoreapp3.1</TargetFramework>
|
||||
<WarningsAsErrors></WarningsAsErrors>
|
||||
<DefaultItemExcludes>**</DefaultItemExcludes>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
|
||||
Version="5.0.0" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
|
||||
Version="3.1.0" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
|
||||
Version="3.0.10" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage"
|
||||
Version="4.0.5" />
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.Storage.Blobs"
|
||||
Version="5.0.0" />
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues"
|
||||
Version="5.0.0" />
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator"
|
||||
Version="1.1.3" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
"""
|
||||
|
||||
SECRETS_TEMPLATE = """\
|
||||
{
|
||||
"masterKey": {
|
||||
|
@ -246,7 +259,8 @@ class WebHostTestCase(unittest.TestCase, metaclass=WebHostTestCaseMeta):
|
|||
self.host_stdout.seek(last_pos)
|
||||
self.host_out = self.host_stdout.read()
|
||||
self.host_stdout_logger.error(
|
||||
f'Captured WebHost stdout:\n{self.host_out}')
|
||||
f'Captured WebHost stdout from {self.host_stdout.name} '
|
||||
f':\n{self.host_out}')
|
||||
finally:
|
||||
if test_exception is not None:
|
||||
raise test_exception
|
||||
|
@ -731,7 +745,10 @@ class _WebHostProxy:
|
|||
self._proc.stderr.close()
|
||||
|
||||
self._proc.terminate()
|
||||
self._proc.wait()
|
||||
try:
|
||||
self._proc.wait(20)
|
||||
except subprocess.TimeoutExpired:
|
||||
self._proc.kill()
|
||||
|
||||
|
||||
def _find_open_port():
|
||||
|
@ -882,7 +899,8 @@ def start_webhost(*, script_dir=None, stdout=None):
|
|||
|
||||
addr = f'http://{LOCALHOST}:{port}'
|
||||
health_check_endpoint = f'{addr}/api/ping'
|
||||
for _ in range(10):
|
||||
host_out = stdout.readlines(100)
|
||||
for _ in range(5):
|
||||
try:
|
||||
r = requests.get(health_check_endpoint,
|
||||
params={'code': 'testFunctionKey'})
|
||||
|
@ -891,13 +909,14 @@ def start_webhost(*, script_dir=None, stdout=None):
|
|||
|
||||
if 200 <= r.status_code < 300:
|
||||
# Give the host a bit more time to settle
|
||||
time.sleep(2)
|
||||
time.sleep(1)
|
||||
break
|
||||
else:
|
||||
print(f'Failed to ping {health_check_endpoint}', flush=True)
|
||||
print(f'Failed to ping {health_check_endpoint}, status code: '
|
||||
f'{r.status_code}', flush=True)
|
||||
except requests.exceptions.ConnectionError:
|
||||
pass
|
||||
time.sleep(2)
|
||||
time.sleep(1)
|
||||
else:
|
||||
proc.terminate()
|
||||
try:
|
||||
|
@ -905,7 +924,8 @@ def start_webhost(*, script_dir=None, stdout=None):
|
|||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
raise RuntimeError('could not start the webworker in time. Please'
|
||||
f' check the log file for details: {stdout.name} ')
|
||||
f' check the log file for details: {stdout.name} \n'
|
||||
f' Captured WebHost stdout:\n{host_out}')
|
||||
|
||||
return _WebHostProxy(proc, addr)
|
||||
|
||||
|
@ -963,11 +983,16 @@ def _setup_func_app(app_root):
|
|||
extensions = app_root / 'bin'
|
||||
ping_func = app_root / 'ping'
|
||||
host_json = app_root / 'host.json'
|
||||
extensions_csproj_file = app_root / 'extensions.csproj'
|
||||
|
||||
if not os.path.isfile(host_json):
|
||||
with open(host_json, 'w') as f:
|
||||
f.write(HOST_JSON_TEMPLATE)
|
||||
|
||||
if not os.path.isfile(extensions_csproj_file):
|
||||
with open(extensions_csproj_file, 'w') as f:
|
||||
f.write(EXTENSION_CSPROJ_TEMPLATE)
|
||||
|
||||
_symlink_dir(TESTS_ROOT / 'common' / 'ping', ping_func)
|
||||
_symlink_dir(EXTENSIONS_PATH, extensions)
|
||||
|
||||
|
@ -976,8 +1001,11 @@ def _teardown_func_app(app_root):
|
|||
extensions = app_root / 'bin'
|
||||
ping_func = app_root / 'ping'
|
||||
host_json = app_root / 'host.json'
|
||||
extensions_csproj_file = app_root / 'extensions.csproj'
|
||||
extensions_obj_file = app_root / 'obj'
|
||||
|
||||
for path in (extensions, ping_func, host_json):
|
||||
for path in (extensions, ping_func, host_json, extensions_csproj_file,
|
||||
extensions_obj_file):
|
||||
remove_path(path)
|
||||
|
||||
|
||||
|
|
|
@ -0,0 +1,4 @@
|
|||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License.
|
||||
|
||||
VERSION = '4.0.0'
|
|
@ -0,0 +1,28 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<package xmlns="http://schemas.microsoft.com/packaging/2012/06/nuspec.xsd">
|
||||
<metadata>
|
||||
<id>Microsoft.Azure.Functions.PythonWorker</id>
|
||||
<version>1.1.0</version>
|
||||
<authors>Microsoft</authors>
|
||||
<owners>Microsoft</owners>
|
||||
<requireLicenseAcceptance>false</requireLicenseAcceptance>
|
||||
<description>Microsoft Azure Functions Python Worker</description>
|
||||
<copyright>© .NET Foundation. All rights reserved.</copyright>
|
||||
</metadata>
|
||||
<files>
|
||||
<file src="..\3.7_WINDOWS_X64\**" target="tools\3.7\WINDOWS\X64" />
|
||||
<file src="..\3.7_WINDOWS_X86\**" target="tools\3.7\WINDOWS\X86" />
|
||||
<file src="..\3.7_LINUX_X64\**" target="tools\3.7\LINUX\X64" />
|
||||
<file src="..\3.7_OSX_X64\**" target="tools\3.7\OSX\X64" />
|
||||
<file src="..\3.8_WINDOWS_X64\**" target="tools\3.8\WINDOWS\X64" />
|
||||
<file src="..\3.8_WINDOWS_X86\**" target="tools\3.8\WINDOWS\X86" />
|
||||
<file src="..\3.8_LINUX_X64\**" target="tools\3.8\LINUX\X64" />
|
||||
<file src="..\3.8_OSX_X64\**" target="tools\3.8\OSX\X64" />
|
||||
<file src="..\3.9_WINDOWS_X64\**" target="tools\3.9\WINDOWS\X64" />
|
||||
<file src="..\3.9_WINDOWS_X86\**" target="tools\3.9\WINDOWS\X86" />
|
||||
<file src="..\3.9_LINUX_X64\**" target="tools\3.9\LINUX\X64" />
|
||||
<file src="..\3.9_OSX_X64\**" target="tools\3.9\OSX\X64" />
|
||||
<file src="..\python\prodV3\worker.config.json" target="tools" />
|
||||
<file src="Microsoft.Azure.Functions.PythonWorker.targets" target="build" />
|
||||
</files>
|
||||
</package>
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"description":{
|
||||
"language":"python",
|
||||
"defaultRuntimeVersion":"3.9",
|
||||
"supportedOperatingSystems":["LINUX", "OSX", "WINDOWS"],
|
||||
"supportedRuntimeVersions":["3.7", "3.8", "3.9"],
|
||||
"supportedArchitectures":["X64", "X86"],
|
||||
"extensions":[".py"],
|
||||
"defaultExecutablePath":"python",
|
||||
"defaultWorkerPath":"%FUNCTIONS_WORKER_RUNTIME_VERSION%/{os}/{architecture}/worker.py"
|
||||
}
|
||||
}
|
|
@ -0,0 +1,74 @@
|
|||
import os
|
||||
import sys
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
# User packages
|
||||
PKGS_PATH = "site/wwwroot/.python_packages"
|
||||
VENV_PKGS_PATH = "site/wwwroot/worker_venv"
|
||||
|
||||
PKGS = "lib/site-packages"
|
||||
|
||||
# Azure environment variables
|
||||
AZURE_WEBSITE_INSTANCE_ID = "WEBSITE_INSTANCE_ID"
|
||||
AZURE_CONTAINER_NAME = "CONTAINER_NAME"
|
||||
AZURE_WEBJOBS_SCRIPT_ROOT = "AzureWebJobsScriptRoot"
|
||||
|
||||
|
||||
def is_azure_environment():
|
||||
"""Check if the function app is running on the cloud"""
|
||||
return (AZURE_CONTAINER_NAME in os.environ
|
||||
or AZURE_WEBSITE_INSTANCE_ID in os.environ)
|
||||
|
||||
|
||||
def add_script_root_to_sys_path():
|
||||
"""Append function project root to module finding sys.path"""
|
||||
functions_script_root = os.getenv(AZURE_WEBJOBS_SCRIPT_ROOT)
|
||||
if functions_script_root is not None:
|
||||
sys.path.append(functions_script_root)
|
||||
|
||||
|
||||
def determine_user_pkg_paths():
|
||||
"""This finds the user packages when function apps are running on the cloud
|
||||
|
||||
For Python 3.7+, we only accept:
|
||||
/home/site/wwwroot/.python_packages/lib/site-packages
|
||||
"""
|
||||
minor_version = sys.version_info[1]
|
||||
|
||||
home = Path.home()
|
||||
pkgs_path = os.path.join(home, PKGS_PATH)
|
||||
user_pkg_paths = []
|
||||
|
||||
if minor_version in (7, 8, 9):
|
||||
user_pkg_paths.append(os.path.join(pkgs_path, PKGS))
|
||||
else:
|
||||
raise RuntimeError(f'Unsupported Python version: 3.{minor_version}')
|
||||
|
||||
return user_pkg_paths
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# worker.py lives in the same directory as azure_functions_worker
|
||||
func_worker_dir = str(Path(__file__).absolute().parent)
|
||||
env = os.environ
|
||||
|
||||
if is_azure_environment():
|
||||
user_pkg_paths = determine_user_pkg_paths()
|
||||
joined_pkg_paths = os.pathsep.join(user_pkg_paths)
|
||||
|
||||
# On cloud, we prioritize third-party user packages
|
||||
# over worker packages in PYTHONPATH
|
||||
env['PYTHONPATH'] = f'{joined_pkg_paths}:{func_worker_dir}'
|
||||
os.execve(sys.executable,
|
||||
[sys.executable, '-m', 'azure_functions_worker']
|
||||
+ sys.argv[1:],
|
||||
env)
|
||||
else:
|
||||
# On local development, we prioritize worker packages over
|
||||
# third-party user packages (in .venv)
|
||||
sys.path.insert(1, func_worker_dir)
|
||||
add_script_root_to_sys_path()
|
||||
from azure_functions_worker import main
|
||||
|
||||
main.main()
|
312
setup.py
312
setup.py
|
@ -1,84 +1,138 @@
|
|||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License.
|
||||
|
||||
import distutils.cmd
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import json
|
||||
import tempfile
|
||||
import urllib.request
|
||||
import zipfile
|
||||
import re
|
||||
from distutils import dir_util
|
||||
from distutils.command import build
|
||||
from distutils.dist import Distribution
|
||||
|
||||
from setuptools import setup
|
||||
from setuptools.command import develop
|
||||
|
||||
from azure_functions_worker import __version__
|
||||
from azure_functions_worker.version import VERSION
|
||||
|
||||
# The GitHub repository of the Azure Functions Host
|
||||
WEBHOST_GITHUB_API = "https://api.github.com/repos/Azure/azure-functions-host"
|
||||
WEBHOST_TAG_PREFIX = "v3."
|
||||
WEBHOST_TAG_PREFIX = "v4."
|
||||
|
||||
# Extensions necessary for non-core bindings.
|
||||
AZURE_EXTENSIONS = """\
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>netcoreapp3.1</TargetFramework>
|
||||
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
|
||||
<WarningsAsErrors></WarningsAsErrors>
|
||||
<DefaultItemExcludes>**</DefaultItemExcludes>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference
|
||||
Include="Microsoft.NET.Sdk.Functions"
|
||||
Version="3.0.3"
|
||||
/>
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
|
||||
Version="3.0.5"
|
||||
/>
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
|
||||
Version="3.0.6"
|
||||
/>
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
|
||||
Version="2.1.0"
|
||||
/>
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Extensions.Storage"
|
||||
Version="3.0.10"
|
||||
/>
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.ServiceBus"
|
||||
Version="3.0.0-beta8"
|
||||
/>
|
||||
</ItemGroup>
|
||||
<PropertyGroup>
|
||||
<TargetFramework>netcoreapp3.1</TargetFramework>
|
||||
<AzureFunctionsVersion>v4</AzureFunctionsVersion>
|
||||
<WarningsAsErrors />
|
||||
<DefaultItemExcludes>**</DefaultItemExcludes>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Sdk.Functions"
|
||||
Version="4.0.1" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB"
|
||||
Version="3.0.10" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs"
|
||||
Version="5.0.0" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid"
|
||||
Version="3.1.0" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage"
|
||||
Version="4.0.5" />
|
||||
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus"
|
||||
Version="4.2.1" />
|
||||
<PackageReference
|
||||
Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator"
|
||||
Version="1.1.3" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
"""
|
||||
|
||||
|
||||
NUGET_CONFIG = """\
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<packageSources>
|
||||
<add key="nuget.org" value="https://www.nuget.org/api/v2/" />
|
||||
<add key="azure_app_service"
|
||||
value="https://www.myget.org/F/azure-appservice/api/v2" />
|
||||
<add key="azure_app_service_staging"
|
||||
value="https://www.myget.org/F/azure-appservice-staging/api/v2" />
|
||||
<add key="buildTools"
|
||||
value="https://www.myget.org/F/30de4ee06dd54956a82013fa17a3accb/" />
|
||||
<add key="AspNetVNext"
|
||||
value="https://www.myget.org/F/aspnetcore-dev/api/v3/index.json" />
|
||||
</packageSources>
|
||||
<packageSources>
|
||||
<add key="nuget.org"
|
||||
value="https://www.nuget.org/api/v2/" />
|
||||
<add key="azure_app_service"
|
||||
value="https://www.myget.org/F/azure-appservice/api/v2" />
|
||||
<add key="azure_app_service_staging"
|
||||
value="https://www.myget.org/F/azure-appservice-staging/api/v2" />
|
||||
<add key="buildTools"
|
||||
value="https://www.myget.org/F/30de4ee06dd54956a82013fa17a3accb/" />
|
||||
<add key="AspNetVNext"
|
||||
value="https://www.myget.org/F/aspnetcore-dev/api/v3/index.json" />
|
||||
</packageSources>
|
||||
</configuration>
|
||||
"""
|
||||
|
||||
|
||||
CLASSIFIERS = [
|
||||
"Development Status :: 5 - Production/Stable",
|
||||
'Programming Language :: Python',
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.7",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Operating System :: Microsoft :: Windows",
|
||||
"Operating System :: POSIX",
|
||||
"Operating System :: MacOS :: MacOS X",
|
||||
"Environment :: Web Environment",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Intended Audience :: Developers",
|
||||
]
|
||||
|
||||
|
||||
PACKAGES = [
|
||||
"azure_functions_worker",
|
||||
"azure_functions_worker.protos",
|
||||
"azure_functions_worker.protos.identity",
|
||||
"azure_functions_worker.protos.shared",
|
||||
"azure_functions_worker.bindings",
|
||||
"azure_functions_worker.bindings.shared_memory_data_transfer",
|
||||
"azure_functions_worker.utils",
|
||||
"azure_functions_worker._thirdparty"
|
||||
]
|
||||
|
||||
|
||||
INSTALL_REQUIRES = [
|
||||
"grpcio~=1.43.0",
|
||||
"grpcio-tools~=1.43.0",
|
||||
"protobuf~=3.19.3",
|
||||
"azure-functions==1.8.0"
|
||||
]
|
||||
|
||||
|
||||
EXTRA_REQUIRES = {
|
||||
"dev": [
|
||||
"azure-eventhub~=5.1.0",
|
||||
"python-dateutil~=2.8.1",
|
||||
"pycryptodome~=3.10.1",
|
||||
"flake8~=3.7.9",
|
||||
"mypy",
|
||||
"pytest",
|
||||
"requests==2.*",
|
||||
"coverage",
|
||||
"pytest-sugar",
|
||||
"pytest-cov",
|
||||
"pytest-xdist",
|
||||
"pytest-randomly",
|
||||
"pytest-instafail",
|
||||
"pytest-rerunfailures",
|
||||
"ptvsd"
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
class BuildGRPC:
|
||||
"""Generate gRPC bindings."""
|
||||
def _gen_grpc(self):
|
||||
|
@ -128,9 +182,10 @@ class BuildGRPC:
|
|||
# https://github.com/protocolbuffers/protobuf/issues/1491
|
||||
self.make_absolute_imports(compiled_files)
|
||||
|
||||
dir_util.copy_tree(built_protos_dir, str(proto_root_dir))
|
||||
dir_util.copy_tree(str(built_protos_dir), str(proto_root_dir))
|
||||
|
||||
def make_absolute_imports(self, compiled_files):
|
||||
@staticmethod
|
||||
def make_absolute_imports(compiled_files):
|
||||
for compiled in compiled_files:
|
||||
with open(compiled, 'r+') as f:
|
||||
content = f.read()
|
||||
|
@ -153,19 +208,19 @@ class BuildGRPC:
|
|||
f.truncate()
|
||||
|
||||
|
||||
class build(build.build, BuildGRPC):
|
||||
class BuildProtos(build.build, BuildGRPC):
|
||||
def run(self, *args, **kwargs):
|
||||
self._gen_grpc()
|
||||
super().run(*args, **kwargs)
|
||||
super().run()
|
||||
|
||||
|
||||
class develop(develop.develop, BuildGRPC):
|
||||
class Development(develop.develop, BuildGRPC):
|
||||
def run(self, *args, **kwargs):
|
||||
self._gen_grpc()
|
||||
super().run(*args, **kwargs)
|
||||
super().run()
|
||||
|
||||
|
||||
class extension(distutils.cmd.Command):
|
||||
class Extension(distutils.cmd.Command):
|
||||
description = (
|
||||
'Resolve WebJobs Extensions from AZURE_EXTENSIONS and NUGET_CONFIG.'
|
||||
)
|
||||
|
@ -174,9 +229,13 @@ class extension(distutils.cmd.Command):
|
|||
'A path to the directory where extension should be installed')
|
||||
]
|
||||
|
||||
def initialize_options(self):
|
||||
def __init__(self, dist: Distribution):
|
||||
super().__init__(dist)
|
||||
self.extensions_dir = None
|
||||
|
||||
def initialize_options(self):
|
||||
pass
|
||||
|
||||
def finalize_options(self):
|
||||
if self.extensions_dir is None:
|
||||
self.extensions_dir = \
|
||||
|
@ -204,7 +263,7 @@ class extension(distutils.cmd.Command):
|
|||
args=['dotnet', 'build', '-o', '.'], check=True,
|
||||
cwd=str(self.extensions_dir),
|
||||
stdout=sys.stdout, stderr=sys.stderr, env=env)
|
||||
except Exception:
|
||||
except Exception: # NoQA
|
||||
print(".NET Core SDK is required to build the extensions. "
|
||||
"Please visit https://aka.ms/dotnet-download")
|
||||
sys.exit(1)
|
||||
|
@ -213,7 +272,7 @@ class extension(distutils.cmd.Command):
|
|||
self._install_extensions()
|
||||
|
||||
|
||||
class webhost(distutils.cmd.Command):
|
||||
class Webhost(distutils.cmd.Command):
|
||||
description = 'Download and setup Azure Functions Web Host.'
|
||||
user_options = [
|
||||
('webhost-version', None,
|
||||
|
@ -222,9 +281,13 @@ class webhost(distutils.cmd.Command):
|
|||
'A path to the directory where Azure Web Host will be installed.'),
|
||||
]
|
||||
|
||||
def initialize_options(self):
|
||||
self.webhost_version = None
|
||||
def __init__(self, dist: Distribution):
|
||||
super().__init__(dist)
|
||||
self.webhost_dir = None
|
||||
self.webhost_version = None
|
||||
|
||||
def initialize_options(self):
|
||||
pass
|
||||
|
||||
def finalize_options(self):
|
||||
if self.webhost_version is None:
|
||||
|
@ -234,7 +297,8 @@ class webhost(distutils.cmd.Command):
|
|||
self.webhost_dir = \
|
||||
pathlib.Path(__file__).parent / 'build' / 'webhost'
|
||||
|
||||
def _get_webhost_version(self) -> str:
|
||||
@staticmethod
|
||||
def _get_webhost_version() -> str:
|
||||
# Return the latest matched version (e.g. 3.0.15278)
|
||||
github_api_url = f'{WEBHOST_GITHUB_API}/tags?page=1&per_page=10'
|
||||
print(f'Checking latest webhost version from {github_api_url}')
|
||||
|
@ -243,12 +307,13 @@ class webhost(distutils.cmd.Command):
|
|||
|
||||
# As tags are placed in time desending order, the latest v3
|
||||
# tag should be the first occurance starts with 'v3.' string
|
||||
latest_v3 = [
|
||||
latest = [
|
||||
gt for gt in tags if gt['name'].startswith(WEBHOST_TAG_PREFIX)
|
||||
]
|
||||
return latest_v3[0]['name'].replace('v', '')
|
||||
return latest[0]['name'].replace('v', '')
|
||||
|
||||
def _download_webhost_zip(self, version: str) -> str:
|
||||
@staticmethod
|
||||
def _download_webhost_zip(version: str) -> str:
|
||||
# Return the path of the downloaded host
|
||||
temporary_file = tempfile.NamedTemporaryFile()
|
||||
zip_url = (
|
||||
|
@ -269,13 +334,15 @@ class webhost(distutils.cmd.Command):
|
|||
print(f'Functions Host is downloaded into {temporary_file.name}')
|
||||
return temporary_file.name
|
||||
|
||||
def _create_webhost_folder(self, dest_folder: pathlib.Path):
|
||||
@staticmethod
|
||||
def _create_webhost_folder(dest_folder: pathlib.Path):
|
||||
if dest_folder.exists():
|
||||
shutil.rmtree(dest_folder)
|
||||
os.makedirs(dest_folder, exist_ok=True)
|
||||
print(f'Functions Host folder is created in {dest_folder}')
|
||||
|
||||
def _extract_webhost_zip(self, version: str, src_zip: str, dest: str):
|
||||
@staticmethod
|
||||
def _extract_webhost_zip(version: str, src_zip: str, dest: str):
|
||||
print(f'Extracting Functions Host from {src_zip}')
|
||||
|
||||
with zipfile.ZipFile(src_zip) as archive:
|
||||
|
@ -311,7 +378,8 @@ class webhost(distutils.cmd.Command):
|
|||
|
||||
print(f'Functions Host is extracted into {dest}')
|
||||
|
||||
def _chmod_protobuf_generation_script(self, webhost_dir: pathlib.Path):
|
||||
@staticmethod
|
||||
def _chmod_protobuf_generation_script(webhost_dir: pathlib.Path):
|
||||
# This script is needed to set to executable in order to build the
|
||||
# WebJobs.Script.Grpc project in Linux and MacOS
|
||||
script_path = (
|
||||
|
@ -321,7 +389,8 @@ class webhost(distutils.cmd.Command):
|
|||
print('Change generate_protos.sh script permission')
|
||||
os.chmod(script_path, 0o555)
|
||||
|
||||
def _compile_webhost(self, webhost_dir: pathlib.Path):
|
||||
@staticmethod
|
||||
def _compile_webhost(webhost_dir: pathlib.Path):
|
||||
print(f'Compiling Functions Host from {webhost_dir}')
|
||||
|
||||
try:
|
||||
|
@ -330,7 +399,7 @@ class webhost(distutils.cmd.Command):
|
|||
check=True,
|
||||
cwd=str(webhost_dir),
|
||||
stdout=sys.stdout, stderr=sys.stderr)
|
||||
except Exception:
|
||||
except Exception: # NoQA
|
||||
print(f"Failed to compile webhost in {webhost_dir}. "
|
||||
".NET Core SDK is required to build the solution. "
|
||||
"Please visit https://aka.ms/dotnet-download",
|
||||
|
@ -350,73 +419,54 @@ class webhost(distutils.cmd.Command):
|
|||
self._compile_webhost(self.webhost_dir)
|
||||
|
||||
|
||||
with open("README.md") as readme:
|
||||
long_description = readme.read()
|
||||
class Clean(distutils.cmd.Command):
|
||||
description = 'Clean up build generated files'
|
||||
user_options = []
|
||||
|
||||
def __init__(self, dist: Distribution):
|
||||
super().__init__(dist)
|
||||
self.dir_list_to_delete = [
|
||||
"build"
|
||||
]
|
||||
|
||||
def initialize_options(self) -> None:
|
||||
pass
|
||||
|
||||
def finalize_options(self) -> None:
|
||||
pass
|
||||
|
||||
def run(self) -> None:
|
||||
for dir_to_delete in self.dir_list_to_delete:
|
||||
dir_delete = pathlib.Path(dir_to_delete)
|
||||
if dir_delete.exists():
|
||||
dir_util.remove_tree(str(dir_delete))
|
||||
|
||||
|
||||
COMMAND_CLASS = {
|
||||
'develop': Development,
|
||||
'build': BuildProtos,
|
||||
'webhost': Webhost,
|
||||
'extension': Extension,
|
||||
'clean': Clean
|
||||
}
|
||||
|
||||
|
||||
setup(
|
||||
name='azure-functions-worker',
|
||||
version=__version__,
|
||||
description='Python Language Worker for Azure Functions Host',
|
||||
author="Microsoft Corp.",
|
||||
name="azure-functions-worker",
|
||||
version=VERSION,
|
||||
description="Python Language Worker for Azure Functions Host",
|
||||
author="Azure Functions team at Microsoft Corp.",
|
||||
author_email="azurefunctions@microsoft.com",
|
||||
keywords="azure azurefunctions python",
|
||||
keywords="azure functions azurefunctions python serverless",
|
||||
url="https://github.com/Azure/azure-functions-python-worker",
|
||||
long_description=long_description,
|
||||
long_description_content_type='text/markdown',
|
||||
classifiers=[
|
||||
'Development Status :: 5 - Production/Stable',
|
||||
'License :: OSI Approved :: MIT License',
|
||||
'Intended Audience :: Developers',
|
||||
'Programming Language :: Python :: 3',
|
||||
'Programming Language :: Python :: 3.6',
|
||||
'Programming Language :: Python :: 3.7',
|
||||
'Programming Language :: Python :: 3.8',
|
||||
'Programming Language :: Python :: 3.9',
|
||||
'Operating System :: Microsoft :: Windows',
|
||||
'Operating System :: POSIX',
|
||||
'Operating System :: MacOS :: MacOS X',
|
||||
'Environment :: Web Environment',
|
||||
],
|
||||
license='MIT',
|
||||
packages=['azure_functions_worker',
|
||||
'azure_functions_worker.protos',
|
||||
'azure_functions_worker.protos.identity',
|
||||
'azure_functions_worker.protos.shared',
|
||||
'azure_functions_worker.bindings',
|
||||
'azure_functions_worker.bindings.shared_memory_data_transfer',
|
||||
'azure_functions_worker.utils',
|
||||
'azure_functions_worker._thirdparty'],
|
||||
install_requires=[
|
||||
'grpcio~=1.33.2',
|
||||
'grpcio-tools~=1.33.2',
|
||||
],
|
||||
extras_require={
|
||||
'dev': [
|
||||
'azure-functions==1.8.0',
|
||||
'azure-eventhub~=5.1.0',
|
||||
'python-dateutil~=2.8.1',
|
||||
'pycryptodome~=3.10.1',
|
||||
'flake8~=3.7.9',
|
||||
'mypy',
|
||||
'pytest',
|
||||
'requests==2.*',
|
||||
'coverage',
|
||||
'pytest-sugar',
|
||||
'pytest-cov',
|
||||
'pytest-xdist',
|
||||
'pytest-randomly',
|
||||
'pytest-instafail',
|
||||
'pytest-rerunfailures',
|
||||
'ptvsd'
|
||||
]
|
||||
},
|
||||
long_description=open("README.md").read(),
|
||||
long_description_content_type="text/markdown",
|
||||
classifiers=CLASSIFIERS,
|
||||
license="MIT",
|
||||
packages=PACKAGES,
|
||||
install_requires=INSTALL_REQUIRES,
|
||||
extras_require=EXTRA_REQUIRES,
|
||||
include_package_data=True,
|
||||
cmdclass={
|
||||
'develop': develop,
|
||||
'build': build,
|
||||
'webhost': webhost,
|
||||
'extension': extension
|
||||
},
|
||||
cmdclass=COMMAND_CLASS,
|
||||
test_suite='tests'
|
||||
)
|
||||
|
|
|
@ -9,7 +9,7 @@
|
|||
{
|
||||
"type": "blob",
|
||||
"direction": "in",
|
||||
"name": "input_file_1",
|
||||
"name": "inputfile1",
|
||||
"dataType": "binary",
|
||||
"connection": "AzureWebJobsStorage",
|
||||
"path": "python-worker-tests/shmem-test-bytes-1.txt"
|
||||
|
@ -17,7 +17,7 @@
|
|||
{
|
||||
"type": "blob",
|
||||
"direction": "in",
|
||||
"name": "input_file_2",
|
||||
"name": "inputfile2",
|
||||
"dataType": "binary",
|
||||
"connection": "AzureWebJobsStorage",
|
||||
"path": "python-worker-tests/shmem-test-bytes-2.txt"
|
||||
|
@ -25,7 +25,7 @@
|
|||
{
|
||||
"type": "blob",
|
||||
"direction": "out",
|
||||
"name": "output_file_1",
|
||||
"name": "outputfile1",
|
||||
"dataType": "binary",
|
||||
"connection": "AzureWebJobsStorage",
|
||||
"path": "python-worker-tests/shmem-test-bytes-out-1.txt"
|
||||
|
@ -33,7 +33,7 @@
|
|||
{
|
||||
"type": "blob",
|
||||
"direction": "out",
|
||||
"name": "output_file_2",
|
||||
"name": "outputfile2",
|
||||
"dataType": "binary",
|
||||
"connection": "AzureWebJobsStorage",
|
||||
"path": "python-worker-tests/shmem-test-bytes-out-2.txt"
|
||||
|
|
|
@ -15,10 +15,10 @@ def _generate_content_and_digest(content_size):
|
|||
|
||||
def main(
|
||||
req: azf.HttpRequest,
|
||||
input_file_1: bytes,
|
||||
input_file_2: bytes,
|
||||
output_file_1: azf.Out[bytes],
|
||||
output_file_2: azf.Out[bytes]) -> azf.HttpResponse:
|
||||
inputfile1: bytes,
|
||||
inputfile2: bytes,
|
||||
outputfile1: azf.Out[bytes],
|
||||
outputfile2: azf.Out[bytes]) -> azf.HttpResponse:
|
||||
"""
|
||||
Read two blobs (bytes) and respond back (in HTTP response) with the number
|
||||
of bytes read from each blob and the MD5 digest of the content of each.
|
||||
|
@ -26,11 +26,11 @@ def main(
|
|||
bytes written in each blob and the MD5 digest of the content of each.
|
||||
The number of bytes to write are specified in the input HTTP request.
|
||||
"""
|
||||
input_content_size_1 = len(input_file_1)
|
||||
input_content_size_2 = len(input_file_2)
|
||||
input_content_size_1 = len(inputfile1)
|
||||
input_content_size_2 = len(inputfile2)
|
||||
|
||||
input_content_md5_1 = hashlib.md5(input_file_1).hexdigest()
|
||||
input_content_md5_2 = hashlib.md5(input_file_2).hexdigest()
|
||||
input_content_md5_1 = hashlib.md5(inputfile1).hexdigest()
|
||||
input_content_md5_2 = hashlib.md5(inputfile2).hexdigest()
|
||||
|
||||
output_content_size_1 = int(req.params['output_content_size_1'])
|
||||
output_content_size_2 = int(req.params['output_content_size_2'])
|
||||
|
@ -40,8 +40,8 @@ def main(
|
|||
output_content_2, output_content_md5_2 = \
|
||||
_generate_content_and_digest(output_content_size_2)
|
||||
|
||||
output_file_1.set(output_content_1)
|
||||
output_file_2.set(output_content_2)
|
||||
outputfile1.set(output_content_1)
|
||||
outputfile2.set(output_content_2)
|
||||
|
||||
response_dict = {
|
||||
'input_content_size_1': input_content_size_1,
|
||||
|
|
|
@ -94,7 +94,7 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
|
|||
r = self.webhost.request('GET', 'get_metadata_batch_triggered')
|
||||
self.assertEqual(r.status_code, 200)
|
||||
|
||||
# Check metadata and events length, events should be batch processed
|
||||
# Check metadata and events length, events should be batched processed
|
||||
events = r.json()
|
||||
self.assertIsInstance(events, list)
|
||||
self.assertGreater(len(events), 1)
|
||||
|
@ -104,7 +104,8 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
|
|||
event = events[event_index]
|
||||
|
||||
# Check if the event is enqueued between start_time and end_time
|
||||
enqueued_time = parser.isoparse(event['enqueued_time'])
|
||||
enqueued_time = parser.isoparse(event['enqueued_time']).astimezone(
|
||||
tz=tz.UTC)
|
||||
self.assertTrue(start_time < enqueued_time < end_time)
|
||||
|
||||
# Check if event properties are properly set
|
||||
|
@ -120,7 +121,8 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
|
|||
enqueued_time = parser.isoparse(sys_props['EnqueuedTimeUtc'])
|
||||
|
||||
# Check event trigger time and other system properties
|
||||
self.assertTrue(start_time < enqueued_time < end_time)
|
||||
self.assertTrue(
|
||||
start_time.timestamp() < enqueued_time.timestamp() < end_time.timestamp()) # NoQA
|
||||
self.assertIsNone(sys_props['PartitionKey'])
|
||||
self.assertGreaterEqual(sys_props['SequenceNumber'], 0)
|
||||
self.assertIsNotNone(sys_props['Offset'])
|
||||
|
|
|
@ -89,8 +89,11 @@ class TestEventHubFunctions(testutils.WebHostTestCase):
|
|||
self.assertIsNotNone(event['metadata'])
|
||||
metadata = event['metadata']
|
||||
sys_props = metadata['SystemProperties']
|
||||
enqueued_time = parser.isoparse(metadata['EnqueuedTimeUtc'])
|
||||
self.assertTrue(start_time < enqueued_time < end_time)
|
||||
enqueued_time = parser.isoparse(metadata['EnqueuedTimeUtc']).astimezone(
|
||||
tz=tz.UTC)
|
||||
|
||||
self.assertTrue(
|
||||
start_time.timestamp() < enqueued_time.timestamp() < end_time.timestamp()) # NoQA
|
||||
self.assertIsNone(sys_props['PartitionKey'])
|
||||
self.assertGreaterEqual(sys_props['SequenceNumber'], 0)
|
||||
self.assertIsNotNone(sys_props['Offset'])
|
||||
|
|
|
@ -76,7 +76,9 @@ class TestQueueFunctions(testutils.WebHostTestCase):
|
|||
def test_queue_return_multiple(self):
|
||||
r = self.webhost.request('POST', 'put_queue_return_multiple',
|
||||
data='foo')
|
||||
self.assertTrue(200 <= r.status_code < 300)
|
||||
self.assertTrue(200 <= r.status_code < 300,
|
||||
f"Returned status code {r.status_code}, "
|
||||
"not in the 200-300 range.")
|
||||
|
||||
# wait for queue_trigger to process the queue item
|
||||
time.sleep(1)
|
||||
|
@ -85,5 +87,7 @@ class TestQueueFunctions(testutils.WebHostTestCase):
|
|||
def test_queue_return_multiple_outparam(self):
|
||||
r = self.webhost.request('POST', 'put_queue_multiple_out',
|
||||
data='foo')
|
||||
self.assertTrue(200 <= r.status_code < 300)
|
||||
self.assertTrue(200 <= r.status_code < 300,
|
||||
f"Returned status code {r.status_code}, "
|
||||
"not in the 200-300 range.")
|
||||
self.assertEqual(r.text, 'HTTP response: foo')
|
||||
|
|
|
@ -2,12 +2,18 @@
|
|||
# Licensed under the MIT License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import skipIf
|
||||
|
||||
from azure_functions_worker import testutils
|
||||
from azure_functions_worker.bindings.shared_memory_data_transfer \
|
||||
import SharedMemoryException
|
||||
|
||||
|
||||
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
|
||||
'shared memory filesystems and thus skipping'
|
||||
' these tests for the time being')
|
||||
class TestFileAccessor(testutils.SharedMemoryTestCase):
|
||||
"""
|
||||
Tests for FileAccessor.
|
||||
|
|
|
@ -2,7 +2,10 @@
|
|||
# Licensed under the MIT License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from azure_functions_worker.bindings.shared_memory_data_transfer \
|
||||
import FileAccessorFactory
|
||||
from azure_functions_worker.bindings.\
|
||||
|
@ -15,6 +18,15 @@ class TestFileAccessorFactory(unittest.TestCase):
|
|||
"""
|
||||
Tests for FileAccessorFactory.
|
||||
"""
|
||||
def setUp(self):
|
||||
env = os.environ.copy()
|
||||
env['FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED'] = "true"
|
||||
self.mock_environ = patch.dict('os.environ', env)
|
||||
self.mock_environ.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.mock_environ.stop()
|
||||
|
||||
@unittest.skipIf(os.name != 'nt',
|
||||
'FileAccessorWindows is only valid on Windows')
|
||||
def test_file_accessor_windows_created(self):
|
||||
|
@ -24,7 +36,7 @@ class TestFileAccessorFactory(unittest.TestCase):
|
|||
file_accessor = FileAccessorFactory.create_file_accessor()
|
||||
self.assertTrue(type(file_accessor) is FileAccessorWindows)
|
||||
|
||||
@unittest.skipIf(os.name == 'nt',
|
||||
@unittest.skipIf(os.name == 'nt' or sys.platform == 'darwin',
|
||||
'FileAccessorUnix is only valid on Unix')
|
||||
def test_file_accessor_unix_created(self):
|
||||
"""
|
||||
|
|
|
@ -3,6 +3,10 @@
|
|||
|
||||
import json
|
||||
import hashlib
|
||||
import time
|
||||
from unittest import skipIf
|
||||
import sys
|
||||
|
||||
from azure_functions_worker.bindings.shared_memory_data_transfer \
|
||||
import SharedMemoryMap
|
||||
from azure_functions_worker.bindings.shared_memory_data_transfer \
|
||||
|
@ -11,6 +15,9 @@ from azure_functions_worker import protos
|
|||
from azure_functions_worker import testutils
|
||||
|
||||
|
||||
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
|
||||
'shared memory filesystems and thus skipping'
|
||||
' these tests for the time being')
|
||||
class TestMockBlobSharedMemoryFunctions(testutils.SharedMemoryTestCase,
|
||||
testutils.AsyncTestCase):
|
||||
"""
|
||||
|
@ -440,14 +447,15 @@ class TestMockBlobSharedMemoryFunctions(testutils.SharedMemoryTestCase,
|
|||
method='GET',
|
||||
query=http_params))),
|
||||
protos.ParameterBinding(
|
||||
name='input_file_1',
|
||||
name='inputfile1',
|
||||
rpc_shared_memory=input_value_1
|
||||
),
|
||||
protos.ParameterBinding(
|
||||
name='input_file_2',
|
||||
name='inputfile2',
|
||||
rpc_shared_memory=input_value_2
|
||||
)
|
||||
])
|
||||
time.sleep(1)
|
||||
|
||||
# Dispose the shared memory map since the function is done using it
|
||||
input_shared_mem_map_1.dispose()
|
||||
|
|
|
@ -5,6 +5,7 @@ import math
|
|||
import os
|
||||
import json
|
||||
import sys
|
||||
from unittest import skipIf
|
||||
from unittest.mock import patch
|
||||
from azure_functions_worker.utils.common import is_envvar_true
|
||||
from azure.functions import meta as bind_meta
|
||||
|
@ -17,12 +18,17 @@ from azure_functions_worker.constants \
|
|||
import FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED
|
||||
|
||||
|
||||
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
|
||||
'shared memory filesystems and thus skipping'
|
||||
' these tests for the time being')
|
||||
class TestSharedMemoryManager(testutils.SharedMemoryTestCase):
|
||||
"""
|
||||
Tests for SharedMemoryManager.
|
||||
"""
|
||||
def setUp(self):
|
||||
self.mock_environ = patch.dict('os.environ', os.environ.copy())
|
||||
env = os.environ.copy()
|
||||
env['FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED'] = "true"
|
||||
self.mock_environ = patch.dict('os.environ', env)
|
||||
self.mock_sys_module = patch.dict('sys.modules', sys.modules.copy())
|
||||
self.mock_sys_path = patch('sys.path', sys.path.copy())
|
||||
self.mock_environ.start()
|
||||
|
|
|
@ -2,7 +2,10 @@
|
|||
# Licensed under the MIT License.
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import skipIf
|
||||
|
||||
from azure_functions_worker import testutils
|
||||
from azure_functions_worker.bindings.shared_memory_data_transfer \
|
||||
import SharedMemoryMap
|
||||
|
@ -12,6 +15,9 @@ from azure_functions_worker.bindings.shared_memory_data_transfer \
|
|||
import SharedMemoryException
|
||||
|
||||
|
||||
@skipIf(sys.platform == 'darwin', 'MacOS M1 machines do not correctly test the'
|
||||
'shared memory filesystems and thus skipping'
|
||||
' these tests for the time being')
|
||||
class TestSharedMemoryMap(testutils.SharedMemoryTestCase):
|
||||
"""
|
||||
Tests for SharedMemoryMap.
|
||||
|
|
|
@ -98,7 +98,7 @@ class TestDependencyManager(unittest.TestCase):
|
|||
def test_initialize_in_windows_core_tools(self):
|
||||
os.environ['AzureWebJobsScriptRoot'] = 'C:\\FunctionApp'
|
||||
sys.path.extend([
|
||||
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
|
||||
'C:\\Users\\user\\AppData\\Roaming\\npm\\'
|
||||
'node_modules\\azure-functions-core-tools\\bin\\'
|
||||
'workers\\python\\3.6\\WINDOWS\\X64',
|
||||
'C:\\FunctionApp\\.venv38\\lib\\site-packages',
|
||||
|
@ -115,7 +115,7 @@ class TestDependencyManager(unittest.TestCase):
|
|||
)
|
||||
self.assertEqual(
|
||||
DependencyManager.worker_deps_path,
|
||||
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\node_modules\\'
|
||||
'C:\\Users\\user\\AppData\\Roaming\\npm\\node_modules\\'
|
||||
'azure-functions-core-tools\\bin\\workers\\python\\3.6\\WINDOWS'
|
||||
'\\X64'
|
||||
)
|
||||
|
@ -183,12 +183,12 @@ class TestDependencyManager(unittest.TestCase):
|
|||
|
||||
def test_get_worker_deps_path_from_windows_core_tools(self):
|
||||
# Test for Windows Core Tools Environment
|
||||
sys.path.append('C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
|
||||
sys.path.append('C:\\Users\\user\\AppData\\Roaming\\npm\\'
|
||||
'node_modules\\azure-functions-core-tools\\bin\\'
|
||||
'workers\\python\\3.6\\WINDOWS\\X64')
|
||||
result = DependencyManager._get_worker_deps_path()
|
||||
self.assertEqual(result,
|
||||
'C:\\Users\\hazeng\\AppData\\Roaming\\npm\\'
|
||||
'C:\\Users\\user\\AppData\\Roaming\\npm\\'
|
||||
'node_modules\\azure-functions-core-tools\\bin\\'
|
||||
'workers\\python\\3.6\\WINDOWS\\X64')
|
||||
|
||||
|
@ -267,7 +267,7 @@ class TestDependencyManager(unittest.TestCase):
|
|||
|
||||
def test_add_to_sys_path_allow_resolution_from_import_statement(self):
|
||||
"""The standard Python import mechanism allows deriving a specific
|
||||
module in a import statement, e.g.
|
||||
module in an import statement, e.g.
|
||||
|
||||
from azure import functions # OK
|
||||
"""
|
||||
|
|
Загрузка…
Ссылка в новой задаче