InnerEye-Inference/configure.py

70 строки
2.9 KiB
Python
Исходник Обычный вид История

2021-03-29 19:38:31 +03:00
import logging
import os
from pathlib import Path
from typing import Dict
from azureml.core import Workspace
from injector import singleton, Binder
from azure_config import AzureConfig
Pushing image to datastore and deleting after inference (#4) * Pushing image to datastore And immediately deleting 'local' copy (from the snapshot) * Deleting image file after bootstar script runs Not tested yet. * two spurious newlines * mypy spotted missing colon * better defaults * Fixing two mypy errors * Test prints Using AzureML Studio to work out is shapshot is affected by the deletion * Saves image to datastore * Including GUID in datastore path * WiP downloading images from datastore in bootstrap script * Overwriting image data zip * tidy up * Returning GUID and tidy-up * Added unit test of image data zip overwrite * Fixing API call to submit with ignored return * Swapping temp image folder name to config * Fixing documentation * Plumbing for datastore name * Swapping from default to named datastore * Fixing main.yml build env * Decreasing indentation by breaking loop https://github.com/microsoft/InnerEye-Inference/pull/4#discussion_r617584432 * pylint not used and commented out line deleted * Fixing flake8 warnings * Swapping imports back to single line After Anton's query on the PR I checked the .flake8 file and our maximum line length is 160 and so I do not need to break up these lines. * Hard coding not-secret settings in workflow As per Anton's request: https://github.com/microsoft/InnerEye-Inference/pull/4#discussion_r618144366 * /= consistency * removing 2 debug print statements * removed brackets * swapping to run.wait_for_completion * swapping to writetext * removing unnecessary initialization * Rationalising 'run' method: paths, arguments, and comments * removing debug print lines * Reverting line lengths I had assumed a max line length of 100, but it is 160 in the .flake8 configuration file so I have reverted the changes I had made to not-too-long lines, and fixed a few others for legibility * Swapping to required=True for params * Typo and unnecessary Path() spotted by Jonathan * Changing parameter to underscores from dashes To bring them into line with the rest of the InnerEye projects. They needed to be different when the unknown args were passed straight on to score.py but that is not how the arguments flow through anymore. * Line length fix in comment to kick off license/cla The 'license/cla' check in the build pipeline has stalled and there is not 'retry' buttong so I am hoping to kick it into action again with an almost vacuaous commit!
2021-04-26 17:31:23 +03:00
from configuration_constants import (API_AUTH_SECRET_ENVIRONMENT_VARIABLE, CLUSTER, WORKSPACE_NAME,
EXPERIMENT_NAME, RESOURCE_GROUP, SUBSCRIPTION_ID,
APPLICATION_ID, TENANT_ID, IMAGE_DATA_FOLDER, DATASTORE_NAME,
AZUREML_SERVICE_PRINCIPAL_SECRET_ENVIRONMENT_VARIABLE)
2021-03-29 19:38:31 +03:00
PROJECT_SECRETS_FILE = Path(__file__).resolve().parent / Path("set_environment.sh")
def read_secret_from_file(secret_name: str) -> str:
"""
Reads a bash file with exports and returns the variables and values as dict
:return: A dictionary with secrets, or None if the file does not exist.
"""
try:
secrets_file = PROJECT_SECRETS_FILE
d: Dict[str, str] = {}
for line in secrets_file.read_text().splitlines():
if line.startswith("#"): continue
parts = line.replace("export", "").strip().split("=", 1)
key = parts[0].strip().upper()
d[key] = parts[1].strip()
return d[secret_name]
except Exception as ex:
logging.error(f"Missing configuration '{secret_name}'")
raise ex
def get_environment_variable(environment_variable_name: str) -> str:
value = os.environ.get(environment_variable_name, None)
if value is None:
value = read_secret_from_file(environment_variable_name)
if value is None:
raise ValueError(environment_variable_name)
return value
# AUTHENTICATION SECRET
API_AUTH_SECRET = get_environment_variable(API_AUTH_SECRET_ENVIRONMENT_VARIABLE)
API_AUTH_SECRET_HEADER_NAME = "API_AUTH_SECRET"
def configure(binder: Binder) -> None:
azure_config = get_azure_config()
workspace = azure_config.get_workspace()
binder.bind(Workspace, to=workspace, scope=singleton)
binder.bind(AzureConfig, to=azure_config, scope=singleton)
def get_azure_config() -> AzureConfig:
return AzureConfig(cluster=get_environment_variable(CLUSTER),
workspace_name=get_environment_variable(WORKSPACE_NAME),
experiment_name=get_environment_variable(EXPERIMENT_NAME),
resource_group=get_environment_variable(RESOURCE_GROUP),
subscription_id=get_environment_variable(SUBSCRIPTION_ID),
application_id=get_environment_variable(APPLICATION_ID),
service_principal_secret=get_environment_variable(
AZUREML_SERVICE_PRINCIPAL_SECRET_ENVIRONMENT_VARIABLE),
Pushing image to datastore and deleting after inference (#4) * Pushing image to datastore And immediately deleting 'local' copy (from the snapshot) * Deleting image file after bootstar script runs Not tested yet. * two spurious newlines * mypy spotted missing colon * better defaults * Fixing two mypy errors * Test prints Using AzureML Studio to work out is shapshot is affected by the deletion * Saves image to datastore * Including GUID in datastore path * WiP downloading images from datastore in bootstrap script * Overwriting image data zip * tidy up * Returning GUID and tidy-up * Added unit test of image data zip overwrite * Fixing API call to submit with ignored return * Swapping temp image folder name to config * Fixing documentation * Plumbing for datastore name * Swapping from default to named datastore * Fixing main.yml build env * Decreasing indentation by breaking loop https://github.com/microsoft/InnerEye-Inference/pull/4#discussion_r617584432 * pylint not used and commented out line deleted * Fixing flake8 warnings * Swapping imports back to single line After Anton's query on the PR I checked the .flake8 file and our maximum line length is 160 and so I do not need to break up these lines. * Hard coding not-secret settings in workflow As per Anton's request: https://github.com/microsoft/InnerEye-Inference/pull/4#discussion_r618144366 * /= consistency * removing 2 debug print statements * removed brackets * swapping to run.wait_for_completion * swapping to writetext * removing unnecessary initialization * Rationalising 'run' method: paths, arguments, and comments * removing debug print lines * Reverting line lengths I had assumed a max line length of 100, but it is 160 in the .flake8 configuration file so I have reverted the changes I had made to not-too-long lines, and fixed a few others for legibility * Swapping to required=True for params * Typo and unnecessary Path() spotted by Jonathan * Changing parameter to underscores from dashes To bring them into line with the rest of the InnerEye projects. They needed to be different when the unknown args were passed straight on to score.py but that is not how the arguments flow through anymore. * Line length fix in comment to kick off license/cla The 'license/cla' check in the build pipeline has stalled and there is not 'retry' buttong so I am hoping to kick it into action again with an almost vacuaous commit!
2021-04-26 17:31:23 +03:00
tenant_id=get_environment_variable(TENANT_ID),
datastore_name=get_environment_variable(DATASTORE_NAME),
image_data_folder=get_environment_variable(IMAGE_DATA_FOLDER))