microsoft
/
az-deep-realtime-score
зеркало из https://github.com/microsoft/az-deep-realtime-score.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

test nbtoolz

This commit is contained in:
YanZhangADS 2018-12-20 12:47:48 +00:00
Родитель 5472817c25
Коммит 0cc3ea7158
2 изменённых файлов: 353 добавлений и 43 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

388
Keras_Tensorflow/.ipynb_checkpoints/00_AMLSetup-checkpoint.ipynb
Просмотреть файл

@ -4,16 +4,61 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Do the pre-req setup\n",
"https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-environment#create-workspace-configuration-file https://github.com/Azure/ViennaDocs/blob/master/PrivatePreview/notebooks/00.configuration.ipynb\n",
"## Installation and configuration¶\n",
"\n",
"This notebook configures your library of notebooks to connect to an Azure Machine Learning Workspace. In this case, a library contains all of the notebooks in the current folder and any nested folders. You can configure this notebook to use an existing workspace or create a new workspace."
"This notebook configures the notebooks in this tutorial to connect to an Azure Machine Learning (AML) Workspace. You can use an existing workspace or create a new one."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import azureml.core\n",
"from azureml.core import Workspace\n",
"from dotenv import set_key, get_key, find_dotenv\n",
"from pathlib import Path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"If you have already completed the prerequisites and selected the correct Kernel for this notebook, the AML Python SDK is already installed. Let's check the AML SDK version."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"tags": [
"stripoutput"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SDK Version: 1.0.2\n"
]
}
],
"source": [
"print(\"SDK Version:\", azureml.core.VERSION)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"tags": [
"stripoutput"
]
},
"outputs": [],
"source": [
"# how to upgrade AML?\n",
@ -22,28 +67,224 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33mTo sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AXTQDGVM3 to authenticate.\u001b[0m\n",
"CloudName IsDefault Name State TenantId\n",
"----------- ----------- -------------------------------------------- -------- ------------------------------------\n",
"AzureCloud False Solution Template Testing Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud True Boston Team Danielle Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Demo Subscription Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Energy Solution Accelerator Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Cosmos_WDG_Core_BnB_100348 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Agile-BI-Azure-Subscription-2 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Visual Studio Enterprise Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ACR-PROD(Converted to EA) Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Azure Cat E2E Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False CAT_Eng Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-WM-Ext Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-BLD Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Boston Team Ilan Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False AG-AzureCAT-AITao-Test-COGSNonProd-IO1685743 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Dev Dashboard Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Metrics Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n"
]
}
],
"source": [
"#Connect to your Azure Subscription\n",
"!az login -o table"
"#!az login -o table\n",
"!az login --use-device-code -o table"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"!az account set --subscription \"<YOUR_SUBSCRIPTION>\""
"!az account set --subscription \"edf507a2-6235-46c5-b560-fd463ba2e771\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"execution_count": 7,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\r\n",
" \"authorization\": {\r\n",
" \"applicationId\": \"0736f41a-0425-4b46-bdb5-1563eff02385\",\r\n",
" \"managedByRoleDefinitionId\": \"91d00862-cf55-46a5-9dce-260bbd92ce25\",\r\n",
" \"roleDefinitionId\": \"376aa7d7-51a9-463d-bd4d-7e1691345612\"\r\n",
" },\r\n",
" \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/providers/Microsoft.MachineLearningServices\",\r\n",
" \"namespace\": \"Microsoft.MachineLearningServices\",\r\n",
" \"registrationState\": \"Registered\",\r\n",
" \"resourceTypes\": [\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"capabilities\": \"SystemAssignedResourceIdentity\",\r\n",
" \"defaultApiVersion\": \"2018-03-01-preview\",\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces/computes\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"operations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/computeOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/workspaceOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/usages\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/vmsizes\"\r\n",
" }\r\n",
" ]\r\n",
"}\r\n"
]
}
],
"source": [
"# register the new RP\n",
"!az provider register -n Microsoft.MachineLearningServices\n",
@ -54,36 +295,71 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SDK Version: 0.1.68\n"
]
}
],
"outputs": [],
"source": [
"import azureml.core\n",
"from azureml.core import Workspace\n",
"\n",
"print(\"SDK Version:\", azureml.core.VERSION)"
"subscription_id ='<YOUR_SUBSCRIPTION_ID>'\n",
"resource_group ='<YOUR_RESOURCE_GROUP>'\n",
"workspace_name = '<YOUR_WORKSPACE_NAME>'\n",
"workspace_region ='<YOUR_WORKSPACE_REGION>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create and initialize a dotenv file for storing parameters used in multiple notebooks."
]
},
{
"cell_type": "code",
"execution_count": 36,
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"#Initialize an Azure ML Workspace\n",
"subscription_id_query = !az account show --query id -o tsv\n",
"subscription_id = account_query.s\n",
"resource_group = \"yanzamlworkspace\" # e.g. myamlworkspace\n",
"workspace_name = \"yanzamlworkspace\" # e.g. myamlworkspace\n",
"workspace_region = \"eastus2\" # e.g. eastus2 or eastus2euap"
"env_path = find_dotenv()\n",
"if env_path == \"\":\n",
" Path(\".env\").touch()\n",
" env_path = find_dotenv()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(True, 'workspace_region', 'eastus2')"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"set_key(env_path, 'subscription_id', subscription_id) # Replace YOUR_AZURE_SUBSCRIPTION\n",
"set_key(env_path, 'resource_group', resource_group)\n",
"set_key(env_path, 'workspace_name', workspace_name)\n",
"set_key(env_path, 'workspace_region', workspace_region)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create the workspace\n",
"This cell will create an AML workspace for you in a subscription, provided you have the correct permissions.\n",
"This will fail when:\n",
"\n",
"1. You do not have permission to create a workspace in the resource group\n",
"2. You do not have permission to create a resource group if it's non-existing.\n",
"3. You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
"\n",
"If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources. If this cell succeeds, you're done configuring AML!"
]
},
{
@ -114,9 +390,46 @@
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"execution_count": 5,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: /workspace/Keras_Tensorflow/aml_config/config.json\n"
]
},
{
"data": {
"text/plain": [
"{'id': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.MachineLearningServices/workspaces/yanzamlworkspace',\n",
" 'name': 'yanzamlworkspace',\n",
" 'location': 'eastus2',\n",
" 'type': 'Microsoft.MachineLearningServices/workspaces',\n",
" 'workspaceid': '7eac6448-0351-4bc4-8e26-22611fa1fd00',\n",
" 'description': '',\n",
" 'friendlyName': 'yanzamlworkspace',\n",
" 'creationTime': '2018-11-27T14:29:41.2875058+00:00',\n",
" 'containerRegistry': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.containerregistry/registries/yanzamlwacrzllyuzdj',\n",
" 'keyVault': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.keyvault/vaults/yanzamlwkeyvaultaxqeyibo',\n",
" 'applicationInsights': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.insights/components/yanzamlwinsightsjotvcvib',\n",
" 'batchaiWorkspace': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.BatchAI/workspaces/yanzamlworkspace',\n",
" 'identityPrincipalId': 'a6041784-6aeb-4404-a289-2a53dd4f39b9',\n",
" 'identityTenantId': '72f988bf-86f1-41af-91ab-2d7cd011db47',\n",
" 'identityType': 'SystemAssigned',\n",
" 'storageAccount': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.storage/storageaccounts/yanzamlwstorageyotvwlmq'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# load workspace configuratio from ./aml_config/config.json file.ß\n",
"my_workspace = Workspace.from_config()\n",
@ -125,6 +438,7 @@
}
],
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
@ -140,7 +454,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
"version": "3.6.7"
}
},
"nbformat": 4,

8
Keras_Tensorflow/00_AMLSetup.ipynb
Просмотреть файл

@ -56,7 +56,7 @@
"execution_count": null,
"metadata": {
"tags": [
"stripout"
"stripoutput"
]
},
"outputs": [],
@ -68,11 +68,7 @@
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"stripout"
]
},
"metadata": {},
"outputs": [
{
"name": "stdout",