microsoft
/
az-deep-realtime-score
зеркало из https://github.com/microsoft/az-deep-realtime-score.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

add readme, remove cell output

This commit is contained in:
YanZhangADS 2018-12-20 16:20:55 +00:00
Родитель 993d28b533
Коммит e6ee0baa69
12 изменённых файлов: 833 добавлений и 3819 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

288
Keras_Tensorflow/.ipynb_checkpoints/00_AMLSetup-checkpoint.ipynb
Просмотреть файл

@ -11,7 +11,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -32,21 +32,13 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SDK Version: 1.0.2\n"
]
}
],
"outputs": [],
"source": [
"print(\"SDK Version:\", azureml.core.VERSION)"
]
@ -60,46 +52,6 @@
]
},
"outputs": [],
"source": [
"# how to upgrade AML?\n",
"# !pip install --upgrade azureml-sdk[notebooks,automl]"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33mTo sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AXTQDGVM3 to authenticate.\u001b[0m\n",
"CloudName IsDefault Name State TenantId\n",
"----------- ----------- -------------------------------------------- -------- ------------------------------------\n",
"AzureCloud False Solution Template Testing Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud True Boston Team Danielle Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Demo Subscription Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Energy Solution Accelerator Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Cosmos_WDG_Core_BnB_100348 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Agile-BI-Azure-Subscription-2 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Visual Studio Enterprise Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ACR-PROD(Converted to EA) Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Azure Cat E2E Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False CAT_Eng Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-WM-Ext Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-BLD Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Boston Team Ilan Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False AG-AzureCAT-AITao-Test-COGSNonProd-IO1685743 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Dev Dashboard Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Metrics Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n"
]
}
],
"source": [
"#Connect to your Azure Subscription\n",
"#!az login -o table\n",
@ -108,187 +60,13 @@
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"!az account set --subscription \"edf507a2-6235-46c5-b560-fd463ba2e771\""
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\r\n",
" \"authorization\": {\r\n",
" \"applicationId\": \"0736f41a-0425-4b46-bdb5-1563eff02385\",\r\n",
" \"managedByRoleDefinitionId\": \"91d00862-cf55-46a5-9dce-260bbd92ce25\",\r\n",
" \"roleDefinitionId\": \"376aa7d7-51a9-463d-bd4d-7e1691345612\"\r\n",
" },\r\n",
" \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/providers/Microsoft.MachineLearningServices\",\r\n",
" \"namespace\": \"Microsoft.MachineLearningServices\",\r\n",
" \"registrationState\": \"Registered\",\r\n",
" \"resourceTypes\": [\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"capabilities\": \"SystemAssignedResourceIdentity\",\r\n",
" \"defaultApiVersion\": \"2018-03-01-preview\",\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces/computes\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"operations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/computeOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/workspaceOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/usages\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/vmsizes\"\r\n",
" }\r\n",
" ]\r\n",
"}\r\n"
]
}
],
"outputs": [],
"source": [
"# register the new RP\n",
"!az provider register -n Microsoft.MachineLearningServices\n",
@ -303,7 +81,6 @@
"metadata": {},
"outputs": [],
"source": [
"subscription_id ='<YOUR_SUBSCRIPTION_ID>'\n",
"resource_group ='<YOUR_RESOURCE_GROUP>'\n",
"workspace_name = '<YOUR_WORKSPACE_NAME>'\n",
"workspace_region ='<YOUR_WORKSPACE_REGION>"
@ -318,7 +95,7 @@
},
{
"cell_type": "code",
"execution_count": 25,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -330,22 +107,10 @@
},
{
"cell_type": "code",
"execution_count": 26,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(True, 'workspace_region', 'eastus2')"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"set_key(env_path, 'subscription_id', subscription_id) # Replace YOUR_AZURE_SUBSCRIPTION\n",
"set_key(env_path, 'resource_group', resource_group)\n",
"set_key(env_path, 'workspace_name', workspace_name)\n",
"set_key(env_path, 'workspace_region', workspace_region)"
@ -394,46 +159,13 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: /workspace/Keras_Tensorflow/aml_config/config.json\n"
]
},
{
"data": {
"text/plain": [
"{'id': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.MachineLearningServices/workspaces/yanzamlworkspace',\n",
" 'name': 'yanzamlworkspace',\n",
" 'location': 'eastus2',\n",
" 'type': 'Microsoft.MachineLearningServices/workspaces',\n",
" 'workspaceid': '7eac6448-0351-4bc4-8e26-22611fa1fd00',\n",
" 'description': '',\n",
" 'friendlyName': 'yanzamlworkspace',\n",
" 'creationTime': '2018-11-27T14:29:41.2875058+00:00',\n",
" 'containerRegistry': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.containerregistry/registries/yanzamlwacrzllyuzdj',\n",
" 'keyVault': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.keyvault/vaults/yanzamlwkeyvaultaxqeyibo',\n",
" 'applicationInsights': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.insights/components/yanzamlwinsightsjotvcvib',\n",
" 'batchaiWorkspace': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.BatchAI/workspaces/yanzamlworkspace',\n",
" 'identityPrincipalId': 'a6041784-6aeb-4404-a289-2a53dd4f39b9',\n",
" 'identityTenantId': '72f988bf-86f1-41af-91ab-2d7cd011db47',\n",
" 'identityType': 'SystemAssigned',\n",
" 'storageAccount': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.storage/storageaccounts/yanzamlwstorageyotvwlmq'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"# load workspace configuratio from ./aml_config/config.json file.ß\n",
"my_workspace = Workspace.from_config()\n",

207
Keras_Tensorflow/.ipynb_checkpoints/01_DevelopModel-checkpoint.ipynb
Просмотреть файл

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

560
Keras_Tensorflow/.ipynb_checkpoints/03_BuildImage-checkpoint.ipynb
Просмотреть файл

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

288
Keras_Tensorflow/00_AMLSetup.ipynb
Просмотреть файл

@ -11,7 +11,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -32,21 +32,13 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SDK Version: 1.0.2\n"
]
}
],
"outputs": [],
"source": [
"print(\"SDK Version:\", azureml.core.VERSION)"
]
@ -60,46 +52,6 @@
]
},
"outputs": [],
"source": [
"# how to upgrade AML?\n",
"# !pip install --upgrade azureml-sdk[notebooks,automl]"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33mTo sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AXTQDGVM3 to authenticate.\u001b[0m\n",
"CloudName IsDefault Name State TenantId\n",
"----------- ----------- -------------------------------------------- -------- ------------------------------------\n",
"AzureCloud False Solution Template Testing Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud True Boston Team Danielle Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Demo Subscription Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Energy Solution Accelerator Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Cosmos_WDG_Core_BnB_100348 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Agile-BI-Azure-Subscription-2 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Visual Studio Enterprise Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ACR-PROD(Converted to EA) Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Azure Cat E2E Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False CAT_Eng Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-WM-Ext Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Core-ES-BLD Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False Boston Team Ilan Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False AG-AzureCAT-AITao-Test-COGSNonProd-IO1685743 Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Dev Dashboard Disabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n",
"AzureCloud False ADS Metrics Enabled 72f988bf-86f1-41af-91ab-2d7cd011db47\n"
]
}
],
"source": [
"#Connect to your Azure Subscription\n",
"#!az login -o table\n",
@ -108,187 +60,13 @@
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"!az account set --subscription \"edf507a2-6235-46c5-b560-fd463ba2e771\""
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\r\n",
" \"authorization\": {\r\n",
" \"applicationId\": \"0736f41a-0425-4b46-bdb5-1563eff02385\",\r\n",
" \"managedByRoleDefinitionId\": \"91d00862-cf55-46a5-9dce-260bbd92ce25\",\r\n",
" \"roleDefinitionId\": \"376aa7d7-51a9-463d-bd4d-7e1691345612\"\r\n",
" },\r\n",
" \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/providers/Microsoft.MachineLearningServices\",\r\n",
" \"namespace\": \"Microsoft.MachineLearningServices\",\r\n",
" \"registrationState\": \"Registered\",\r\n",
" \"resourceTypes\": [\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"capabilities\": \"SystemAssignedResourceIdentity\",\r\n",
" \"defaultApiVersion\": \"2018-03-01-preview\",\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"workspaces/computes\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"operations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US 2\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/computeOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\",\r\n",
" \"2018-03-01-preview\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/workspaceOperationsStatus\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/usages\"\r\n",
" },\r\n",
" {\r\n",
" \"aliases\": null,\r\n",
" \"apiVersions\": [\r\n",
" \"2018-11-19\"\r\n",
" ],\r\n",
" \"locations\": [\r\n",
" \"East US\",\r\n",
" \"Australia East\",\r\n",
" \"East US 2\",\r\n",
" \"West US 2\",\r\n",
" \"West Central US\",\r\n",
" \"Southeast Asia\",\r\n",
" \"West Europe\",\r\n",
" \"South Central US\",\r\n",
" \"East US 2 EUAP\"\r\n",
" ],\r\n",
" \"properties\": null,\r\n",
" \"resourceType\": \"locations/vmsizes\"\r\n",
" }\r\n",
" ]\r\n",
"}\r\n"
]
}
],
"outputs": [],
"source": [
"# register the new RP\n",
"!az provider register -n Microsoft.MachineLearningServices\n",
@ -303,7 +81,6 @@
"metadata": {},
"outputs": [],
"source": [
"subscription_id ='<YOUR_SUBSCRIPTION_ID>'\n",
"resource_group ='<YOUR_RESOURCE_GROUP>'\n",
"workspace_name = '<YOUR_WORKSPACE_NAME>'\n",
"workspace_region ='<YOUR_WORKSPACE_REGION>"
@ -318,7 +95,7 @@
},
{
"cell_type": "code",
"execution_count": 25,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -330,22 +107,10 @@
},
{
"cell_type": "code",
"execution_count": 26,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(True, 'workspace_region', 'eastus2')"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"set_key(env_path, 'subscription_id', subscription_id) # Replace YOUR_AZURE_SUBSCRIPTION\n",
"set_key(env_path, 'resource_group', resource_group)\n",
"set_key(env_path, 'workspace_name', workspace_name)\n",
"set_key(env_path, 'workspace_region', workspace_region)"
@ -394,46 +159,13 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"tags": [
"stripout"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: /workspace/Keras_Tensorflow/aml_config/config.json\n"
]
},
{
"data": {
"text/plain": [
"{'id': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.MachineLearningServices/workspaces/yanzamlworkspace',\n",
" 'name': 'yanzamlworkspace',\n",
" 'location': 'eastus2',\n",
" 'type': 'Microsoft.MachineLearningServices/workspaces',\n",
" 'workspaceid': '7eac6448-0351-4bc4-8e26-22611fa1fd00',\n",
" 'description': '',\n",
" 'friendlyName': 'yanzamlworkspace',\n",
" 'creationTime': '2018-11-27T14:29:41.2875058+00:00',\n",
" 'containerRegistry': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.containerregistry/registries/yanzamlwacrzllyuzdj',\n",
" 'keyVault': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.keyvault/vaults/yanzamlwkeyvaultaxqeyibo',\n",
" 'applicationInsights': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.insights/components/yanzamlwinsightsjotvcvib',\n",
" 'batchaiWorkspace': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/yanzamlworkspace/providers/Microsoft.BatchAI/workspaces/yanzamlworkspace',\n",
" 'identityPrincipalId': 'a6041784-6aeb-4404-a289-2a53dd4f39b9',\n",
" 'identityTenantId': '72f988bf-86f1-41af-91ab-2d7cd011db47',\n",
" 'identityType': 'SystemAssigned',\n",
" 'storageAccount': '/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourcegroups/yanzamlworkspace/providers/microsoft.storage/storageaccounts/yanzamlwstorageyotvwlmq'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"# load workspace configuratio from ./aml_config/config.json file.ß\n",
"my_workspace = Workspace.from_config()\n",

223
Keras_Tensorflow/01_DevelopModel.ipynb
Просмотреть файл

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

1469
Keras_Tensorflow/03_BuildImage.ipynb
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

48
Keras_Tensorflow/04_DeployOnAKS.ipynb
Просмотреть файл

@ -101,21 +101,9 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: /data/home/mylogin/notebooks/yanzrepo/DistributedDeepLearning/AKSDeploymentTutorial_AML/Keras_Tensorflow/aml_config/config.json\n",
"yanzamlworkspace\n",
"yanzamlworkspace\n",
"eastus2\n",
"edf507a2-6235-46c5-b560-fd463ba2e771\n"
]
}
],
"outputs": [],
"source": [
"from azureml.core.workspace import Workspace\n",
"\n",
@ -199,7 +187,7 @@
"# Attach an existing AKS cluster\n",
"\n",
"#attach_config = AksCompute.attach_configuration(resource_group=ws.resource_group,\n",
" cluster_name='my-aks-gpu1b579211166')\n",
"# cluster_name='my-aks-gpu1b579211166')\n",
"#aks_target = ComputeTarget.attach(ws, aks_name, attach_config)\n",
"#aks_target.wait_for_completion(True)"
]
@ -220,9 +208,6 @@
"source": [
"#Deploy web service to AKS\n",
"#Set the web service configuration (using customized configuration)\n",
"#aks_config =AksWebservice.deploy_configuration(memory_gb=4.0, autoscale_enabled=False, num_replicas=1)\n",
"#aks_config =AksWebservice.deploy_configuration(cpu_cores=1, memory_gb=4.0, autoscale_enabled=False, num_replicas=1, enable_app_insights=True)\n",
"#aks_config = AksWebservice.deploy_configuration(memory_gb=2.0, enable_app_insights=True)\n",
"aks_config = AksWebservice.deploy_configuration(autoscale_enabled=False, num_replicas=1)"
]
},
@ -237,26 +222,6 @@
"image = ws.images[image_name]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ContainerImage(workspace=<azureml.core.workspace.Workspace object at 0x7fb2197845c0>, name=image1, id=image1:13, tags={'name': 'AKS', 'project': 'AML'}, properties={}, version=13)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"image"
]
},
{
"cell_type": "code",
"execution_count": 10,
@ -408,6 +373,13 @@
"# s = Webservice(ws, 'my-aks-service-1')\n",
"# s.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Having deplied web service succesfully, we can now move on to [Test Web app](05_TestWebApp.ipynb)."
]
}
],
"metadata": {

72
Keras_Tensorflow/05_TestWebApp.ipynb
Просмотреть файл

@ -49,21 +49,9 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: /data/home/mylogin/notebooks/yanzrepo/DistributedDeepLearning/AKSDeploymentTutorial_AML/Keras_Tensorflow/aml_config/config.json\n",
"yanzamlworkspace\n",
"yanzamlworkspace\n",
"eastus2\n",
"edf507a2-6235-46c5-b560-fd463ba2e771\n"
]
}
],
"outputs": [],
"source": [
"ws = Workspace.from_config()\n",
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep=\"\\n\")"
@ -78,24 +66,12 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'my-aks-service-1'"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"aks_service_name = get_key(env_path, 'aks_service_name')\n",
"aks_service = AksWebservice(ws, name=aks_service_name)\n",
"aks_service.name"
"aks_service = AksWebservice(ws, name=aks_service_name)"
]
},
{
@ -128,26 +104,6 @@
"api_key = aks_service.get_keys()[0]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'IfHMAwsUFAyMO2erQ7wmuBwMVDozG2Sj'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"api_key"
]
},
{
"cell_type": "markdown",
"metadata": {},
@ -234,24 +190,6 @@
"r.json()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[{'image': [['n02127052', 'lynx', '0.9816483'], ['n02128385', 'leopard', '0.0077441484'], ['n02123159', 'tiger_cat', '0.003686138']]}, 'Computed in 98.45 ms']\n"
]
}
],
"source": [
"resp = aks_service.run(input_data = jsonimg)\n",
"print(resp)"
]
},
{
"cell_type": "markdown",
"metadata": {},

1329
Keras_Tensorflow/06_SpeedTestWebApp.ipynb
Просмотреть файл

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

100
Keras_Tensorflow/07_TearDown.ipynb Normal file
Просмотреть файл

@ -0,0 +1,100 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tear it all down\n",
"Once you are done with your cluster you can use the following two commands to destroy it all."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext dotenv\n",
"%dotenv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you are done with your cluster you can use the following two commands to destroy it all. First, delete the application."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"deployment.apps \"azure-dl\" deleted\n",
"service \"azure-dl\" deleted\n"
]
}
],
"source": [
"!kubectl delete -f az-dl.json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, you delete the AKS cluster. This step may take a few minutes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!az aks delete -n $aks_name -g $resource_group -y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, you should delete the resource group. This also deletes the AKS cluster and can be used instead of the above command if the resource group is only used for this purpose."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!az group delete --name $resource_group -y"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda env:AKSDeployment]",
"language": "python",
"name": "conda-env-AKSDeployment-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

65
Keras_Tensorflow/README.md Normal file
Просмотреть файл

@ -0,0 +1,65 @@
# Deploy Deep Learning CNN on Kubernetes Cluster with GPUs - Keras
## Prerequisites
1. Linux(Ubuntu).
2. [Anaconda Python](https://www.anaconda.com/download)
3. [Docker](https://docs.docker.com/v17.12/install/linux/docker-ee/ubuntu) installed.
4. [Azure account](https://azure.microsoft.com).
The tutorial was developed on an [Azure Ubuntu
DSVM](https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro),
which addresses the first three prerequisites.
## Setup
To set up your environment to run these notebooks, please follow these steps. They setup the notebooks to use Docker and Azure seamlessly.
1. Create a _Linux_ DSVM (NC6 or above to use GPU).
2. Clone, fork, or download the zip file for this repository:
```
git clone https://github.com/Microsoft/AKSDeploymentTutorial_AML.git
```
3. Add your user to the docker group:
```
sudo usermod -aG docker $USER
```
4. Navigate to _./AKSDeploymentTutorial\_AML/Keras\_Tensorflow_ directory
5. Create the Python virtual environment using the tutorial_env.yml:
```
conda env create -f tutorial_env.yml
```
6. Activate the virtual environment:
```
source activate tutorial_env
```
7. Login to Azure:
```
az login
```
8. If you have more than one Azure subscription, select it:
```
az account set --subscription <Your Azure Subscription>
```
9. Start the Jupyter notebook server in the virtual environment:
```
jupyter notebook
```
## Steps
After following the setup instructions above, run the Jupyter notebooks in order starting with the first notebook.
## Cleaning up
To remove the conda environment created see [here](https://conda.io/docs/commands/env/conda-env-remove.html). The last Jupyter notebook also gives details on deleting Azure resources associated with this repository.
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repositories using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

3
Keras_Tensorflow/tutorial_env.yml
Просмотреть файл

@ -15,9 +15,6 @@ dependencies:
- aiohttp==3.3.2
- toolz==0.9.0
- tqdm==4.23.4
#- scikit-learn
#- zlib==1.2.11
#- libgcc=5.2.0
- tornado==4.5.3
- azure-cli==2.0.41
- tensorflow-gpu==1.10.0