Updates notebooks
This commit is contained in:
MSalvaris
Родитель
f3ea8e9a20
Коммит
deb7facfa7
|
|
@ -0,0 +1,785 @@
|
||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"collapsed": true
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Deploy Web App on Azure Container Services (AKS)\n",
|
||||||
|
"In this notebook, we will set up an Azure Container Service which will be managed by Kubernetes. We will then take the Docker image we created earlier that contains our app and deploy it to the AKS cluster. Then, we will check everything is working by sending an image to it and getting it scored.\n",
|
||||||
|
" \n",
|
||||||
|
"The process is split into the following steps:\n",
|
||||||
|
"* [Define our resource names](#section1)\n",
|
||||||
|
"* [Login to Azure](#section2)\n",
|
||||||
|
"* [Create resource group and create AKS](#section3)\n",
|
||||||
|
"* [Connect to AKS](#section4)\n",
|
||||||
|
"* [Deploy our app](#section5)\n",
|
||||||
|
"\n",
|
||||||
|
"This guide assumes is designed to be run on linux and requires that the Azure CLI is installed."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 1,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import os\n",
|
||||||
|
"import json\n",
|
||||||
|
"from testing_utilities import write_json_to_file\n",
|
||||||
|
"%load_ext dotenv"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"<a id='section1'></a>\n",
|
||||||
|
"## Setup\n",
|
||||||
|
"Below are the various name definitions for the resources needed to setup AKS."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 6,
|
||||||
|
"metadata": {
|
||||||
|
"tags": [
|
||||||
|
"parameters"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"Appending to .env\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"%%writefile --append .env\n",
|
||||||
|
"# This cell is tagged `parameters`\n",
|
||||||
|
"# Please modify the values below as you see fit\n",
|
||||||
|
"\n",
|
||||||
|
"# If you have multiple subscriptions select the subscription you want to use \n",
|
||||||
|
"selected_subscription = \"Team Danielle Internal\"\n",
|
||||||
|
"\n",
|
||||||
|
"# Resource group, name and location for AKS cluster.\n",
|
||||||
|
"resource_group = \"mabouaks\" \n",
|
||||||
|
"aks_name = \"mabouaks\"\n",
|
||||||
|
"location = \"eastus\""
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 10,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"%dotenv -o\n",
|
||||||
|
"image_name = os.getenv('docker_login') + os.getenv('image_repo')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"<a id='section2'></a>\n",
|
||||||
|
"## Azure account login\n",
|
||||||
|
"If you are not already logged in to an Azure account, the command below will initiate a login. It will pop up a browser where you can select an Azure account."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 11,
|
||||||
|
"metadata": {
|
||||||
|
"scrolled": false
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"Name CloudName SubscriptionId State IsDefault\n",
|
||||||
|
"---------------------------------------------- ----------- ------------------------------------ ------- -----------\n",
|
||||||
|
"Boston DS Dev AzureCloud 0ca618d2-22a8-413a-96d0-0f1b531129c3 Enabled False\n",
|
||||||
|
"Solution Template Testing AzureCloud 3bcfa59c-82a0-44f9-ac08-b3479370bace Enabled False\n",
|
||||||
|
"DEMO - how RepDemo are you AzureCloud fe4d94f0-dc5b-4c09-9b85-863413b0192b Enabled False\n",
|
||||||
|
"Microsoft Azure Internal - Demos AzureCloud b3a823e7-7472-4111-becc-9a211a558a05 Enabled False\n",
|
||||||
|
"Azure Stack Diagnostics CI and Production VaaS AzureCloud a8183b2d-7a4c-45e9-8736-dac11b84ff14 Enabled False\n",
|
||||||
|
"Azure Cat E2E AzureCloud fc4ea3c9-1d30-4f18-b33b-7404e7da0123 Enabled False\n",
|
||||||
|
"Core-ES-BLD AzureCloud 54e18c35-3863-4a17-8e52-b5aa1e65847e Enabled False\n",
|
||||||
|
"Cosmos_WDG_Core_BnB_100348 AzureCloud dae41bd3-9db4-4b9b-943e-832b57cac828 Enabled False\n",
|
||||||
|
"CFPlatformBuild AzureCloud e326c6f8-2c46-450f-9706-a03b90f06a0f Enabled False\n",
|
||||||
|
"Data Wrangling Preview AzureCloud 215613ac-9dfb-488c-be46-c387e999b127 Enabled False\n",
|
||||||
|
"Microsoft Azure Internal - Mohamed AzureCloud cc53b927-25cb-43c2-a741-a50b97c46532 Enabled False\n",
|
||||||
|
"Boston-DS-Brandon-Dev AzureCloud e984a9db-1a27-4f54-98fc-282cf0dcda04 Enabled False\n",
|
||||||
|
"Team Danielle Internal AzureCloud edf507a2-6235-46c5-b560-fd463ba2e771 Enabled True\n",
|
||||||
|
"Boston Engineering AzureCloud bc4170f0-cc6e-49d2-ba65-bc00a7a4df6b Enabled False\n",
|
||||||
|
"Team TJ AzureCloud 0eccc365-be66-4b08-a242-3c6fdb53cb61 Enabled False\n",
|
||||||
|
"Azure Internal - TATK AzureCloud 872ff0da-188e-4461-8cf7-26e1c3e28ebb Enabled False\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"%%bash\n",
|
||||||
|
"list=`az account list -o table`\n",
|
||||||
|
"if [ \"$list\" == '[]' ] || [ \"$list\" == '' ]; then \n",
|
||||||
|
" az login -o table\n",
|
||||||
|
"else\n",
|
||||||
|
" az account list -o table \n",
|
||||||
|
"fi"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 12,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"\u001b[0m"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!az account set --subscription \"$selected_subscription\""
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 13,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"{\r\n",
|
||||||
|
" \"environmentName\": \"AzureCloud\",\r\n",
|
||||||
|
" \"id\": \"edf507a2-6235-46c5-b560-fd463ba2e771\",\r\n",
|
||||||
|
" \"isDefault\": true,\r\n",
|
||||||
|
" \"name\": \"Team Danielle Internal\",\r\n",
|
||||||
|
" \"state\": \"Enabled\",\r\n",
|
||||||
|
" \"tenantId\": \"72f988bf-86f1-41af-91ab-2d7cd011db47\",\r\n",
|
||||||
|
" \"user\": {\r\n",
|
||||||
|
" \"name\": \"mabou@microsoft.com\",\r\n",
|
||||||
|
" \"type\": \"user\"\r\n",
|
||||||
|
" }\r\n",
|
||||||
|
"}\r\n",
|
||||||
|
"\u001b[0m"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!az account show"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 14,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"\u001b[33mRegistering is still on-going. You can monitor using 'az provider show -n Microsoft.ContainerService'\u001b[0m\r\n",
|
||||||
|
"\u001b[0m"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!az provider register -n Microsoft.ContainerService"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 15,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"{\r\n",
|
||||||
|
" \"authorization\": {\r\n",
|
||||||
|
" \"applicationId\": \"7319c514-987d-4e9b-ac3d-d38c4f427f4c\",\r\n",
|
||||||
|
" \"managedByRoleDefinitionId\": \"8e3af657-a8ff-443c-a75c-2fe8c4bcb635\",\r\n",
|
||||||
|
" \"roleDefinitionId\": \"1b4a0c7f-2217-416f-acfa-cf73452fdc1c\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/providers/Microsoft.ContainerService\",\r\n",
|
||||||
|
" \"namespace\": \"Microsoft.ContainerService\",\r\n",
|
||||||
|
" \"registrationState\": \"Registered\",\r\n",
|
||||||
|
" \"resourceTypes\": [\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2017-07-01\",\r\n",
|
||||||
|
" \"2017-01-31\",\r\n",
|
||||||
|
" \"2016-09-30\",\r\n",
|
||||||
|
" \"2016-03-30\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"capabilities\": \"None\",\r\n",
|
||||||
|
" \"locations\": [\r\n",
|
||||||
|
" \"Japan East\",\r\n",
|
||||||
|
" \"Central US\",\r\n",
|
||||||
|
" \"East US 2\",\r\n",
|
||||||
|
" \"Japan West\",\r\n",
|
||||||
|
" \"East Asia\",\r\n",
|
||||||
|
" \"South Central US\",\r\n",
|
||||||
|
" \"Australia East\",\r\n",
|
||||||
|
" \"Australia Southeast\",\r\n",
|
||||||
|
" \"Brazil South\",\r\n",
|
||||||
|
" \"Southeast Asia\",\r\n",
|
||||||
|
" \"West US\",\r\n",
|
||||||
|
" \"North Central US\",\r\n",
|
||||||
|
" \"West Europe\",\r\n",
|
||||||
|
" \"North Europe\",\r\n",
|
||||||
|
" \"East US\",\r\n",
|
||||||
|
" \"UK West\",\r\n",
|
||||||
|
" \"UK South\",\r\n",
|
||||||
|
" \"West Central US\",\r\n",
|
||||||
|
" \"West US 2\",\r\n",
|
||||||
|
" \"South India\",\r\n",
|
||||||
|
" \"Central India\",\r\n",
|
||||||
|
" \"West India\",\r\n",
|
||||||
|
" \"Canada East\",\r\n",
|
||||||
|
" \"Canada Central\",\r\n",
|
||||||
|
" \"Korea South\",\r\n",
|
||||||
|
" \"Korea Central\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"containerServices\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2018-03-31\",\r\n",
|
||||||
|
" \"2017-08-31\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"capabilities\": \"None\",\r\n",
|
||||||
|
" \"locations\": [\r\n",
|
||||||
|
" \"East US\",\r\n",
|
||||||
|
" \"West Europe\",\r\n",
|
||||||
|
" \"Central US\",\r\n",
|
||||||
|
" \"Canada Central\",\r\n",
|
||||||
|
" \"Canada East\",\r\n",
|
||||||
|
" \"UK South\",\r\n",
|
||||||
|
" \"West US\",\r\n",
|
||||||
|
" \"West US 2\",\r\n",
|
||||||
|
" \"Australia East\",\r\n",
|
||||||
|
" \"North Europe\",\r\n",
|
||||||
|
" \"Japan East\",\r\n",
|
||||||
|
" \"East US 2\",\r\n",
|
||||||
|
" \"Southeast Asia\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"managedClusters\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2017-08-31\",\r\n",
|
||||||
|
" \"2017-01-31\",\r\n",
|
||||||
|
" \"2016-09-30\",\r\n",
|
||||||
|
" \"2016-03-30\",\r\n",
|
||||||
|
" \"2015-11-01-preview\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"locations\": [],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"locations\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2017-08-31\",\r\n",
|
||||||
|
" \"2016-03-30\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"locations\": [\r\n",
|
||||||
|
" \"East US\",\r\n",
|
||||||
|
" \"West Europe\",\r\n",
|
||||||
|
" \"Central US\",\r\n",
|
||||||
|
" \"UK West\",\r\n",
|
||||||
|
" \"West Central US\",\r\n",
|
||||||
|
" \"West US 2\",\r\n",
|
||||||
|
" \"South India\",\r\n",
|
||||||
|
" \"Central India\",\r\n",
|
||||||
|
" \"West India\",\r\n",
|
||||||
|
" \"Canada East\",\r\n",
|
||||||
|
" \"Canada Central\",\r\n",
|
||||||
|
" \"Korea South\",\r\n",
|
||||||
|
" \"Korea Central\",\r\n",
|
||||||
|
" \"UK South\",\r\n",
|
||||||
|
" \"Australia East\",\r\n",
|
||||||
|
" \"North Europe\",\r\n",
|
||||||
|
" \"Japan East\",\r\n",
|
||||||
|
" \"East US 2\",\r\n",
|
||||||
|
" \"Southeast Asia\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"locations/operationresults\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2017-07-01\",\r\n",
|
||||||
|
" \"2017-01-31\",\r\n",
|
||||||
|
" \"2016-09-30\",\r\n",
|
||||||
|
" \"2016-03-30\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"locations\": [\r\n",
|
||||||
|
" \"Japan East\",\r\n",
|
||||||
|
" \"Central US\",\r\n",
|
||||||
|
" \"East US 2\",\r\n",
|
||||||
|
" \"Japan West\",\r\n",
|
||||||
|
" \"East Asia\",\r\n",
|
||||||
|
" \"South Central US\",\r\n",
|
||||||
|
" \"Australia East\",\r\n",
|
||||||
|
" \"Australia Southeast\",\r\n",
|
||||||
|
" \"Brazil South\",\r\n",
|
||||||
|
" \"Southeast Asia\",\r\n",
|
||||||
|
" \"West US\",\r\n",
|
||||||
|
" \"North Central US\",\r\n",
|
||||||
|
" \"West Europe\",\r\n",
|
||||||
|
" \"North Europe\",\r\n",
|
||||||
|
" \"East US\",\r\n",
|
||||||
|
" \"UK West\",\r\n",
|
||||||
|
" \"UK South\",\r\n",
|
||||||
|
" \"West Central US\",\r\n",
|
||||||
|
" \"West US 2\",\r\n",
|
||||||
|
" \"South India\",\r\n",
|
||||||
|
" \"Central India\",\r\n",
|
||||||
|
" \"West India\",\r\n",
|
||||||
|
" \"Canada East\",\r\n",
|
||||||
|
" \"Canada Central\",\r\n",
|
||||||
|
" \"Korea South\",\r\n",
|
||||||
|
" \"Korea Central\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"locations/operations\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2018-03-31\",\r\n",
|
||||||
|
" \"2017-08-31\",\r\n",
|
||||||
|
" \"2017-07-01\",\r\n",
|
||||||
|
" \"2017-01-31\",\r\n",
|
||||||
|
" \"2016-09-30\",\r\n",
|
||||||
|
" \"2016-03-30\",\r\n",
|
||||||
|
" \"2015-11-01-preview\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"locations\": [],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"operations\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" {\r\n",
|
||||||
|
" \"aliases\": null,\r\n",
|
||||||
|
" \"apiVersions\": [\r\n",
|
||||||
|
" \"2017-09-30\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"locations\": [\r\n",
|
||||||
|
" \"East US\",\r\n",
|
||||||
|
" \"West Europe\",\r\n",
|
||||||
|
" \"Central US\",\r\n",
|
||||||
|
" \"Canada East\",\r\n",
|
||||||
|
" \"Canada Central\",\r\n",
|
||||||
|
" \"UK South\",\r\n",
|
||||||
|
" \"West US\",\r\n",
|
||||||
|
" \"West US 2\",\r\n",
|
||||||
|
" \"Australia East\",\r\n",
|
||||||
|
" \"North Europe\",\r\n",
|
||||||
|
" \"Japan East\",\r\n",
|
||||||
|
" \"East US 2\",\r\n",
|
||||||
|
" \"Southeast Asia\"\r\n",
|
||||||
|
" ],\r\n",
|
||||||
|
" \"properties\": null,\r\n",
|
||||||
|
" \"resourceType\": \"locations/orchestrators\"\r\n",
|
||||||
|
" }\r\n",
|
||||||
|
" ]\r\n",
|
||||||
|
"}\r\n",
|
||||||
|
"\u001b[0m"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!az provider show -n Microsoft.ContainerService"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"<a id='section3'></a>\n",
|
||||||
|
"## Create resource group and create AKS"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"### Create resource group\n",
|
||||||
|
"Azure encourages the use of groups to organise all the Azure components you deploy. That way it is easier to find them but also we can deleted a number of resources simply by deleting the group."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 16,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"{\r\n",
|
||||||
|
" \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/mabouaks\",\r\n",
|
||||||
|
" \"location\": \"eastus\",\r\n",
|
||||||
|
" \"managedBy\": null,\r\n",
|
||||||
|
" \"name\": \"mabouaks\",\r\n",
|
||||||
|
" \"properties\": {\r\n",
|
||||||
|
" \"provisioningState\": \"Succeeded\"\r\n",
|
||||||
|
" },\r\n",
|
||||||
|
" \"tags\": null\r\n",
|
||||||
|
"}\r\n",
|
||||||
|
"\u001b[0m"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
" !az group create --name $resource_group --location $location"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Below, we create the AKS cluster in the resource group we created earlier. This can take up to 15 minutes."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"\u001b[33mSSH key files '/home/mabou/.ssh/id_rsa' and '/home/mabou/.ssh/id_rsa.pub' have been generated under ~/.ssh to allow SSH access to the VM. If using machines without permanent storage like Azure Cloud Shell without an attached file share, back up your keys to a safe location\u001b[0m\n",
|
||||||
|
"\u001b[K - Running ...principal creation[##################################] 100.0000%\r"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!az aks create --resource-group $resource_group --name $aks_name --node-count 1 --generate-ssh-keys -s Standard_NC6"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"### Install kubectl CLI\n",
|
||||||
|
"\n",
|
||||||
|
"To connect to the Kubernetes cluster, we will use kubectl, the Kubernetes command-line client. To install, run the following:"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!sudo az aks install-cli"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"<a id='section4'></a>\n",
|
||||||
|
"## Connect to AKS cluster\n",
|
||||||
|
"\n",
|
||||||
|
"To configure kubectl to connect to the Kubernetes cluster, run the following command:"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!az aks get-credentials --resource-group $resource_group --name $aks_name"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Let's verify connection by listing the nodes."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl get nodes"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Let's check the pods on our cluster."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl get pods --all-namespaces"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"<a id='section5'></a>\n",
|
||||||
|
"## Deploy application\n",
|
||||||
|
"\n",
|
||||||
|
"Below we define our Kubernetes manifest file for our service and load balancer. Note that we have to specify the volume mounts to the drivers that are located on the node.\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"app_template = {\n",
|
||||||
|
" \"apiVersion\": \"apps/v1beta1\",\n",
|
||||||
|
" \"kind\": \"Deployment\",\n",
|
||||||
|
" \"metadata\": {\n",
|
||||||
|
" \"name\": \"azure-dl\"\n",
|
||||||
|
" },\n",
|
||||||
|
" \"spec\":{\n",
|
||||||
|
" \"replicas\":1,\n",
|
||||||
|
" \"template\":{\n",
|
||||||
|
" \"metadata\":{\n",
|
||||||
|
" \"labels\":{\n",
|
||||||
|
" \"app\":\"azure-dl\"\n",
|
||||||
|
" }\n",
|
||||||
|
" },\n",
|
||||||
|
" \"spec\":{\n",
|
||||||
|
" \"containers\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"name\": \"azure-dl\",\n",
|
||||||
|
" \"image\": image_name,\n",
|
||||||
|
" \"env\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"name\": \"LD_LIBRARY_PATH\",\n",
|
||||||
|
" \"value\": \"$LD_LIBRARY_PATH:/usr/local/nvidia/lib64:/opt/conda/envs/py3.6/lib\"\n",
|
||||||
|
" }\n",
|
||||||
|
" ],\n",
|
||||||
|
" \"ports\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"containerPort\":80,\n",
|
||||||
|
" \"name\":\"model\"\n",
|
||||||
|
" }\n",
|
||||||
|
" ],\n",
|
||||||
|
" \"volumeMounts\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"mountPath\": \"/usr/local/nvidia\",\n",
|
||||||
|
" \"name\": \"nvidia\"\n",
|
||||||
|
" }\n",
|
||||||
|
" ],\n",
|
||||||
|
" \"resources\":{\n",
|
||||||
|
" \"requests\":{\n",
|
||||||
|
" \"alpha.kubernetes.io/nvidia-gpu\": 1\n",
|
||||||
|
" },\n",
|
||||||
|
" \"limits\":{\n",
|
||||||
|
" \"alpha.kubernetes.io/nvidia-gpu\": 1\n",
|
||||||
|
" }\n",
|
||||||
|
" } \n",
|
||||||
|
" }\n",
|
||||||
|
" ],\n",
|
||||||
|
" \"volumes\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"name\": \"nvidia\",\n",
|
||||||
|
" \"hostPath\":{\n",
|
||||||
|
" \"path\":\"/usr/local/nvidia\"\n",
|
||||||
|
" },\n",
|
||||||
|
" },\n",
|
||||||
|
" ]\n",
|
||||||
|
" }\n",
|
||||||
|
" }\n",
|
||||||
|
" }\n",
|
||||||
|
"}\n",
|
||||||
|
"\n",
|
||||||
|
"service_temp = {\n",
|
||||||
|
" \"apiVersion\": \"v1\",\n",
|
||||||
|
" \"kind\": \"Service\",\n",
|
||||||
|
" \"metadata\": {\n",
|
||||||
|
" \"name\": \"azure-dl\"\n",
|
||||||
|
" },\n",
|
||||||
|
" \"spec\":{\n",
|
||||||
|
" \"type\": \"LoadBalancer\",\n",
|
||||||
|
" \"ports\":[\n",
|
||||||
|
" {\n",
|
||||||
|
" \"port\":80\n",
|
||||||
|
" }\n",
|
||||||
|
" ],\n",
|
||||||
|
" \"selector\":{\n",
|
||||||
|
" \"app\":\"azure-dl\"\n",
|
||||||
|
" }\n",
|
||||||
|
" }\n",
|
||||||
|
"}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"write_json_to_file(app_template, 'az-dl.json') # We write the service template to the json file"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"write_json_to_file(service_temp, 'az-dl.json', mode='a') # We add the loadbelanacer template to the json file"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Let's check the manifest created."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!cat az-dl.json"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Next, we will use kubectl create command to deploy our application."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl create -f az-dl.json"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Let's check if the pod is deployed."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl get pods --all-namespaces"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"If anything goes wrong you can use the commands below to observe the events on the node as well as review the logs."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl get events"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"pod_json = !kubectl get pods -o json\n",
|
||||||
|
"pod_dict = json.loads(''.join(pod_json))\n",
|
||||||
|
"!kubectl logs {pod_dict['items'][0]['metadata']['name']}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"It can take a few minutes for the service to populate the EXTERNAL-IP field. This will be the IP you use to call the service. You can also specify an IP to use please see the AKS documentation for further details."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!kubectl get service azure-dl"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Next, we will [test our web application](05_TestWebApp.ipynb) deployed on AKS. "
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"anaconda-cloud": {},
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python [conda env:AKSDeployment]",
|
||||||
|
"language": "python",
|
||||||
|
"name": "conda-env-AKSDeployment-py"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.5.5"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 1
|
||||||
|
}
|
||||||
Различия файлов скрыты, потому что одна или несколько строк слишком длинны
|
|
@ -4,13 +4,7 @@
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Load Test deployed web application"
|
"### Load Test deployed web application\n",
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"This notebook pulls some images and tests them against the deployed web application. We submit requests asychronously which should reduce the contribution of latency."
|
"This notebook pulls some images and tests them against the deployed web application. We submit requests asychronously which should reduce the contribution of latency."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|
@ -29,7 +23,6 @@
|
||||||
"import matplotlib.pyplot as plt\n",
|
"import matplotlib.pyplot as plt\n",
|
||||||
"from testing_utilities import to_img, gen_variations_of_one_image\n",
|
"from testing_utilities import to_img, gen_variations_of_one_image\n",
|
||||||
"from tqdm import tqdm\n",
|
"from tqdm import tqdm\n",
|
||||||
"import requests\n",
|
|
||||||
"\n",
|
"\n",
|
||||||
"%matplotlib inline"
|
"%matplotlib inline"
|
||||||
]
|
]
|
||||||
|
|
@ -72,7 +65,7 @@
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Get the IP address of our service."
|
"Get the IP address of our service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -121,7 +114,7 @@
|
||||||
{
|
{
|
||||||
"data": {
|
"data": {
|
||||||
"text/plain": [
|
"text/plain": [
|
||||||
"<matplotlib.image.AxesImage at 0x7fafbd8fd1d0>"
|
"<matplotlib.image.AxesImage at 0x7fa8055a7e10>"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"execution_count": 7,
|
"execution_count": 7,
|
||||||
|
|
@ -213,7 +206,9 @@
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 13,
|
"execution_count": 13,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"scrolled": false
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"async def run(url_list, num_concurrent=CONCURRENT_REQUESTS):\n",
|
"async def run(url_list, num_concurrent=CONCURRENT_REQUESTS):\n",
|
||||||
|
|
@ -236,27 +231,29 @@
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Below we run the 100 requests against our deployed service."
|
"Below we run the 100 requests against our deployed service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 14,
|
"execution_count": 14,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"scrolled": false
|
||||||
|
},
|
||||||
"outputs": [
|
"outputs": [
|
||||||
{
|
{
|
||||||
"name": "stderr",
|
"name": "stderr",
|
||||||
"output_type": "stream",
|
"output_type": "stream",
|
||||||
"text": [
|
"text": [
|
||||||
"100%|██████████| 100/100 [00:07<00:00, 13.27it/s]"
|
"100%|██████████| 100/100 [00:05<00:00, 17.94it/s]"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "stdout",
|
"name": "stdout",
|
||||||
"output_type": "stream",
|
"output_type": "stream",
|
||||||
"text": [
|
"text": [
|
||||||
"Total Elapsed 7.540950823575258\n",
|
"Total Elapsed 5.579234362579882\n",
|
||||||
"Avg time taken 75.41 ms\n"
|
"Avg time taken 55.79 ms\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -276,6 +273,13 @@
|
||||||
"print('Avg time taken {0:4.2f} ms'.format(1000*elapsed/len(url_list)))"
|
"print('Avg time taken {0:4.2f} ms'.format(1000*elapsed/len(url_list)))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"Below we can see the output of some of our calls"
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 15,
|
"execution_count": 15,
|
||||||
|
|
@ -284,12 +288,24 @@
|
||||||
{
|
{
|
||||||
"data": {
|
"data": {
|
||||||
"text/plain": [
|
"text/plain": [
|
||||||
"[({'result': \"({'image': [('n02127052', 'lynx', 0.9815654), ('n02128385', 'leopard', 0.0077465144), ('n02123159', 'tiger_cat', 0.0037241662)]}, 'Computed in 99.44 ms')\"},\n",
|
"[({'result': [{'image': [['n02127052 lynx, catamount', 0.9974566102027893],\n",
|
||||||
" 0.2683806661516428),\n",
|
" ['n02128385 leopard, Panthera pardus', 0.0015052433591336012],\n",
|
||||||
" ({'result': \"({'image': [('n02127052', 'lynx', 0.9816606), ('n02128385', 'leopard', 0.0077482197), ('n02123159', 'tiger_cat', 0.003675996)]}, 'Computed in 97.08 ms')\"},\n",
|
" ['n02128757 snow leopard, ounce, Panthera uncia',\n",
|
||||||
" 0.3648473871871829),\n",
|
" 0.0005146243493072689]]},\n",
|
||||||
" ({'result': \"({'image': [('n02127052', 'lynx', 0.9816492), ('n02128385', 'leopard', 0.007743971), ('n02123159', 'tiger_cat', 0.0036858919)]}, 'Computed in 97.15 ms')\"},\n",
|
" 'Computed in 73.49 ms']},\n",
|
||||||
" 0.4713786570355296)]"
|
" 0.24386161286383867),\n",
|
||||||
|
" ({'result': [{'image': [['n02127052 lynx, catamount', 0.9974561333656311],\n",
|
||||||
|
" ['n02128385 leopard, Panthera pardus', 0.0015048063360154629],\n",
|
||||||
|
" ['n02128757 snow leopard, ounce, Panthera uncia',\n",
|
||||||
|
" 0.0005157074192538857]]},\n",
|
||||||
|
" 'Computed in 74.62 ms']},\n",
|
||||||
|
" 0.32052970863878727),\n",
|
||||||
|
" ({'result': [{'image': [['n02127052 lynx, catamount', 0.9974557757377625],\n",
|
||||||
|
" ['n02128385 leopard, Panthera pardus', 0.0015042462619021535],\n",
|
||||||
|
" ['n02128757 snow leopard, ounce, Panthera uncia',\n",
|
||||||
|
" 0.0005160688306204975]]},\n",
|
||||||
|
" 'Computed in 73.0 ms']},\n",
|
||||||
|
" 0.3948906175792217)]"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"execution_count": 15,
|
"execution_count": 15,
|
||||||
|
|
@ -315,7 +331,7 @@
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"source": [
|
"source": [
|
||||||
"num_succesful=[eval(i[0]['result'])[0]['image'][0][0] for i in complete_responses].count('n02127052')\n",
|
"num_succesful=[i[0]['result'][0]['image'][0][0] for i in complete_responses].count('n02127052 lynx, catamount')\n",
|
||||||
"print('Succesful {} out of {}'.format(num_succesful, len(url_list)))"
|
"print('Succesful {} out of {}'.format(num_succesful, len(url_list)))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|
@ -327,8 +343,12 @@
|
||||||
{
|
{
|
||||||
"data": {
|
"data": {
|
||||||
"text/plain": [
|
"text/plain": [
|
||||||
"({'result': \"({'image': [('n02127052', 'lynx', 0.9815654), ('n02128385', 'leopard', 0.0077465144), ('n02123159', 'tiger_cat', 0.0037241662)]}, 'Computed in 99.44 ms')\"},\n",
|
"({'result': [{'image': [['n02127052 lynx, catamount', 0.9974566102027893],\n",
|
||||||
" 0.2683806661516428)"
|
" ['n02128385 leopard, Panthera pardus', 0.0015052433591336012],\n",
|
||||||
|
" ['n02128757 snow leopard, ounce, Panthera uncia',\n",
|
||||||
|
" 0.0005146243493072689]]},\n",
|
||||||
|
" 'Computed in 73.49 ms']},\n",
|
||||||
|
" 0.24386161286383867)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"execution_count": 17,
|
"execution_count": 17,
|
||||||
|
|
@ -356,15 +376,16 @@
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"To tear down the cluster and all related resources go to the [tear down the cluster](07_TearDown.ipynb) notebook."
|
"To tear down the cluster and all related resources go to the [tear down the cluster](07_TearDown.ipynb) notebook."
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
"anaconda-cloud": {},
|
||||||
"kernelspec": {
|
"kernelspec": {
|
||||||
"display_name": "Python [conda env:AKSDeploymentKeras]",
|
"display_name": "Python [conda env:AKSDeployment]",
|
||||||
"language": "python",
|
"language": "python",
|
||||||
"name": "conda-env-AKSDeploymentKeras-py"
|
"name": "conda-env-AKSDeployment-py"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"language_info": {
|
||||||
"codemirror_mode": {
|
"codemirror_mode": {
|
||||||
|
|
@ -380,5 +401,5 @@
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"nbformat": 4,
|
"nbformat": 4,
|
||||||
"nbformat_minor": 2
|
"nbformat_minor": 1
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Загрузка…
Ссылка в новой задаче