Azure
/
ai-toolkit-iot-edge
зеркало из https://github.com/Azure/ai-toolkit-iot-edge.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Merge pull request #13 from kgremban/master 

Update anomaly detection tutorial
This commit is contained in:
Ted Way 2019-03-11 12:02:51 -07:00 коммит произвёл GitHub
Родитель 11707ac8cd c8ff6f157d
Коммит 82d57f5ce0
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
5 изменённых файлов: 128 добавлений и 261 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

283
IoT Edge anomaly detection tutorial/00-anomaly-detection-tutorial.ipynb
Просмотреть файл

@ -28,33 +28,36 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Enter the resource group in Azure where you want to provision the resources \n",
"resource_group_name = \"tedway-iotedge-workshop\"\n",
"resource_group_name = \"\"\n",
"\n",
"# Enter Azure region where your services will be provisioned\n",
"azure_region=\"eastus2\"\n",
"# Enter Azure region where your services will be provisioned, for example \"eastus2\"\n",
"azure_region=\"\"\n",
"\n",
"# Pick an Azure IoT Hub name\n",
"iot_hub_name=\"tedway-iothub\"\n",
"# Enter your Azure IoT Hub name \n",
"# If you don't have an IoT Hub, pick a name to make a new one \n",
"iot_hub_name=\"\"\n",
"\n",
"# Pick a name for your IoT Edge device\n",
"# Enter your IoT Edge device ID \n",
"# If you don't have an IoT Edge device registered, pick a name to create a new one \n",
"# This is NOT the name of your VM, but it's just an entry in your IoT Hub, so you can pick any name\n",
"iot_device_id=\"iot-edge-device\"\n",
"iot_device_id=\"\"\n",
"\n",
"# Pick a subscription ID to provision your services\n",
"subscription_id = \"75f78a03-482f-4fd8-8c71-5ddc08f92726\"\n",
"# Provide your Azure subscription ID to provision your services\n",
"subscription_id = \"\"\n",
"\n",
"# Pick a name for your Azure ML workspace\n",
"aml_workspace_name = \"tedway-aml-iotedge\""
"# Provide your Azure ML service workspace name \n",
"# If you don't have a workspace, pick a name to create a new one\n",
"aml_workspace_name = \"\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -74,7 +77,17 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load the IoT extension for Azure CLI\n",
"!az extension add --name azure-cli-iot-ext"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -83,17 +96,9 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SDK version: 1.0.10\n"
]
}
],
"outputs": [],
"source": [
"# Check core SDK version number\n",
"import azureml.core\n",
@ -114,7 +119,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1: Provision Azure ML workspace\n",
"### 2.1: Provision Azure ML workspace (optional)\n",
"After you've done this once you no longer have to run this cell anymore"
]
},
@ -136,7 +141,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2: Provision IoT Hub\n",
"### 2.2: Provision IoT Hub (optional)\n",
"If you get an error because there's already one free hub in your subscription, change the SKU to S1. If you get an error that the IoT Hub name isn't available, it means that someone else already has a hub with that name. \n"
]
},
@ -164,7 +169,7 @@
"metadata": {},
"source": [
"## 3: Load resources\n",
"Load the Azure ML workspace and get the IoT Edge device connection string in your IoT Hub."
"Load the Azure ML workspace and get the IoT Edge device connection string from your IoT Hub."
]
},
{
@ -176,33 +181,13 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"tags": [
"create workspace"
]
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Falling back to use azure cli credentials. This fall back to use azure cli credentials will be removed in the next release. \n",
"Make sure your code doesn't require 'az login' to have happened before using azureml-sdk, except the case when you are specifying AzureCliAuthentication in azureml-sdk.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found the config file in: c:\\Users\\tedway\\git\\ai-toolkit-iot-edge\\IoT Edge anomaly detection tutorial\\aml_config\\config.json\n",
"tedway-aml-iotedge\n",
"tedway-iotedge-workshop\n",
"eastus2\n",
"75f78a03-482f-4fd8-8c71-5ddc08f92726\n"
]
}
],
"outputs": [],
"source": [
"# Initialize a workspace object from persisted configuration\n",
"from azureml.core import Workspace\n",
@ -220,21 +205,11 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"cs\": \"HostName=tedway-iothub.azure-devices.net;DeviceId=iot-edge-device;SharedAccessKey=dJpK8912eAOw+VFF4llgmfSq/o9+AbiJJmNkNHCPWQU=\"\n",
"}\n"
]
}
],
"outputs": [],
"source": [
"# Get the connection string that you will need to enter in the IoT Edge device\n",
"!az iot hub device-identity show-connection-string --device-id $iot_device_id --hub-name $iot_hub_name"
@ -244,14 +219,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3: Anomaly Detection Model"
"## 4: Anomaly Detection Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.1 Train the model"
"## 4.1 Train the model"
]
},
{
@ -266,7 +241,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -294,7 +269,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -304,22 +279,9 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
" max_features=None, max_leaf_nodes=None,\n",
" min_impurity_decrease=0.0, min_impurity_split=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
" splitter='best')\n"
]
}
],
"outputs": [],
"source": [
"# Split data 65%-35% into training set and test set\n",
"X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.35, random_state=0)\n",
@ -336,19 +298,11 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy is 0.9907834101382489\n"
]
}
],
"outputs": [],
"source": [
"# Evaluate the test set\n",
"accuracy = clf1.score(X_test, Y_test)\n",
@ -358,17 +312,9 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exported the model to model.pkl\n"
]
}
],
"outputs": [],
"source": [
"# Serialize the model and write to disk\n",
"f = open('model.pkl', 'wb')\n",
@ -379,19 +325,9 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Import the model from model.pkl\n",
"New sample: [[33.66995566, 2.44341267, 21.39450979, 26]]\n",
"Predicted class is [1]\n"
]
}
],
"outputs": [],
"source": [
"# Test the model by importing it and providing a sample data point\n",
"print(\"Import the model from model.pkl\")\n",
@ -417,7 +353,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.2 Register Model"
"### 4.2 Register Model"
]
},
{
@ -431,21 +367,13 @@
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": null,
"metadata": {
"tags": [
"register model from file"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Registering model model.pkl\n"
]
}
],
"outputs": [],
"source": [
"from azureml.core.model import Model\n",
"\n",
@ -458,17 +386,9 @@
},
{
"cell_type": "code",
"execution_count": 18,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"model.pkl\tSample anomaly detection model for IOT tutorial\t6\n"
]
}
],
"outputs": [],
"source": [
"print(model.name, model.description, model.version, sep = '\\t')"
]
@ -477,7 +397,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.2 Create Docker Image"
"### 4.2 Create Docker Image"
]
},
{
@ -489,17 +409,9 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting iot_score.py\n"
]
}
],
"outputs": [],
"source": [
"%%writefile iot_score.py\n",
"# This script generates the scoring file\n",
@ -540,7 +452,7 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@ -562,21 +474,13 @@
},
{
"cell_type": "code",
"execution_count": 21,
"execution_count": null,
"metadata": {
"tags": [
"create image"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating image\n"
]
}
],
"outputs": [],
"source": [
"from azureml.core.image import Image, ContainerImage\n",
"\n",
@ -603,22 +507,13 @@
},
{
"cell_type": "code",
"execution_count": 22,
"execution_count": null,
"metadata": {
"tags": [
"create image"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running...................................\n",
"SucceededImage creation operation finished for image tempanomalydetection:6, operation \"Succeeded\"\n"
]
}
],
"outputs": [],
"source": [
"image.wait_for_creation(show_output = True)"
]
@ -648,13 +543,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4: Test model on Azure Container Instance\n",
"## 5: Test model on Azure Container Instance\n",
"You can test the performance of your model by deploying the container to ACI. Note that the service creation can take few minutes."
]
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": null,
"metadata": {
"tags": [
"deploy service",
@ -673,30 +568,18 @@
},
{
"cell_type": "code",
"execution_count": 24,
"execution_count": null,
"metadata": {
"tags": [
"deploy service",
"aci"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tempsensor-iotedge-ml-6\n",
"Creating service\n",
"Running.....................\n",
"SucceededACI service creation operation finished, operation \"Succeeded\"\n",
"Healthy\n"
]
}
],
"outputs": [],
"source": [
"from azureml.core.webservice import Webservice\n",
"\n",
"aci_service_name = 'tempsensor-iotedge-ml-6'\n",
"aci_service_name = 'tempsensor-iotedge-ml-1'\n",
"print(aci_service_name)\n",
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
" image = image,\n",
@ -710,7 +593,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.1: Test web service"
"### 5.1: Test web service"
]
},
{
@ -722,22 +605,14 @@
},
{
"cell_type": "code",
"execution_count": 25,
"execution_count": null,
"metadata": {
"tags": [
"deploy service",
"aci"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['{\"machine\": {\"temperature\": 31.16469009, \"pressure\": 2.158002669}, \"ambient\": {\"temperature\": 21.17794693, \"humidity\": 25}, \"timeCreated\": \"2017-10-27T18:14:02.4911177Z\", \"anomaly\": false}']\n"
]
}
],
"outputs": [],
"source": [
"import json\n",
"\n",
@ -761,7 +636,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2: Delete ACI to clean up"
"### 5.2: Delete ACI to clean up"
]
},
{
@ -782,25 +657,15 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5: Deploy container to Azure IoT Edge device\n",
"## 6: Deploy container to Azure IoT Edge device\n",
"Create a deployment.json file that contains the modules you want to deploy to the device and the routes. Then push this file to the IoT Hub, which will then send it to the IoT Edge device. The IoT Edge agent will then pull the Docker images and run them."
]
},
{
"cell_type": "code",
"execution_count": 26,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tedwayamacrszmxuxxl.azurecr.io/tempanomalydetection:6\n",
"tedwayamacrszmxuxxl\n",
"75f78a03-482f-4fd8-8c71-5ddc08f92726\n"
]
}
],
"outputs": [],
"source": [
"# Getting your container details\n",
"container_reg = ws.get_details()[\"containerRegistry\"]\n",
@ -820,7 +685,7 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [

4
IoT Edge anomaly detection tutorial/README.md
Просмотреть файл

@ -1,5 +1,5 @@
# Azure IoT Edge Anomaly Detection Tutorial
To run through this [tutorial](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-machine-learning/), follow the instructions [here](https://aka.ms/aml-tutorial/ai-iot-power-couple).
The Azure IoT Edge anomaly detection sample was built for the [Deploy Azure Machine Learning as an IoT Edge module](https://docs.microsoft.com/azure/iot-edge/tutorial-deploy-machine-learning/) tutorial. Follow the instructions in that article to prepare your IoT Edge device and other Azure resources before beginning.
The [sample notebook](00-anomaly-detection-tutorial.ipynb) and all files are available in this repo, but you will still need to provision an IoT Edge device. Follow the instructions [here](https://aka.ms/aml-tutorial/ai-iot-power-couple).
The [sample notebook](00-anomaly-detection-tutorial.ipynb) and all files are available in this repo.

6
IoT Edge anomaly detection tutorial/aml_config/config.json
Просмотреть файл

@ -1,5 +1,5 @@
{
"subscription_id": "75f78a03-482f-4fd8-8c71-5ddc08f92726",
"resource_group": "tedway-iotedge-workshop",
"workspace_name": "tedway-aml-iotedge"
"subscription_id": "<subscription id>",
"resource_group": "<resource group name>",
"workspace_name": "<workspace name>"
}

11
IoT Edge anomaly detection tutorial/deployment.json
Просмотреть файл

@ -9,10 +9,11 @@
"settings": {
"minDockerVersion": "v1.25",
"loggingOptions": "",
"registryCredentials": {"tedwayamacrszmxuxxl": {
"username": "tedwayamacrszmxuxxl",
"password":"Rsq7sMoh28aFMm4AjAsfsmyPORe=Lk36",
"address":"tedwayamacrszmxuxxl.azurecr.io"
"registryCredentials": {
"registry": {
"username": "username",
"password":"password",
"address":"registry.azurecr.io"
}
}
}
@ -62,7 +63,7 @@
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "tedwayamacrszmxuxxl.azurecr.io/tempanomalydetection:6",
"image": "registry.azurecr.io/tempanomalydetection:6",
"createOptions": "{\"HostConfig\":{\"Binds\":[\"/data/misc/camera:/app/vam_model_folder\"],\"NetworkMode\":\"host\"},\"NetworkingConfig\":{\"EndpointsConfig\":{\"host\":{}}},\"WorkingDir\":\"/var/azureml-app\"}"
}
}

9
IoT Edge anomaly detection tutorial/iot-workshop-deployment-template.json
Просмотреть файл

@ -9,7 +9,8 @@
"settings": {
"minDockerVersion": "v1.25",
"loggingOptions": "",
"registryCredentials": {"__REGISTRY_NAME": {
"registryCredentials": {
"__REGISTRY_NAME": {
"username": "__REGISTRY_USER_NAME",
"password":"__REGISTRY_PASSWORD",
"address":"__REGISTRY_NAME.azurecr.io"
@ -73,8 +74,8 @@
"properties.desired": {
"schemaVersion": "1.0",
"routes": {
"sensorToMachineLearning": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/machinelearningmodule/inputs/amlInput\")",
"machineLearningToIoTHub": "FROM /messages/modules/machinelearningmodule/outputs/amlOutput INTO $upstream"
"sensorToMachineLearning": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/__MODULE_NAME/inputs/amlInput\")",
"machineLearningToIoTHub": "FROM /messages/modules/__MODULE_NAME/outputs/amlOutput INTO $upstream"
},
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
@ -84,7 +85,7 @@
"tempSensor": {
"properties.desired": {}
},
"machinelearningmodule": {
"__MODULE_NAME": {
"properties.desired": {}
}
}