Azure
/
iotedgedev
зеркало из https://github.com/Azure/iotedgedev.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Sep each class into own file, add --monitor-events, beef up readme

This commit is contained in:
Jon Gallant 2018-02-03 23:09:28 -08:00
Родитель 66e4cc286c
Коммит fa5c0ef192
21 изменённых файлов: 1047 добавлений и 937 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

2
.gitignore поставляемый
Просмотреть файл

@ -77,3 +77,5 @@ py36
.pypirc
test_project
README
node_modules

291
README.md
Просмотреть файл

@ -1,40 +1,71 @@
# Azure IoT Edge Dev Tool
The Azure IoT Edge Dev Tool **greatly simplifies** your Azure IoT Edge development process. It has everything you need to get started and helps with your day-to-day IoT Edge development.
The **Azure IoT Edge Dev Tool** greatly simplifies [Azure IoT Edge](https://azure.microsoft.com/en-us/services/iot-edge/) development down to simple CLI commands driven by Environment Variables. This tool will:
You will be able to do all of the following with simple one line commands.
- Get you started with IoT Edge development with the [IoT Edge Dev Container](#dev-machine-setup) and IoT Edge Project Scaffolding that contains a sample module and all the required configuration files.
- Speed up your inner-loop dev (dev, debug, test) by reducing multi-step build & deploy processes into one-line CLI commands and well as drive your outer-loop CI/CD pipeline. You can use all the same commands in both stages of your development life-cycle.
1. Install the Azure IoT Edge Dev Tool: `pip install azure-iot-edge-dev-tool`
1. Create a new Edge project: `iotedgedev project --create`
1. Build, Push and Deploy modules: `iotedgedev modules --build --deploy`
1. Setup and Start the Edge Runtime: `iotedgedev runtime --setup --start`
1. View and Save Docker log files: `iotedgedev docker --logs`
1. Use a Custom Container Registry: `iotedgedev docker --setup-registry`
The **Azure IoT Edge Dev Tool** enables you to do all of the following with simple one-line CLI commands.
1. **Install**: Install the Azure IoT Edge Dev Tool:
`pip install azure-iot-edge-dev-tool`
1. **Project**: Create a new IoT Edge project that includes a sample module and all the the required configuration files.
`iotedgedev project --create edgeproject1`
1. **Build & Deploy**: Build, Push and Deploy modules:
`iotedgedev modules --build --deploy`
> This will `build`, `publish`, `docker build, tag and push` and `deploy modules` to your IoT Edge device.
1. **Setup & Start**: Setup and Start the IoT Edge Runtime:
`iotedgedev runtime --setup --start`
1. **View Messages**: View Messages Sent from IoT Edge to IoT Hub:
`iotedgedev iothub --monitor-events`
1. **View Logs**: View and Save Docker log files:
`iotedgedev docker --logs`
1. **Setup Custom Registry**: Use a Custom Container Registry:
`iotedgedev docker --setup-registry`
The project was created by Microsofties that work with IoT Edge customers, who have found it very helpful. We hope to get a variation of this officially supported by the Azure IoT team in the near future. Your contributions and feedback to this project will help us build an amazing developer experience, so please do not hesitate to participate.
Please see [Azure IoT Edge Dev Resources](https://github.com/jonbgallant/azure-iot-edge-dev) for links to official docs and other IoT Edge dev information.
## Videos
### [Azure IoT Edge Dev Tool in 2 Minutes](https://www.youtube.com/watch?v=NsnxMshMhmA)
[![Azure IoT Edge Dev Tool in 2 Minutes](assets/edgedevtool2mins.png)](https://www.youtube.com/watch?v=NsnxMshMhmA)
https://www.youtube.com/watch?v=NsnxMshMhmA
[![Azure IoT Edge Dev Tool with Windows Subsystem for Linux (WSL)](assets/edgedevtoolwsl.png)](https://www.youtube.com/watch?v=k5ZtTmHgs_8)
https://www.youtube.com/watch?v=k5ZtTmHgs_8
### [Azure IoT Edge Dev Tool: Introduction](https://www.youtube.com/watch?v=lcDFX8PXqUQ)
[![Azure IoT Edge Dev Tool: Introduction](assets/edgedevtoolintro.png)](https://www.youtube.com/watch?v=lcDFX8PXqUQ)
https://www.youtube.com/watch?v=lcDFX8PXqUQ
### [Azure IoT Edge Dev Tool with Windows Subsystem for Linux (WSL)](https://www.youtube.com/watch?v=k5ZtTmHgs_8)
[![Azure IoT Edge Dev Tool with Windows Subsystem for Linux (WSL)](assets/edgedevtoolwsl.png)](https://www.youtube.com/watch?v=k5ZtTmHgs_8)
## Setup
### Azure Setup
1. [Create Azure IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-getstarted#create-an-iot-hub)
1. [Create Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal)
- Make sure you enable Admin Access when you create the Azure Container Registry
1. Create Edge Device using the Azure Portal
#### Manual Setup
1. [**Create Azure IoT Hub**](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-getstarted#create-an-iot-hub)
1. **Create Edge Device** using the Azure Portal
- In your IoT Hub, click "IoT Edge", then click "Add IoT Edge Device"
1. [**Create Azure Container Registry**](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal)
> Only needed when you want to push your modules to a central registry. You can run a local registry by setting the .env/CONTAINER_REGISTRY_SERVER setting to `localhost:5000`.
- Make sure you enable Admin Access when you create the Azure Container Registry
#### Automated Setup
You can also deploy the IoT Hub and Container Registry with this **Deploy to Azure** template:
@ -42,9 +73,34 @@ You can also deploy the IoT Hub and Container Registry with this **Deploy to Azu
### Dev Machine Setup
Here's what you need to do to get `iotedgedev` running on your dev machine. If you are using a separate Edge device, like a Raspberry Pi, you do not need to run all of these steps on your Edge device, you can just use `iotedgectl` directly . See the [Edge Device Setup](#edge-device-setup) section below for more information on setting up your Edge device.
Here's what you need to do to get Azure IoT Edge Dev Tool (aka `iotedgedev`) running on your dev machine. If you are using a separate Edge device, like a Raspberry Pi, you do not need to run all of these steps on your IoT Edge device, you can just use `iotedgectl` directly on the device. See the [IoT Edge Device Setup](#iot-edge-device-setup) section below for more information on setting up your Edge device.
> Note: See the ["Test Coverage"](#test-coverage) section below to see what this module has been tested with.
> Note: See the ["Test Coverage"](#test-coverage) section below to see what the Edge Dev Tool has been tested with.
#### IoT Edge Dev Tool Container
You can use the IoT Edge Dev Tool container to avoid having to install all the dependencies on your local dev machine.
> (Only runs on Linux Containers at this time. Windows Container support coming soon.)
1. Install Docker
1. Create Local Folder on Host to contain your projects.
`c:/temp/edgeprojects`
1. Start Container:
`docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v c:/temp/edgeprojects:/home jongallant/iotedgedev`
1. Change to the `/home` directory:
`cd /home`
1. Skip all the install steps below and jump to the [Usage](#usage) section below.
#### Local Dev Machine Setup
> You don't need to do this if you are running the container as described in the previous step.
1. Install **[Docker](https://docs.docker.com/engine/installation/)**
- Windows
@ -69,18 +125,16 @@ Here's what you need to do to get `iotedgedev` running on your dev machine. If y
sudo sh -c "echo Defaults env_keep += \"DOCKER_HOST\" >> /etc/sudoers.d/docker"
```
- Linux
1. **Do not** install via `sudo apt install docker.io`. Use the proper steps for [CE here](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-docker-ce), or use the [convenience script](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-convenience-script).
- We've seen some issues with docker.io. If Edge doesn't run for you, then try installing Docker CE directly instead of via docker.io. Use the [CE install steps](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-docker-ce), or use the [convenience script](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-convenience-script).
1. Install **Python 2.7 or Python 3**
1. Install **Python 2.7+ or Python 3.6+**
- Windows - [Install from Python's website](https://www.python.org/downloads/)
- Linux - `sudo apt install python-pip` or `sudo apt install python3-pip`
1. Install **[.NET Core SDK](https://www.microsoft.com/net/core#windowscmd)**
- The .NET Core SDK does not run on ARM, so you do not need to install this on Raspberry Pi.
1. Install **Dependencies**
1. Install **System Dependencies**
> You can also run under a Python Virtual Environment. See the [Python Virtual Environment Setup](#python-virtual-environment-setup) instructions below for details on how to set that up.
@ -96,108 +150,76 @@ Here's what you need to do to get `iotedgedev` running on your dev machine. If y
sudo pip install --upgrade setuptools pip
sudo apt install python2.7-dev libffi-dev libssl-dev -y
```
1. Install **IoTHub-Explorer**
Currently, the only way to view IoT Hub Messages via CLI is with the iothub-explorer npm package. You need to install that with the following command.
```
npm i -g iothub-explorer
```
You can also use [Device Explorer](https://github.com/Azure/azure-iot-sdk-csharp/releases) as well, but it is Windows only. The Azure IoT team is working on porting the monitor-events functionality to the official az iot CLI - but for now we'll use the npm package.
1. Install **`azure-iot-edge-dev-tool`**
> You do not need to run this on the Raspberry Pi Edge device. See the [Edge Device Setup](#edge-device-setup) section below for more information on setting up your Edge device.
> You do not need to run this on the Raspberry Pi Edge device. See the [IoT Edge Device Setup](#iot-edge-device-setup) section below for more information on setting up your Edge device.
1. Via Pip - Use this if you want to use the tool as is.
```
pip install azure-iot-edge-dev-tool
```
1. Via GitHub - Use this if you want to make changes to this tool.
1. Clone or Fork this Repository
> Replace `project-name` with the name of your project.
`git clone https://github.com/jonbgallant/azure-iot-edge-dev-tool.git project-name`
1. Install Dependencies
> Use 'sudo' for Mac/Linux/RaspberryPi.
`pip install -U -r requirements.txt`
1. Install iotedgedev Module
The following command will make `iotedgedev` available in your terminal.
From the root of repo:
`pip install -e .`
## Usage
Here's how you create a project, build and deploy our modules, and then setup and start the Edge runtime.
Here's how you create a project, build and deploy modules, and then setup and start the Edge runtime.
### Step 1: Create Azure IoT Edge Project
The following command will setup the folder structure required for this module
> Replace `edge-project` with the name of your project. Use `.` to create in the current folder.
> Replace `edgeproject1` with the name of your project. Use `.` to create in the current folder.
```
iotedgedev project --create edge-project
iotedgedev project --create edgeproject1
```
#### Folder Structure
When you create a new project, it will have the following contents:
1. **config folder** - Contains sample config files for both modules and runtime.
1. **build folder** - Contains the files outputted by the .NET Core SDK.
1. **modules folders** - Contains all of the modules for your Edge project.
1. **modules folders** - Contains all of the modules for your IoT Edge project.
- The iotedgedev module assumes that you'll structure your Dockerfiles exactly like the filter-module sample. Have a Docker folder in the root of the project, then subfolders within that to support multiple Docker files.
> It is important that you follow this structure or the module will not work. Please make suggestions or fork this project if you would like a different behavior.
1. **.env** - Contains all the required Environment Variables for your IoT Edge Project.
1. **.gitignore** - A .gitignore file for your IoT Edge projects.
1. **Configuration File Templates** - There are two files in the root of the project:
- `deployment.template.json` - Contains the config that is deployed to your IoT Edge device. It contains references to your modules, module routes and desired property information.
- `runtime.template.json` - Contains the config used by your IoT Edge runtime. It contains your device connection string, your container registry settings and is used when you call the `runtime --setup` command.
1. **.config folder** - All expanded config files are copied to this folder and these files are used at runtime.
1. **build folder** - Contains the files outputted by the .NET Core SDK.
1. **logs folder** - Contains all the Docker log files for the Runtime and your modules.
1. **.env** - Contains all the required Environment Variables for Edge Project. Rename to .env and add your settings.
1. **.gitignore** - A suggested .gitignore file for your Edge projects.
### Step 2: Update Environment Variables
The settings used for this module are stored in a .env file in the root of your project. System or User Environment Variables take precedence over values in .env file.
1. Open `.env` and set variables
1. Runtime Home Directory
You only need to set the `IOTHUB_CONNECTION_STRING` AND `DEVICE_CONNNECTION_STRING` settings, which you can get from the Azure Portal.
- For Linux/Raspberry Pi, change: `RUNTIME_HOME_DIR="/etc/azure-iot-edge"`
1. Active Modules
- You can tell `iotedgedev` which modules you want to build by including them in the `ACTIVE_MODULES` setting. Comma separated.
1. Active Docker Directories
- You can tell `iotedgedev` which Docker files to build and push by including the Dockerfile's parent folder in the `ACTIVE_DOCKER_DIRS` setting.
1. At a minimum, you need to set the following values, which you you can get from your Azure account:
```
IOTHUB_NAME="iot hub name"
IOTHUB_KEY="iot hub key"
DEVICE_CONNECTION_STRING="edge device connection string"
EDGE_DEVICE_ID="edge device id"
RUNTIME_HOST_NAME="the computer name that your edge will run on"
CONTAINER_REGISTRY_SERVER=""
CONTAINER_REGISTRY_USERNAME=""
CONTAINER_REGISTRY_PASSWORD=""
IOTHUB_CONNECTION_STRING=""
DEVICE_CONNECTION_STRING=""
```
1. Update Config
If you are running on Raspberry Pi you need to use the arm32v7 Dockerfile. Open `config/modules.json`, find the `filter-module` line and replace `linux-x64` with `arm32v7`.
Replace this:
`"image": "${CONTAINER_REGISTRY_SERVER}/filter-module:linux-x64-${CONTAINER_TAG}",`
With this:
`"image": "${CONTAINER_REGISTRY_SERVER}/filter-module:arm32v7-${CONTAINER_TAG}",`
> You can use the `DOTENV_FILE` Environment Variable to point to a different .env file, such as .env.integration or .env.test. This is helpful in CI/CD pipeline scenarios where you'll want to target different environments and devices to ensure all scenarios are tested.
### Step 3: Build and Deploy Modules
@ -207,11 +229,11 @@ The settings used for this module are stored in a .env file in the root of your
iotedgedev modules --build --deploy
```
The- `--build` command will build each module in the `modules` folder and push it to your container registry. The- `--deploy` command will apply the `build/modules.json` configuration file to your Edge device.
The- `--build` command will build each module in the `modules` folder and push it to your container registry. The- `--deploy` command will apply the generated `.config/deployment.json` configuration file to your IoT Edge device.
You can configure what modules will be built and deployed using the `ACTIVE_MODULES` env var in the `.env` file.
You can configure what modules will be built and deployed by using the `ACTIVE_MODULES` env var in the `.env` file. You can configure which Dockerfiles get built and deployed by using the `ACTIVE_DOCKER_DIRS` env var.
### Step 4: Setup and Start the Edge Runtime
### Step 4: Setup and Start the IoT Edge Runtime
> Use 'sudo' for Linux/RaspberryPi
@ -219,26 +241,25 @@ You can configure what modules will be built and deployed using the `ACTIVE_MODU
iotedgedev runtime --setup --start
```
The- `--setup` command will apply the `/build/runtime.json` file to your Edge device. The- `--start` command will start the Edge runtime.
The- `--setup` command will apply the `/.config/runtime.json` file to your IoT Edge device. The- `--start` command will start the IoT Edge runtime.
### Step 5: Monitor Messages
#### Device Explorer
```
iotedgedev iothub --monitor-events
```
You can use the [Device Explorer](https://github.com/Azure/azure-iot-sdk-csharp/releases/download/2017-12-2/SetupDeviceExplorer.msi) to monitor the messages that are sent to your IoT Hub.
#### iothub-explorer
You can also use the [iothub-explorer](https://github.com/Azure/iothub-explorer) npm package:
This will print messages sent from the device specified in DEVICE_CONNECTION_STRING. To use this command, you first need to install the [iothub-explorer](https://github.com/Azure/iothub-explorer) npm package with the following command:
```bash
npm i -g iothub-explorer
iothub-explorer --login "iothubowner connection string" monitor-events "your device id"
```
> NOTE: iothub-explorer is included in the [IoT Edge Dev Tool Container](#iot-edge-dev-tool-container)
### Step 6: Create a new Module
After you have everything running from the Edge Tool project template, the next step is to develop all the custom modules you need for your scenario. Here's how you do that:
After you have everything running from the IoT Edge Tool project template, the next step is to develop all the custom modules you need for your scenario. Here's how you do that:
1. Install the .NET Core Module Template
@ -258,8 +279,6 @@ After you have everything running from the Edge Tool project template, the next
Now, when you run `iotedgedev modules --build` you will see that `mymodule` is also built and pushed to your container registry.
> Without `iotedgedev` you would have had to do many more steps to get to this point.
1. Add Message Property
We are going to add a new custom property to the message as it is flowing through the system so we can see that our module is working.
@ -274,7 +293,7 @@ After you have everything running from the Edge Tool project template, the next
1. Add Module to Config
1. Open `/config/modules.json`.
1. Open `deployment.template.json`.
1. Copy and paste the filter-module section and change filter-module to mymodule.
```javascript
@ -295,7 +314,7 @@ After you have everything running from the Edge Tool project template, the next
By default, your new module acts as a simple pass through. It receives messages and passes them to the output. We will add `mymodule` in between our existing `sensor` and `filter` modules.
1. Open `/config/modules.json`
1. Open `deployment.template.json`
1. Replace `$edgeHub.properties.desired.routes` with the following:
```javascript
@ -310,7 +329,7 @@ After you have everything running from the Edge Tool project template, the next
Now that we have the module created, code added, and config updated, we are going to rebuild, deploy our module.
> You will notice that the Edge Runtime automatically detects a new deployment, retrieves the new module, applies the new route and keeps sending messages.
> You will notice that the IoT Edge Runtime automatically detects a new deployment, retrieves the new module, applies the new route and keeps sending messages.
`iotedgedev modules --build --deploy`
@ -318,11 +337,8 @@ After you have everything running from the Edge Tool project template, the next
Now when we view the messages flowing through the system, we'll see an additional 'abc' property:
Here's how with the [iothub-explorer](https://github.com/Azure/iothub-explorer) npm package:
```bash
npm i -g iothub-explorer
iothub-explorer --login "iothubowner connection string" monitor-events "your device id"
iotedgedev iothub --monitor-events
```
```javascript
@ -343,7 +359,7 @@ After you have everything running from the Edge Tool project template, the next
}
```
That's all there is to it. You can now get started implementing your Edge scenario!
That's all there is to it. You can now get started implementing your IoT Edge scenario!
## Commands
The `iotedgedev` module has the following commands:
@ -353,6 +369,11 @@ The `iotedgedev` module has the following commands:
`iotedgedev project --help`
- `--create TEXT` Creates a new Azure IoT Edge project. Use `--create .` to create in current folder. Use `--create TEXT` to create in a subfolder.
**iothub**
`iotedgedev iothub --help`
- `--monitor-events` Displays events that are sent from IoT Hub device to IoT Hub.
**runtime**
`iotedgedev runtime --help`
@ -366,7 +387,7 @@ The `iotedgedev` module has the following commands:
`iotedgedev modules --help`
- `--build` Builds and pushes modules specified in ACTIVE_MODULES Environment Variable to specified container registry.
- `--deploy` Deploys modules to Edge device using modules.json in build/config directory.
- `--deploy` Deploys modules to Edge device using deployment.json in the /.config directory.
**docker**
@ -387,7 +408,7 @@ The `iotedgedev` module has the following commands:
### Setup Container Registry
You can also use `iotedgedev` to host the Edge runtime from your own Azure Container Registry or a Local Container Registry. Set the `.env` values for your Container Registry and run the following command. It will pull all the Edge containers from Dockerhub, tag them and upload them to the container registry you have specified in `.env`.
You can also use `iotedgedev` to host the IoT Edge runtime from your own Azure Container Registry or a Local Container Registry. Set the `.env` values for your Container Registry and run the following command. It will pull all the IoT Edge containers from Dockerhub, tag them and upload them to the container registry you have specified in `.env`.
> Use 'sudo' for Linux/RaspberryPi
@ -399,7 +420,7 @@ iotedgedev docker --setup-registry
### View Docker Logs
#### Show Logs
The iotedgedev module also include a "Show Logs" command that will open a new command prompt for each module it finds in your Edge config. Just run the following command:
The iotedgedev module also include a "Show Logs" command that will open a new command prompt for each module it finds in your IoT Edge config. Just run the following command:
> Note: I haven't figured out how to launch new SSH windows in a reliable way. It's in the backlog. For now, you must be on the desktop of the machine to run this command.
@ -435,22 +456,22 @@ Instead of using a cloud based container registry, you can use a local Docker re
`iotedgedev` will look for `localhost` in your setting and take care of the rest for you.
## Edge Device Setup
## IoT Edge Device Setup
The `iotedgedev` module is intended to help with Edge development and doesn't necessarily need to be taken on as a dependency in production or integration environments, where you'll likely want to use the `iotedgectl` module directly. You can use `iotedgedev` to generate your runtime.json file on your dev machine, copy that to your Edge device and then use the following command to setup and start your Edge Runtime.
The `iotedgedev` module is intended to help with IoT Edge development and doesn't necessarily need to be taken on as a dependency in production or integration environments, where you'll likely want to use the `iotedgectl` module directly. You can use `iotedgedev` to generate your runtime.json file on your dev machine, copy that to your IoT Edge device and then use the following command to setup and start your IoT Edge Runtime.
```
iotedgectl setup --config-file runtime.json
iotedgectl start
```
Having said that, there's nothing stopping you from deploying `iotedgedev` to your Edge device. It may be helpful if you want to run the `iotedgedev docker --clean` command to clean up Docker containers and images. Or if you want to run `iotedgedev docker --show-logs` to see all the log files on the device or `iotedgedev docker --save-logs` to output to the LOGS_PATH directory.
Having said that, there's nothing stopping you from deploying `iotedgedev` to your IoT Edge device. It may be helpful if you want to run the `iotedgedev docker --clean` command to clean up Docker containers and images. Or if you want to run `iotedgedev docker --show-logs` to see all the log files on the device or `iotedgedev docker --save-logs` to output to the LOGS_PATH directory.
> Please note that the .NET Core SDK does not support ARM, so you will not be able to run `modules --build` or `modules --deploy` directly on a Raspberry Pi.
### Raspberry Pi
Whether you use `iotedgedev` or directly use `iotedgecgtl` on the Raspberry Pi, you will still need to run the following commands before you run the Edge Runtime.
Whether you use `iotedgedev` or directly use `iotedgecgtl` on the Raspberry Pi, you will still need to run the following commands before you run the IoT Edge Runtime.
```
sudo pip install --upgrade setuptools pip
@ -458,6 +479,15 @@ Whether you use `iotedgedev` or directly use `iotedgecgtl` on the Raspberry Pi,
sudo pip install -U azure-iot-edge-dev-tool
```
##### Config Changes
If you are running on Raspberry Pi you need to use the arm32v7 Dockerfile. Open `deployment.template.json`, find the `filter-module` line and replace `linux-x64` with `arm32v7`.
Replace this:
`"image": "${CONTAINER_REGISTRY_SERVER}/filter-module:linux-x64-${CONTAINER_TAG}",`
With this:
`"image": "${CONTAINER_REGISTRY_SERVER}/filter-module:arm32v7-${CONTAINER_TAG}",`
## Python Virtual Environment Setup
You can run `iotedgedev` inside a Python Vritual Environment.
@ -497,7 +527,7 @@ This module has been tested with the following:
- Windows Subsystem for Linux, Ubuntu 16.04
- Ubuntu 16.04
- Mac Sierra 10.12.6
- Raspberry Pi with Raspbian Stretch (**.NET Core Runtime Only**, .NET Core SDK not supported on ARM.) - You cannot use Raspberry Pi as a Edge dev machine, but it can host the Edge runtime.
- Raspberry Pi with Raspbian Stretch (**.NET Core Runtime Only**, .NET Core SDK not supported on ARM.) - You cannot use Raspberry Pi as a IoT Edge dev machine, but it can host the IoT Edge runtime.
- Python 2.7.13 and Python 3.6.3
- Docker Version 17.09.1-ce-win42 (14687), Channel: stable, 3176a6a
@ -521,7 +551,7 @@ This module has been tested with the following:
1. Latest Docker Image Not Pulled from Registry
By design, Docker only pulls images that have been changed. When you are developing you tend to push new images on a frequent basis, which Docker will not pull unless it has a unique tag. You could assign every push a new tag, or you can simply run the following command on the Edge device.
By design, Docker only pulls images that have been changed. When you are developing you tend to push new images on a frequent basis, which Docker will not pull unless it has a unique tag. You could assign every push a new tag, or you can simply run the following command on the IoT Edge device.
```
iotedgedev runtime --restart
@ -562,6 +592,11 @@ Please use the [GitHub issues page](https://github.com/jonbgallant/azure-iot-edg
Please fork, branch and pull-request any changes you'd like to make.
#### Contributor Dev Machine Setup
1. Clone or Fork this Repository
`git clone https://github.com/jonbgallant/azure-iot-edge-dev-tool.git`
1. Install **[Microsoft Visual C++ Build Tools](http://landinghub.visualstudio.com/visual-cpp-build-tools)**
1. Install **OpenSSL 1.1.0g**
@ -569,7 +604,7 @@ Please fork, branch and pull-request any changes you'd like to make.
1. [Download from OpenSSL's website](https://www.openssl.org/source/openssl-1.1.0g.tar.gz)
1. Extract the downloaded .tar.gz to C:\OpenSSL\
1. Make sure both **Python 2.7 and Python 3** are installed
1. Make sure both **Python 2.7 and Python 3.6** are installed
1. Install dependencies
@ -579,11 +614,19 @@ Please fork, branch and pull-request any changes you'd like to make.
pip install -r requirements_dev.txt
```
#### Run the tests
1. Editable Mode
Run the following command from the root of the project to see changes to iotedgedev commands as you change code.
```
pip install -e .
```
#### Run Tests
Run the following command to run tests.
tox
`tox`

Двоичные данные
assets/edgedevtool2mins.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 29 KiB

После

Ширина:  |  Высота:  |  Размер: 14 KiB

Двоичные данные
assets/edgedevtoolintro.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 29 KiB

После

Ширина:  |  Высота:  |  Размер: 14 KiB

Двоичные данные
assets/edgedevtoolwsl.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 33 KiB

После

Ширина:  |  Высота:  |  Размер: 15 KiB

30
iotedgedev/cli.py
Просмотреть файл

@ -5,13 +5,14 @@ from __future__ import absolute_import
import click
import sys
from .iotedgedev import Docker
from .iotedgedev import Modules
from .iotedgedev import Runtime
from .iotedgedev import Project
from .iotedgedev import Utility
from .iotedgedev import EnvVars
from .iotedgedev import Output
from .dockercls import Docker
from .modules import Modules
from .runtime import Runtime
from .project import Project
from .utility import Utility
from .envvars import EnvVars
from .output import Output
from .iothub import IoTHub
output = Output()
envvars = EnvVars(output)
@ -50,6 +51,18 @@ def project(create):
proj = Project(output)
proj.create(create)
@click.command(context_settings=CONTEXT_SETTINGS)
@click.option(
'--monitor-events',
default=False,
required=False,
is_flag=True,
help="Displays events that are sent from IoT Hub device to IoT Hub.")
def iothub(monitor_events):
if monitor_events:
utility = Utility(envvars, output)
ih = IoTHub(envvars, output, utility)
ih.monitor_events()
@click.command(context_settings=CONTEXT_SETTINGS)
@click.option(
@ -63,7 +76,7 @@ def project(create):
default=False,
required=False,
is_flag=True,
help="Deploys modules to Edge device using modules.json in build/config directory.")
help="Deploys modules to Edge device using deployment.json in the /.config directory.")
def modules(build, deploy):
utility = Utility(envvars, output)
dock = Docker(envvars, utility, output)
@ -215,6 +228,7 @@ main.add_command(runtime)
main.add_command(modules)
main.add_command(docker)
main.add_command(project)
main.add_command(iothub)
if __name__ == "__main__":

6
iotedgedev/connectionstring.py
Просмотреть файл

@ -3,11 +3,15 @@ class ConnectionString:
self.value = value
self.data = dict()
if self.value:
parts = value.split(';')
if len(parts) > 0:
for part in parts:
subpart = part.split('=', 1)
if len(subpart) == 2:
self.data[subpart[0].lower()] = subpart[1].strip()
if self.data:
self.HostName = self["hostname"]
self.SharedAccessKey = self["sharedaccesskey"]
@ -19,6 +23,7 @@ class IoTHubConnectionString(ConnectionString):
def __init__(self, value):
ConnectionString.__init__(self, value)
if self.value:
self.SharedAccessKeyName = self["sharedaccesskeyname"]
@ -26,4 +31,5 @@ class DeviceConnectionString(ConnectionString):
def __init__(self, value):
ConnectionString.__init__(self, value)
if self.value:
self.DeviceId = self["deviceid"]

266
iotedgedev/dockercls.py Normal file
Просмотреть файл

@ -0,0 +1,266 @@
import docker
from enum import Enum
import os
import zipfile
from .moduletype import ModuleType
class Docker:
def __init__(self, envvars, utility, output):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.output = output
if self.envvars.DOCKER_HOST:
self.docker_client = docker.DockerClient(
base_url=self.envvars.DOCKER_HOST)
self.docker_api = docker.APIClient(
base_url=self.envvars.DOCKER_HOST)
else:
self.docker_client = docker.from_env()
self.docker_api = docker.APIClient()
def init_registry(self):
self.output.header("INITIALIZING CONTAINER REGISTRY")
self.output.info("REGISTRY: " + self.envvars.CONTAINER_REGISTRY_SERVER)
if "localhost" in self.envvars.CONTAINER_REGISTRY_SERVER:
self.init_local_registry()
self.login_registry()
self.output.line()
def init_local_registry(self):
parts = self.envvars.CONTAINER_REGISTRY_SERVER.split(":")
if len(parts) < 2:
self.output.error("You must specific a port for your local registry server. Expected: 'localhost:5000'. Found: " +
self.envvars.CONTAINER_REGISTRY_SERVER)
sys.exit()
port = parts[1]
ports = {'{0}/tcp'.format(port): int(port)}
try:
self.output.info("Looking for local 'registry' container")
self.docker_client.containers.get("registry")
self.output.info("Found local 'registry' container")
except docker.errors.NotFound:
self.output.info("Local 'registry' container not found")
try:
self.output.info("Looking for local 'registry' image")
self.docker_client.images.get("registry:2")
self.output.info("Local 'registry' image found")
except docker.errors.ImageNotFound:
self.output.info("Local 'registry' image not found")
self.output.info("Pulling 'registry' image")
self.docker_client.images.pull("registry", tag="2")
self.output.info("Running registry container")
self.docker_client.containers.run(
"registry:2", detach=True, name="registry", ports=ports, restart_policy={"Name": "always"})
def login_registry(self):
try:
if "localhost" in self.envvars.CONTAINER_REGISTRY_SERVER:
client_login_status = self.docker_client.login(
self.envvars.CONTAINER_REGISTRY_SERVER)
api_login_status = self.docker_api.login(
self.envvars.CONTAINER_REGISTRY_SERVER)
else:
client_login_status = self.docker_client.login(registry=self.envvars.CONTAINER_REGISTRY_SERVER,
username=self.envvars.CONTAINER_REGISTRY_USERNAME,
password=self.envvars.CONTAINER_REGISTRY_PASSWORD)
api_login_status = self.docker_api.login(registry=self.envvars.CONTAINER_REGISTRY_SERVER,
username=self.envvars.CONTAINER_REGISTRY_USERNAME,
password=self.envvars.CONTAINER_REGISTRY_PASSWORD)
self.output.info("Successfully logged into container registry: " +
self.envvars.CONTAINER_REGISTRY_SERVER)
except Exception as ex:
self.output.error(
"Could not login to Container Registry. 1. Make sure Docker is running locally. 2. Verify your credentials in CONTAINER_REGISTRY_ environment variables. 2. If you are using WSL, then please set DOCKER_HOST Environment Variable. See the projects readme for full instructions.")
self.output.error(str(ex))
sys.exit(-1)
def setup_registry(self):
self.output.header("SETTING UP CONTAINER REGISTRY")
self.init_registry()
self.output.info("PUSHING EDGE IMAGES TO CONTAINER REGISTRY")
image_names = ["azureiotedge-agent", "azureiotedge-hub",
"azureiotedge-simulated-temperature-sensor"]
for image_name in image_names:
microsoft_image_name = "microsoft/{0}:{1}".format(
image_name, self.envvars.RUNTIME_TAG)
container_registry_image_name = "{0}/{1}:{2}".format(
self.envvars.CONTAINER_REGISTRY_SERVER, image_name, self.envvars.RUNTIME_TAG)
# Pull image from Microsoft Docker Hub
try:
self.output.info(
"PULLING IMAGE: '{0}'".format(microsoft_image_name))
image_pull = self.docker_client.images.pull(
microsoft_image_name)
self.output.info(
"SUCCESSFULLY PULLED IMAGE: '{0}'".format(microsoft_image_name))
self.output.info(str(image_pull))
except docker.errors.APIError as e:
self.output.error(
"ERROR WHILE PULLING IMAGE: '{0}'".format(microsoft_image_name))
self.output.error(e)
# Tagging Image with Container Registry Name
try:
tag_result = self.docker_api.tag(
image=microsoft_image_name, repository=container_registry_image_name)
except docker.errors.APIError as e:
self.output.error(
"ERROR WHILE TAGGING IMAGE: '{0}'".format(microsoft_image_name))
self.output.error(e)
# Push Image to Container Registry
try:
self.output.info("PUSHING IMAGE: '{0}'".format(
container_registry_image_name))
for line in self.docker_client.images.push(repository=container_registry_image_name, tag=self.envvars.RUNTIME_TAG, stream=True, auth_config={"username": self.envvars.CONTAINER_REGISTRY_USERNAME, "password": self.envvars.CONTAINER_REGISTRY_PASSWORD}):
self.output.procout(self.utility.decode(line))
self.output.info("SUCCESSFULLY PUSHED IMAGE: '{0}'".format(
container_registry_image_name))
except docker.errors.APIError as e:
self.output.error("ERROR WHILE PUSHING IMAGE: '{0}'".format(
container_registry_image_name))
self.output.error(e)
self.setup_registry_in_config(image_names)
self.utility.set_config(force=True)
self.output.footer("Container Registry Setup Complete")
def setup_registry_in_config(self, image_names):
self.output.info(
"Replacing 'microsoft/' with '{CONTAINER_REGISTRY_SERVER}/' in config files.")
# Replace microsoft/ with ${CONTAINER_REGISTRY_SERVER}
for config_file in self.utility.get_config_files():
config_file_contents = self.utility.get_file_contents(config_file)
for image_name in image_names:
config_file_contents = config_file_contents.replace(
"microsoft/" + image_name, "${CONTAINER_REGISTRY_SERVER}/" + image_name)
with open(config_file, "w") as config_file_build:
config_file_build.write(config_file_contents)
def remove_modules(self):
self.output.info(
"Removing Edge Modules Containers and Images from Docker")
modules_in_config = self.utility.get_modules_in_config(ModuleType.User)
for module in modules_in_config:
self.output.info("Searching for {0} Containers".format(module))
containers = self.docker_client.containers.list(
filters={"name": module})
for container in containers:
self.output.info("Removing Container: " + str(container))
container.remove(force=True)
self.output.info("Searching for {0} Images".format(module))
for image in self.docker_client.images.list():
if module in str(image):
self.output.info(
"Removing Module Image: " + str(image))
self.docker_client.images.remove(
image=image.id, force=True)
def remove_containers(self):
self.output.info("Removing Containers....")
containers = self.docker_client.containers.list(all=True)
self.output.info("Found {0} Containers".format(len(containers)))
for container in containers:
self.output.info("Removing Container: {0}:{1}".format(
container.id, container.name))
container.remove(force=True)
self.output.info("Containers Removed")
def remove_images(self):
self.output.info("Removing Dangling Images....")
images = self.docker_client.images.list(
all=True, filters={"dangling": True})
self.output.info("Found {0} Images".format(len(images)))
for image in images:
self.output.info("Removing Image: {0}".format(str(image.id)))
self.docker_client.images.remove(image=image.id, force=True)
self.output.info("Images Removed")
self.output.info("Removing Images....")
images = self.docker_client.images.list()
self.output.info("Found {0} Images".format(len(images)))
for image in images:
self.output.info("Removing Image: {0}".format(str(image.id)))
self.docker_client.images.remove(image=image.id, force=True)
self.output.info("Images Removed")
def handle_logs_cmd(self, show, save):
# Create LOGS_PATH dir if it doesn't exist
if save and not os.path.exists(self.envvars.LOGS_PATH):
os.makedirs(self.envvars.LOGS_PATH)
modules_in_config = self.utility.get_modules_in_config(ModuleType.Both)
for module in modules_in_config:
if show:
try:
command = self.envvars.LOGS_CMD.format(module)
os.system(command)
except Exception as ex:
self.output.error(
"Error while trying to open module log '{0}' with command '{1}'. Try iotedgedev docker --save-logs instead.".format(module, command))
self.output.error(str(ex))
if save:
try:
self.utility.exe_proc(["docker", "logs", module, ">",
os.path.join(self.envvars.LOGS_PATH, module + ".log")], True)
except Exception as ex:
self.output.error(
"Error while trying to save module log file '{0}'".format(module))
self.output.error(str(ex))
if save:
self.zip_logs()
def zip_logs(self):
log_files = [os.path.join(self.envvars.LOGS_PATH, f)
for f in os.listdir(self.envvars.LOGS_PATH) if f.endswith(".log")]
zip_path = os.path.join(self.envvars.LOGS_PATH, 'edge-logs.zip')
self.output.info("Creating {0} file".format(zip_path))
zipf = zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED)
for log_file in log_files:
self.output.info("Adding {0} to zip". format(log_file))
zipf.write(log_file)
zipf.close()
self.output.info("Log files successfully saved to: " + zip_path)

138
iotedgedev/envvars.py Normal file
Просмотреть файл

@ -0,0 +1,138 @@
from dotenv import load_dotenv
import os
import platform
import socket
import sys
from .connectionstring import IoTHubConnectionString, DeviceConnectionString
class EnvVars:
def __init__(self, output):
self.output = output
self.checked = False
def load_dotenv(self):
dotenv_file = self.get_dotenv_file()
dotenv_path = os.path.join(os.getcwd(), dotenv_file)
try:
if os.path.exists(dotenv_path):
load_dotenv(dotenv_path)
self.output.info(
"Environment Variables loaded from: {0} ({1})".format(dotenv_file, dotenv_path))
else:
self.output.info(
"{0} file not found on disk. Without a file on disk, you must specify all Environment Variables at the system level. ({1})".format(dotenv_file, dotenv_path))
except Exception as ex:
self.output.error("Error while trying to load .env file: {0}. {1}".format(
dotenv_path, str(ex)))
def get_dotenv_file(self):
default_dotenv_file = ".env"
if not "DOTENV_FILE" in os.environ:
return default_dotenv_file
else:
dotenv_file_from_environ = os.environ["DOTENV_FILE"].strip(
"\"").strip("'")
if dotenv_file_from_environ:
return dotenv_file_from_environ
return default_dotenv_file
def check(self):
if not self.checked:
self.load_dotenv()
try:
try:
self.IOTHUB_CONNECTION_STRING = self.get_envvar(
"IOTHUB_CONNECTION_STRING")
self.IOTHUB_CONNECTION_INFO = IoTHubConnectionString(
self.IOTHUB_CONNECTION_STRING)
except Exception as ex:
self.output.error("Unable to parse IOTHUB_CONNECTION_STRING Environment Variable. Please ensure that you have the right connection string set.")
self.output.error(str(ex))
sys.exit(-1)
try:
self.DEVICE_CONNECTION_STRING = self.get_envvar(
"DEVICE_CONNECTION_STRING")
self.DEVICE_CONNECTION_INFO = DeviceConnectionString(
self.DEVICE_CONNECTION_STRING)
except Exception as ex:
self.output.error("Unable to parse DEVICE_CONNECTION_STRING Environment Variable. Please ensure that you have the right connection string set.")
self.output.error(str(ex))
sys.exit(-1)
self.RUNTIME_HOST_NAME = self.get_envvar("RUNTIME_HOST_NAME")
if self.RUNTIME_HOST_NAME == ".":
self.set_envvar("RUNTIME_HOST_NAME", socket.gethostname())
self.RUNTIME_HOME_DIR = self.get_envvar("RUNTIME_HOME_DIR")
if self.RUNTIME_HOME_DIR == ".":
self.set_envvar("RUNTIME_HOME_DIR", self.get_runtime_home_dir())
self.RUNTIME_CONFIG_DIR = self.get_envvar("RUNTIME_CONFIG_DIR")
if self.RUNTIME_CONFIG_DIR == ".":
self.set_envvar("RUNTIME_CONFIG_DIR", self.get_runtime_config_dir())
self.ACTIVE_MODULES = self.get_envvar("ACTIVE_MODULES")
self.ACTIVE_DOCKER_DIRS = self.get_envvar("ACTIVE_DOCKER_DIRS")
self.CONTAINER_REGISTRY_SERVER = self.get_envvar(
"CONTAINER_REGISTRY_SERVER", False)
self.CONTAINER_REGISTRY_USERNAME = self.get_envvar(
"CONTAINER_REGISTRY_USERNAME", False)
self.CONTAINER_REGISTRY_PASSWORD = self.get_envvar(
"CONTAINER_REGISTRY_PASSWORD", False)
self.CONTAINER_TAG = self.get_envvar("CONTAINER_TAG", False)
self.RUNTIME_TAG = self.get_envvar("RUNTIME_TAG")
self.RUNTIME_VERBOSITY = self.get_envvar("RUNTIME_VERBOSITY")
self.MODULES_CONFIG_FILE = self.get_envvar(
"MODULES_CONFIG_FILE")
self.RUNTIME_CONFIG_FILE = self.get_envvar(
"RUNTIME_CONFIG_FILE")
self.LOGS_PATH = self.get_envvar("LOGS_PATH")
self.MODULES_PATH = self.get_envvar("MODULES_PATH")
self.IOT_REST_API_VERSION = self.get_envvar(
"IOT_REST_API_VERSION")
self.DOTNET_VERBOSITY = self.get_envvar("DOTNET_VERBOSITY")
self.LOGS_CMD = self.get_envvar("LOGS_CMD")
if "DOCKER_HOST" in os.environ:
self.DOCKER_HOST = self.get_envvar("DOCKER_HOST", False)
else:
self.DOCKER_HOST = None
except Exception as ex:
self.output.error(
"Environment variables not configured correctly. Run `iotedgedev project --create [name]` to create a new project with sample .env file. Please see README for variable configuration options. Tip: You might just need to restart your command prompt to refresh your Environment Variables.")
self.output.error("Variable that caused exception: " + str(ex))
sys.exit(-1)
self.checked = True
def get_envvar(self, key, required=True):
val = os.environ[key].strip()
if not val and required:
self.output.error("Environment Variable {0} not set. Either add to .env file or to your system's Environment Variables".format(key))
sys.exit(-1)
return val
def set_envvar(self, key, value):
os.environ[key] = value
def get_runtime_home_dir(self):
plat = platform.system().lower()
if plat == "linux" or plat == "darwin":
return "/var/lib/azure-iot-edge"
else:
return os.environ["PROGRAMDATA"].replace("\\", "\\\\") + "\\\\azure-iot-edge\\\data"
def get_runtime_config_dir(self):
plat = platform.system().lower()
if plat == "linux" or plat == "darwin":
return "/etc/azure-iot-edge"
else:
return os.environ["PROGRAMDATA"].replace("\\", "\\\\") + "\\\\azure-iot-edge\\\\config"

785
iotedgedev/iotedgedev.py
Просмотреть файл

@ -1,785 +0,0 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import
import requests
import uuid
import docker
import os
import subprocess
import sys
import json
import click
import zipfile
import socket
import platform
from base64 import b64encode, b64decode
from hashlib import sha256
from time import time
import fnmatch
import inspect
from hmac import HMAC
from shutil import copyfile
from enum import Enum
from distutils.util import strtobool
from dotenv import load_dotenv
from .connectionstring import IoTHubConnectionString, DeviceConnectionString
if sys.version_info.major >= 3:
from urllib.parse import quote, urlencode
else:
from urllib import quote, urlencode
#print("Python Version: " + sys.version)
class Output:
def info(self, text):
click.secho(text, fg='yellow')
def error(self, text):
click.secho(text, fg='red')
def header(self, text):
click.secho("======== {0} ========".format(text).upper(), fg='white')
def footer(self, text):
self.info(text.upper())
self.line()
def procout(self, text):
click.secho(text, dim=True)
def line(self):
click.secho("")
class EnvVars:
def __init__(self, output):
self.output = output
self.checked = False
def load_dotenv(self):
dotenv_file = self.get_dotenv_file()
dotenv_path = os.path.join(os.getcwd(), dotenv_file)
try:
if os.path.exists(dotenv_path):
load_dotenv(dotenv_path)
self.output.info(
"Environment Variables loaded from: {0} ({1})".format(dotenv_file, dotenv_path))
else:
self.output.info(
"{0} file not found on disk. Without a file on disk, you must specify all Environment Variables at the system level. ({1})".format(dotenv_file, dotenv_path))
except Exception as e:
self.output.error("ERROR: Error while trying to load .env file: {0}. {1}".format(
dotenv_path, str(e)))
def get_dotenv_file(self):
default_dotenv_file = ".env"
if not "DOTENV_FILE" in os.environ:
return default_dotenv_file
else:
dotenv_file_from_environ = os.environ["DOTENV_FILE"].strip(
"\"").strip("'")
if dotenv_file_from_environ:
return dotenv_file_from_environ
return default_dotenv_file
def check(self):
if not self.checked:
self.load_dotenv()
try:
try:
self.IOTHUB_CONNECTION_STRING = self.get_envvar(
"IOTHUB_CONNECTION_STRING")
self.IOTHUB_CONNECTION_INFO = IoTHubConnectionString(
self.IOTHUB_CONNECTION_STRING)
except:
self.output.error("ERROR: Unable to parse IOTHUB_CONNECTION_STRING Environment Variable. Please ensure that you have the right connection string set.")
sys.exit(-1)
try:
self.DEVICE_CONNECTION_STRING = self.get_envvar(
"DEVICE_CONNECTION_STRING")
self.DEVICE_CONNECTION_INFO = DeviceConnectionString(
self.DEVICE_CONNECTION_STRING)
except:
self.output.error("ERROR: Unable to parse DEVICE_CONNECTION_STRING Environment Variable. Please ensure that you have the right connection string set.")
sys.exit(-1)
self.RUNTIME_HOST_NAME = self.get_envvar("RUNTIME_HOST_NAME")
if self.RUNTIME_HOST_NAME == ".":
self.set_envvar("RUNTIME_HOST_NAME", socket.gethostname())
self.RUNTIME_HOME_DIR = self.get_envvar("RUNTIME_HOME_DIR")
if self.RUNTIME_HOME_DIR == ".":
self.set_envvar("RUNTIME_HOME_DIR", self.get_runtime_home_dir())
self.RUNTIME_CONFIG_DIR = self.get_envvar("RUNTIME_CONFIG_DIR")
if self.RUNTIME_CONFIG_DIR == ".":
self.set_envvar("RUNTIME_CONFIG_DIR", self.get_runtime_config_dir())
self.ACTIVE_MODULES = self.get_envvar("ACTIVE_MODULES")
self.ACTIVE_DOCKER_DIRS = self.get_envvar("ACTIVE_DOCKER_DIRS")
self.CONTAINER_REGISTRY_SERVER = self.get_envvar(
"CONTAINER_REGISTRY_SERVER")
self.CONTAINER_REGISTRY_USERNAME = self.get_envvar(
"CONTAINER_REGISTRY_USERNAME")
self.CONTAINER_REGISTRY_PASSWORD = self.get_envvar(
"CONTAINER_REGISTRY_PASSWORD")
self.CONTAINER_TAG = self.get_envvar("CONTAINER_TAG")
self.RUNTIME_TAG = self.get_envvar("RUNTIME_TAG")
self.RUNTIME_VERBOSITY = self.get_envvar("RUNTIME_VERBOSITY")
self.MODULES_CONFIG_FILE = self.get_envvar(
"MODULES_CONFIG_FILE")
self.RUNTIME_CONFIG_FILE = self.get_envvar(
"RUNTIME_CONFIG_FILE")
self.LOGS_PATH = self.get_envvar("LOGS_PATH")
self.MODULES_PATH = self.get_envvar("MODULES_PATH")
self.IOT_REST_API_VERSION = self.get_envvar(
"IOT_REST_API_VERSION")
self.DOTNET_VERBOSITY = self.get_envvar("DOTNET_VERBOSITY")
self.LOGS_CMD = self.get_envvar("LOGS_CMD")
if "DOCKER_HOST" in os.environ:
self.DOCKER_HOST = self.get_envvar("DOCKER_HOST")
else:
self.DOCKER_HOST = None
except Exception as e:
self.output.error(
"Environment variables not configured correctly. Run `iotedgedev project --create [name]` to create a new project with sample .env file. Please see README for variable configuration options. Tip: You might just need to restart your command prompt to refresh your Environment Variables.")
self.output.error("Variable that caused exception: " + str(e))
sys.exit(-1)
self.checked = True
def get_envvar(self, key):
return os.environ[key].strip()
def set_envvar(self, key, value):
os.environ[key] = value
def get_runtime_home_dir(self):
plat = platform.system().lower()
if plat == "linux" or plat == "darwin":
return "/var/lib/azure-iot-edge"
else:
return os.environ["PROGRAMDATA"].replace("\\", "\\\\") + "\\\\azure-iot-edge\\\data"
def get_runtime_config_dir(self):
plat = platform.system().lower()
if plat == "linux" or plat == "darwin":
return "/etc/azure-iot-edge"
else:
return os.environ["PROGRAMDATA"].replace("\\", "\\\\") + "\\\\azure-iot-edge\\\\config"
class Utility:
def __init__(self, envvars, output):
self.envvars = envvars
self.envvars.check()
self.output = output
self.config_set = False
def exe_proc(self, params, shell=False):
proc = subprocess.Popen(
params, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
stdout_data, stderr_data = proc.communicate()
if stdout_data != "":
self.output.procout(self.decode(stdout_data))
if proc.returncode != 0:
self.output.error(self.decode(stderr_data))
sys.exit()
def find_files(self, directory, pattern):
# find all files in directory that match the pattern.
for root, dirs, files in os.walk(directory):
for basename in files:
if fnmatch.fnmatch(basename, pattern):
filename = os.path.join(root, basename)
yield filename
def get_iot_hub_sas_token(self, uri, key, policy_name, expiry=3600):
ttl = time() + expiry
sign_key = "%s\n%d" % ((quote(uri)), int(ttl))
signature = b64encode(
HMAC(b64decode(key), sign_key.encode("utf-8"), sha256).digest())
rawtoken = {
"sr": uri,
"sig": signature,
"se": str(int(ttl))
}
if policy_name is not None:
rawtoken["skn"] = policy_name
return "SharedAccessSignature " + urlencode(rawtoken)
def get_file_contents(self, file):
with open(file, "r") as file:
return file.read()
def decode(self, val):
return val.decode("utf-8").strip()
def get_config_files(self):
# config files are in root of project
return [os.path.join(os.getcwd(), f) for f in os.listdir(os.getcwd()) if f.endswith("template.json")]
def get_active_modules(self):
return [module.strip()
for module in self.envvars.ACTIVE_MODULES.split(",") if module]
def get_modules_in_config(self, moduleType):
modules_config = json.load(open(self.envvars.MODULES_CONFIG_FILE))
props = modules_config["moduleContent"]["$edgeAgent"]["properties.desired"]
system_modules = props["systemModules"]
user_modules = props["modules"]
if moduleType == ModuleType.System:
return system_modules
elif moduleType == ModuleType.User:
return user_modules
else:
return_modules = {}
return_modules.update(system_modules)
return_modules.update(user_modules)
return return_modules
def set_config(self, force=False):
if not self.config_set or force:
self.envvars.check()
self.output.header("PROCESSING CONFIG FILES")
config_output_dir = ".config"
# Create config dir if it doesn't exist
if not os.path.exists(config_output_dir):
os.makedirs(config_output_dir)
config_files = self.get_config_files()
if len(config_files) == 0:
self.output.info(
"Unable to find config files in project root directory")
sys.exit()
# Expand envars and rewrite to .config/
for config_file in config_files:
build_config_file = os.path.join(
config_output_dir, os.path.basename(config_file).replace(".template", ""))
self.output.info("Expanding '{0}' to '{1}'".format(
os.path.basename(config_file), build_config_file))
config_file_expanded = os.path.expandvars(
self.get_file_contents(config_file))
with open(build_config_file, "w") as config_file_build:
config_file_build.write(config_file_expanded)
self.output.line()
self.config_set = True
class Project:
def __init__(self, output):
self.output = output
def create(self, name):
self.output.header("CREATING AZURE IOT EDGE PROJECT")
try:
template_zip = os.path.join(os.path.split(
__file__)[0], "template", "template.zip")
except Exception as e:
self.output.error("Error while trying to load template.zip")
self.output.error(e)
if name == ".":
name = ""
zipf = zipfile.ZipFile(template_zip)
zipf.extractall(name)
os.rename(os.path.join(name, ".env.tmp"), os.path.join(name, ".env"))
self.output.footer("Azure IoT Edge project created")
class Runtime:
def __init__(self, envvars, utility, output, dock):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.dock = dock
self.output = output
def start(self):
self.output.header("Starting Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose",
self.envvars.RUNTIME_VERBOSITY, "start"])
def stop(self):
self.output.header("Stopping Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose",
self.envvars.RUNTIME_VERBOSITY, "stop"])
def setup(self):
self.output.header("Setting Up Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose", self.envvars.RUNTIME_VERBOSITY,
"setup", "--config-file", self.envvars.RUNTIME_CONFIG_FILE])
def status(self):
self.output.header("Getting Edge Runtime Status")
self.utility.exe_proc(["iotedgectl", "--verbose", self.envvars.RUNTIME_VERBOSITY,
"status"])
def restart(self):
self.stop()
self.dock.remove_modules()
self.setup()
self.start()
class Modules:
def __init__(self, envvars, utility, output, dock):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.output = output
self.dock = dock
self.dock.init_registry()
def build(self):
self.output.header("BUILDING MODULES")
# Get all the modules to build as specified in config.
modules_to_process = self.utility.get_active_modules()
for module in os.listdir(self.envvars.MODULES_PATH):
if len(
modules_to_process) == 0 or modules_to_process[0] == "*" or module in modules_to_process:
module_dir = os.path.join(self.envvars.MODULES_PATH, module)
self.output.info("BUILDING MODULE: {0}".format(module_dir))
# Find first proj file in module dir and use it.
project_files = [os.path.join(module_dir, f) for f in os.listdir(
module_dir) if f.endswith("proj")]
if len(project_files) == 0:
self.output.error("No project file found for module.")
continue
self.utility.exe_proc(["dotnet", "build", project_files[0],
"-v", self.envvars.DOTNET_VERBOSITY])
# Get all docker files in project
docker_files = self.utility.find_files(
module_dir, "Dockerfile*")
# Filter by Docker Dirs in envvars
docker_dirs_process = [docker_dir.strip()
for docker_dir in self.envvars.ACTIVE_DOCKER_DIRS.split(",") if docker_dir]
# Process each Dockerfile found
for docker_file in docker_files:
docker_file_parent_folder = os.path.basename(
os.path.dirname(docker_file))
if len(
docker_dirs_process) == 0 or docker_dirs_process[0] == "*" or docker_file_parent_folder in docker_dirs_process:
self.output.info(
"PROCESSING DOCKER FILE: " + docker_file)
docker_file_name = os.path.basename(docker_file)
# assume /Docker/{runtime}/Dockerfile folder structure
# image name will be the same as the module folder name, filter-module
# tag will be {runtime}{ext}{container_tag}, i.e. linux-x64-debug-jong
# runtime is the Dockerfile immediate parent folder name
# ext is Dockerfile extension for example with Dockerfile.debug, debug is the mod
# CONTAINER_TAG is env var
# i.e. when found: filter-module/Docker/linux-x64/Dockerfile.debug and CONTAINER_TAG = jong
# we'll get: filtermodule:linux-x64-debug-jong
runtime = os.path.basename(
os.path.dirname(docker_file))
ext = "" if os.path.splitext(docker_file)[
1] == "" else "-" + os.path.splitext(docker_file)[1][1:]
container_tag = "" if self.envvars.CONTAINER_TAG == "" else "-" + \
self.envvars.CONTAINER_TAG
tag_name = runtime + ext + container_tag
# construct the build output path
build_path = os.path.join(
os.getcwd(), "build", "modules", module, runtime)
if not os.path.exists(build_path):
os.makedirs(build_path)
# dotnet publish
self.output.info(
"PUBLISHING PROJECT: " + project_files[0])
self.utility.exe_proc(["dotnet", "publish", project_files[0], "-f", "netcoreapp2.0",
"-o", build_path, "-v", self.envvars.DOTNET_VERBOSITY])
# copy Dockerfile to publish dir
build_dockerfile = os.path.join(
build_path, docker_file_name)
copyfile(docker_file, build_dockerfile)
image_destination_name = "{0}/{1}:{2}".format(
self.envvars.CONTAINER_REGISTRY_SERVER, module, tag_name).lower()
self.output.info(
"BUILDING DOCKER IMAGE: " + image_destination_name)
# cd to the build output to build the docker image
project_dir = os.getcwd()
os.chdir(build_path)
# BUILD DOCKER IMAGE
build_result = self.dock.docker_client.images.build(
tag=image_destination_name, path=".", dockerfile=docker_file_name)
self.output.info(
"DOCKER IMAGE DETAILS: {0}".format(build_result))
# CD BACK UP
os.chdir(project_dir)
# PUSH TO CONTAINER REGISTRY
self.output.info(
"PUSHING DOCKER IMAGE TO: " + image_destination_name)
for line in self.dock.docker_client.images.push(repository=image_destination_name, tag=tag_name, stream=True, auth_config={
"username": self.envvars.CONTAINER_REGISTRY_USERNAME, "password": self.envvars.CONTAINER_REGISTRY_PASSWORD}):
self.output.procout(self.utility.decode(line))
self.output.footer("BUILD COMPLETE")
def deploy(self):
self.output.header("DEPLOYING MODULES")
self.deploy_device_configuration(
self.envvars.IOTHUB_CONNECTION_INFO.HostName, self.envvars.IOTHUB_CONNECTION_INFO.SharedAccessKey,
self.envvars.DEVICE_CONNECTION_INFO.DeviceId, self.envvars.MODULES_CONFIG_FILE,
self.envvars.IOTHUB_CONNECTION_INFO.SharedAccessKeyName, self.envvars.IOT_REST_API_VERSION)
self.output.footer("DEPLOY COMPLETE")
def deploy_device_configuration(
self, host_name, iothub_key, device_id, config_file, iothub_policy_name, api_version):
resource_uri = host_name
token_expiration_period = 60
deploy_uri = "https://{0}/devices/{1}/applyConfigurationContent?api-version={2}".format(
resource_uri, device_id, api_version)
iot_hub_sas_token = self.utility.get_iot_hub_sas_token(
resource_uri, iothub_key, iothub_policy_name, token_expiration_period)
deploy_response = requests.post(deploy_uri,
headers={
"Authorization": iot_hub_sas_token,
"Content-Type": "application/json"
},
data=self.utility.get_file_contents(
config_file)
)
if deploy_response.status_code == 204:
self.output.info(
"Edge Device configuration successfully deployed to '{0}'.".format(device_id))
else:
self.output.info(deploy_uri)
self.output.info(deploy_response.status_code)
self.output.info(deploy_response.text)
self.output.error(
"There was an error deploying the configuration. Please make sure your IOTHUB_CONNECTION_STRING and DEVICE_CONNECTION_STRING Environment Variables are correct.")
class Docker:
def __init__(self, envvars, utility, output):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.output = output
if self.envvars.DOCKER_HOST:
self.docker_client = docker.DockerClient(
base_url=self.envvars.DOCKER_HOST)
self.docker_api = docker.APIClient(
base_url=self.envvars.DOCKER_HOST)
else:
self.docker_client = docker.from_env()
self.docker_api = docker.APIClient()
def init_registry(self):
self.output.header("INITIALIZING CONTAINER REGISTRY")
self.output.info("REGISTRY: " + self.envvars.CONTAINER_REGISTRY_SERVER)
if "localhost" in self.envvars.CONTAINER_REGISTRY_SERVER:
self.init_local_registry()
self.login_registry()
self.output.line()
def init_local_registry(self):
parts = self.envvars.CONTAINER_REGISTRY_SERVER.split(":")
if len(parts) < 2:
self.output.error("You must specific a port for your local registry server. Expected: 'localhost:5000'. Found: " +
self.envvars.CONTAINER_REGISTRY_SERVER)
sys.exit()
port = parts[1]
ports = {'{0}/tcp'.format(port): int(port)}
try:
self.output.info("Looking for local 'registry' container")
self.docker_client.containers.get("registry")
self.output.info("Found local 'registry' container")
except docker.errors.NotFound:
self.output.info("Local 'registry' container not found")
try:
self.output.info("Looking for local 'registry' image")
self.docker_client.images.get("registry:2")
self.output.info("Local 'registry' image found")
except docker.errors.ImageNotFound:
self.output.info("Local 'registry' image not found")
self.output.info("Pulling 'registry' image")
self.docker_client.images.pull("registry", tag="2")
self.output.info("Running registry container")
self.docker_client.containers.run(
"registry:2", detach=True, name="registry", ports=ports, restart_policy={"Name": "always"})
def login_registry(self):
try:
if "localhost" in self.envvars.CONTAINER_REGISTRY_SERVER:
client_login_status = self.docker_client.login(
self.envvars.CONTAINER_REGISTRY_SERVER)
api_login_status = self.docker_api.login(
self.envvars.CONTAINER_REGISTRY_SERVER)
else:
client_login_status = self.docker_client.login(registry=self.envvars.CONTAINER_REGISTRY_SERVER,
username=self.envvars.CONTAINER_REGISTRY_USERNAME,
password=self.envvars.CONTAINER_REGISTRY_PASSWORD)
api_login_status = self.docker_api.login(registry=self.envvars.CONTAINER_REGISTRY_SERVER,
username=self.envvars.CONTAINER_REGISTRY_USERNAME,
password=self.envvars.CONTAINER_REGISTRY_PASSWORD)
self.output.info("Successfully logged into container registry: " +
self.envvars.CONTAINER_REGISTRY_SERVER)
except Exception as ex:
self.output.error(
"ERROR: Could not login to Container Registry. 1. Make sure Docker is running locally. 2. Verify your credentials in CONTAINER_REGISTRY_ environment variables. 2. If you are using WSL, then please set DOCKER_HOST Environment Variable. See the projects readme for full instructions.")
self.output.error(str(ex))
sys.exit(-1)
def setup_registry(self):
self.output.header("SETTING UP CONTAINER REGISTRY")
self.init_registry()
self.output.info("PUSHING EDGE IMAGES TO CONTAINER REGISTRY")
image_names = ["azureiotedge-agent", "azureiotedge-hub",
"azureiotedge-simulated-temperature-sensor"]
for image_name in image_names:
microsoft_image_name = "microsoft/{0}:{1}".format(
image_name, self.envvars.RUNTIME_TAG)
container_registry_image_name = "{0}/{1}:{2}".format(
self.envvars.CONTAINER_REGISTRY_SERVER, image_name, self.envvars.RUNTIME_TAG)
# Pull image from Microsoft Docker Hub
try:
self.output.info(
"PULLING IMAGE: '{0}'".format(microsoft_image_name))
image_pull = self.docker_client.images.pull(
microsoft_image_name)
self.output.info(
"SUCCESSFULLY PULLED IMAGE: '{0}'".format(microsoft_image_name))
self.output.info(str(image_pull))
except docker.errors.APIError as e:
self.output.error(
"ERROR WHILE PULLING IMAGE: '{0}'".format(microsoft_image_name))
self.output.error(e)
# Tagging Image with Container Registry Name
try:
tag_result = self.docker_api.tag(
image=microsoft_image_name, repository=container_registry_image_name)
except docker.errors.APIError as e:
self.output.error(
"ERROR WHILE TAGGING IMAGE: '{0}'".format(microsoft_image_name))
self.output.error(e)
# Push Image to Container Registry
try:
self.output.info("PUSHING IMAGE: '{0}'".format(
container_registry_image_name))
for line in self.docker_client.images.push(repository=container_registry_image_name, tag=self.envvars.RUNTIME_TAG, stream=True, auth_config={"username": self.envvars.CONTAINER_REGISTRY_USERNAME, "password": self.envvars.CONTAINER_REGISTRY_PASSWORD}):
self.output.procout(self.utility.decode(line))
self.output.info("SUCCESSFULLY PUSHED IMAGE: '{0}'".format(
container_registry_image_name))
except docker.errors.APIError as e:
self.output.error("ERROR WHILE PUSHING IMAGE: '{0}'".format(
container_registry_image_name))
self.output.error(e)
self.setup_registry_in_config(image_names)
self.utility.set_config(force=True)
self.output.footer("Container Registry Setup Complete")
def setup_registry_in_config(self, image_names):
self.output.info(
"Replacing 'microsoft/' with '{CONTAINER_REGISTRY_SERVER}/' in config files.")
# Replace microsoft/ with ${CONTAINER_REGISTRY_SERVER}
for config_file in self.utility.get_config_files():
config_file_contents = self.utility.get_file_contents(config_file)
for image_name in image_names:
config_file_contents = config_file_contents.replace(
"microsoft/" + image_name, "${CONTAINER_REGISTRY_SERVER}/" + image_name)
with open(config_file, "w") as config_file_build:
config_file_build.write(config_file_contents)
def remove_modules(self):
self.output.info(
"Removing Edge Modules Containers and Images from Docker")
modules_in_config = self.utility.get_modules_in_config(ModuleType.User)
for module in modules_in_config:
self.output.info("Searching for {0} Containers".format(module))
containers = self.docker_client.containers.list(
filters={"name": module})
for container in containers:
self.output.info("Removing Container: " + str(container))
container.remove(force=True)
self.output.info("Searching for {0} Images".format(module))
for image in self.docker_client.images.list():
if module in str(image):
self.output.info(
"Removing Module Image: " + str(image))
self.docker_client.images.remove(
image=image.id, force=True)
def remove_containers(self):
self.output.info("Removing Containers....")
containers = self.docker_client.containers.list(all=True)
self.output.info("Found {0} Containers".format(len(containers)))
for container in containers:
self.output.info("Removing Container: {0}:{1}".format(
container.id, container.name))
container.remove(force=True)
self.output.info("Containers Removed")
def remove_images(self):
self.output.info("Removing Dangling Images....")
images = self.docker_client.images.list(
all=True, filters={"dangling": True})
self.output.info("Found {0} Images".format(len(images)))
for image in images:
self.output.info("Removing Image: {0}".format(str(image.id)))
self.docker_client.images.remove(image=image.id, force=True)
self.output.info("Images Removed")
self.output.info("Removing Images....")
images = self.docker_client.images.list()
self.output.info("Found {0} Images".format(len(images)))
for image in images:
self.output.info("Removing Image: {0}".format(str(image.id)))
self.docker_client.images.remove(image=image.id, force=True)
self.output.info("Images Removed")
def handle_logs_cmd(self, show, save):
# Create LOGS_PATH dir if it doesn't exist
if save and not os.path.exists(self.envvars.LOGS_PATH):
os.makedirs(self.envvars.LOGS_PATH)
modules_in_config = self.utility.get_modules_in_config(ModuleType.Both)
for module in modules_in_config:
if show:
try:
command = self.envvars.LOGS_CMD.format(module)
os.system(command)
except Exception as e:
self.output.error(
"Error while trying to open module log '{0}' with command '{1}'. Try iotedgedev docker --save-logs instead.".format(module, command))
self.output.error(e)
if save:
try:
self.utility.exe_proc(["docker", "logs", module, ">",
os.path.join(self.envvars.LOGS_PATH, module + ".log")], True)
except Exception as e:
self.output.error(
"Error while trying to save module log file '{0}'".format(module))
self.output.error(e)
if save:
self.zip_logs()
def zip_logs(self):
log_files = [os.path.join(self.envvars.LOGS_PATH, f)
for f in os.listdir(self.envvars.LOGS_PATH) if f.endswith(".log")]
zip_path = os.path.join(self.envvars.LOGS_PATH, 'edge-logs.zip')
self.output.info("Creating {0} file".format(zip_path))
zipf = zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED)
for log_file in log_files:
self.output.info("Adding {0} to zip". format(log_file))
zipf.write(log_file)
zipf.close()
self.output.info("Log files successfully saved to: " + zip_path)
class ModuleType(Enum):
System = 1
User = 2
Both = 3

13
iotedgedev/iothub.py Normal file
Просмотреть файл

@ -0,0 +1,13 @@
class IoTHub:
def __init__(self, envvars, output, utility):
self.envvars = envvars
self.envvars.check()
self.output = output
self.utility = utility
def monitor_events(self):
try:
self.utility.call_proc(['iothub-explorer', '--login', self.envvars.IOTHUB_CONNECTION_STRING, 'monitor-events', self.envvars.DEVICE_CONNECTION_INFO.DeviceId], shell=True)
except Exception as ex:
self.output.error("Problem while trying to call iothub-explorer. Please ensure that you have installed the iothub-explorer npm package with: npm i -g iothub-explorer.")
self.output.error(str(ex))

165
iotedgedev/modules.py Normal file
Просмотреть файл

@ -0,0 +1,165 @@
import os
import requests
from shutil import copyfile
class Modules:
def __init__(self, envvars, utility, output, dock):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.output = output
self.dock = dock
self.dock.init_registry()
def build(self):
self.output.header("BUILDING MODULES")
# Get all the modules to build as specified in config.
modules_to_process = self.utility.get_active_modules()
for module in os.listdir(self.envvars.MODULES_PATH):
if len(
modules_to_process) == 0 or modules_to_process[0] == "*" or module in modules_to_process:
module_dir = os.path.join(self.envvars.MODULES_PATH, module)
self.output.info("BUILDING MODULE: {0}".format(module_dir))
# Find first proj file in module dir and use it.
project_files = [os.path.join(module_dir, f) for f in os.listdir(
module_dir) if f.endswith("proj")]
if len(project_files) == 0:
self.output.error("No project file found for module.")
continue
self.utility.exe_proc(["dotnet", "build", project_files[0],
"-v", self.envvars.DOTNET_VERBOSITY])
# Get all docker files in project
docker_files = self.utility.find_files(
module_dir, "Dockerfile*")
# Filter by Docker Dirs in envvars
docker_dirs_process = [docker_dir.strip()
for docker_dir in self.envvars.ACTIVE_DOCKER_DIRS.split(",") if docker_dir]
# Process each Dockerfile found
for docker_file in docker_files:
docker_file_parent_folder = os.path.basename(
os.path.dirname(docker_file))
if len(
docker_dirs_process) == 0 or docker_dirs_process[0] == "*" or docker_file_parent_folder in docker_dirs_process:
self.output.info(
"PROCESSING DOCKER FILE: " + docker_file)
docker_file_name = os.path.basename(docker_file)
# assume /Docker/{runtime}/Dockerfile folder structure
# image name will be the same as the module folder name, filter-module
# tag will be {runtime}{ext}{container_tag}, i.e. linux-x64-debug-jong
# runtime is the Dockerfile immediate parent folder name
# ext is Dockerfile extension for example with Dockerfile.debug, debug is the mod
# CONTAINER_TAG is env var
# i.e. when found: filter-module/Docker/linux-x64/Dockerfile.debug and CONTAINER_TAG = jong
# we'll get: filtermodule:linux-x64-debug-jong
runtime = os.path.basename(
os.path.dirname(docker_file))
ext = "" if os.path.splitext(docker_file)[
1] == "" else "-" + os.path.splitext(docker_file)[1][1:]
container_tag = "" if self.envvars.CONTAINER_TAG == "" else "-" + \
self.envvars.CONTAINER_TAG
tag_name = runtime + ext + container_tag
# construct the build output path
build_path = os.path.join(
os.getcwd(), "build", "modules", module, runtime)
if not os.path.exists(build_path):
os.makedirs(build_path)
# dotnet publish
self.output.info(
"PUBLISHING PROJECT: " + project_files[0])
self.utility.exe_proc(["dotnet", "publish", project_files[0], "-f", "netcoreapp2.0",
"-o", build_path, "-v", self.envvars.DOTNET_VERBOSITY])
# copy Dockerfile to publish dir
build_dockerfile = os.path.join(
build_path, docker_file_name)
copyfile(docker_file, build_dockerfile)
image_destination_name = "{0}/{1}:{2}".format(
self.envvars.CONTAINER_REGISTRY_SERVER, module, tag_name).lower()
self.output.info(
"BUILDING DOCKER IMAGE: " + image_destination_name)
# cd to the build output to build the docker image
project_dir = os.getcwd()
os.chdir(build_path)
# BUILD DOCKER IMAGE
build_result = self.dock.docker_client.images.build(
tag=image_destination_name, path=".", dockerfile=docker_file_name)
self.output.info(
"DOCKER IMAGE DETAILS: {0}".format(build_result))
# CD BACK UP
os.chdir(project_dir)
# PUSH TO CONTAINER REGISTRY
self.output.info(
"PUSHING DOCKER IMAGE TO: " + image_destination_name)
for line in self.dock.docker_client.images.push(repository=image_destination_name, tag=tag_name, stream=True, auth_config={
"username": self.envvars.CONTAINER_REGISTRY_USERNAME, "password": self.envvars.CONTAINER_REGISTRY_PASSWORD}):
self.output.procout(self.utility.decode(line))
self.output.footer("BUILD COMPLETE")
def deploy(self):
self.output.header("DEPLOYING MODULES")
self.deploy_device_configuration(
self.envvars.IOTHUB_CONNECTION_INFO.HostName, self.envvars.IOTHUB_CONNECTION_INFO.SharedAccessKey,
self.envvars.DEVICE_CONNECTION_INFO.DeviceId, self.envvars.MODULES_CONFIG_FILE,
self.envvars.IOTHUB_CONNECTION_INFO.SharedAccessKeyName, self.envvars.IOT_REST_API_VERSION)
self.output.footer("DEPLOY COMPLETE")
def deploy_device_configuration(
self, host_name, iothub_key, device_id, config_file, iothub_policy_name, api_version):
resource_uri = host_name
token_expiration_period = 60
deploy_uri = "https://{0}/devices/{1}/applyConfigurationContent?api-version={2}".format(
resource_uri, device_id, api_version)
iot_hub_sas_token = self.utility.get_iot_hub_sas_token(
resource_uri, iothub_key, iothub_policy_name, token_expiration_period)
deploy_response = requests.post(deploy_uri,
headers={
"Authorization": iot_hub_sas_token,
"Content-Type": "application/json"
},
data=self.utility.get_file_contents(
config_file)
)
if deploy_response.status_code == 204:
self.output.info(
"Edge Device configuration successfully deployed to '{0}'.".format(device_id))
else:
self.output.info(deploy_uri)
self.output.info(deploy_response.status_code)
self.output.info(deploy_response.text)
self.output.error(
"There was an error deploying the configuration. Please make sure your IOTHUB_CONNECTION_STRING and DEVICE_CONNECTION_STRING Environment Variables are correct.")

6
iotedgedev/moduletype.py Normal file
Просмотреть файл

@ -0,0 +1,6 @@
from enum import Enum
class ModuleType(Enum):
System = 1
User = 2
Both = 3

22
iotedgedev/output.py Normal file
Просмотреть файл

@ -0,0 +1,22 @@
import click
class Output:
def info(self, text):
click.secho(text, fg='yellow')
def error(self, text):
click.secho("ERROR: " + text, fg='red')
def header(self, text):
click.secho("======== {0} ========".format(text).upper(), fg='white')
def footer(self, text):
self.info(text.upper())
self.line()
def procout(self, text):
click.secho(text, dim=True)
def line(self):
click.secho("")

25
iotedgedev/project.py Normal file
Просмотреть файл

@ -0,0 +1,25 @@
import os
import zipfile
class Project:
def __init__(self, output):
self.output = output
def create(self, name):
self.output.header("CREATING AZURE IOT EDGE PROJECT")
try:
template_zip = os.path.join(os.path.split(
__file__)[0], "template", "template.zip")
except Exception as ex:
self.output.error("Error while trying to load template.zip")
self.output.error(str(ex))
if name == ".":
name = ""
zipf = zipfile.ZipFile(template_zip)
zipf.extractall(name)
os.rename(os.path.join(name, ".env.tmp"), os.path.join(name, ".env"))
self.output.footer("Azure IoT Edge project created")

35
iotedgedev/runtime.py Normal file
Просмотреть файл

@ -0,0 +1,35 @@
class Runtime:
def __init__(self, envvars, utility, output, dock):
self.envvars = envvars
self.envvars.check()
self.utility = utility
self.utility.set_config()
self.dock = dock
self.output = output
def start(self):
self.output.header("Starting Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose",
self.envvars.RUNTIME_VERBOSITY, "start"])
def stop(self):
self.output.header("Stopping Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose",
self.envvars.RUNTIME_VERBOSITY, "stop"])
def setup(self):
self.output.header("Setting Up Edge Runtime")
self.utility.exe_proc(["iotedgectl", "--verbose", self.envvars.RUNTIME_VERBOSITY,
"setup", "--config-file", self.envvars.RUNTIME_CONFIG_FILE])
def status(self):
self.output.header("Getting Edge Runtime Status")
self.utility.exe_proc(["iotedgectl", "--verbose", self.envvars.RUNTIME_VERBOSITY,
"status"])
def restart(self):
self.stop()
self.dock.remove_modules()
self.setup()
self.start()

1
iotedgedev/template/.gitignore поставляемый
Просмотреть файл

@ -7,3 +7,4 @@ build
.env
venv
logs
.config

Двоичные данные
iotedgedev/template/template.zip
Просмотреть файл

Двоичный файл не отображается.

135
iotedgedev/utility.py Normal file
Просмотреть файл

@ -0,0 +1,135 @@
from base64 import b64encode, b64decode
import fnmatch
from hashlib import sha256
from hmac import HMAC
import json
import os
import requests
import subprocess
import sys
from time import time
if sys.version_info.major >= 3:
from urllib.parse import quote, urlencode
else:
from urllib import quote, urlencode
from .moduletype import ModuleType
class Utility:
def __init__(self, envvars, output):
self.envvars = envvars
self.envvars.check()
self.output = output
self.config_set = False
def exe_proc(self, params, shell=False):
proc = subprocess.Popen(
params, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
stdout_data, stderr_data = proc.communicate()
if stdout_data != "":
self.output.procout(self.decode(stdout_data))
if proc.returncode != 0:
self.output.error(self.decode(stderr_data))
sys.exit()
def call_proc(self, params, shell=False):
subprocess.check_call(params, shell=shell)
def find_files(self, directory, pattern):
# find all files in directory that match the pattern.
for root, dirs, files in os.walk(directory):
for basename in files:
if fnmatch.fnmatch(basename, pattern):
filename = os.path.join(root, basename)
yield filename
def get_iot_hub_sas_token(self, uri, key, policy_name, expiry=3600):
ttl = time() + expiry
sign_key = "%s\n%d" % ((quote(uri)), int(ttl))
signature = b64encode(
HMAC(b64decode(key), sign_key.encode("utf-8"), sha256).digest())
rawtoken = {
"sr": uri,
"sig": signature,
"se": str(int(ttl))
}
if policy_name is not None:
rawtoken["skn"] = policy_name
return "SharedAccessSignature " + urlencode(rawtoken)
def get_file_contents(self, file):
with open(file, "r") as file:
return file.read()
def decode(self, val):
return val.decode("utf-8").strip()
def get_config_files(self):
# config files are in root of project
return [os.path.join(os.getcwd(), f) for f in os.listdir(os.getcwd()) if f.endswith("template.json")]
def get_active_modules(self):
return [module.strip()
for module in self.envvars.ACTIVE_MODULES.split(",") if module]
def get_modules_in_config(self, moduleType):
modules_config = json.load(open(self.envvars.MODULES_CONFIG_FILE))
props = modules_config["moduleContent"]["$edgeAgent"]["properties.desired"]
system_modules = props["systemModules"]
user_modules = props["modules"]
if moduleType == ModuleType.System:
return system_modules
elif moduleType == ModuleType.User:
return user_modules
else:
return_modules = {}
return_modules.update(system_modules)
return_modules.update(user_modules)
return return_modules
def set_config(self, force=False):
if not self.config_set or force:
self.envvars.check()
self.output.header("PROCESSING CONFIG FILES")
config_output_dir = ".config"
# Create config dir if it doesn't exist
if not os.path.exists(config_output_dir):
os.makedirs(config_output_dir)
config_files = self.get_config_files()
if len(config_files) == 0:
self.output.info(
"Unable to find config files in project root directory")
sys.exit()
# Expand envars and rewrite to .config/
for config_file in config_files:
build_config_file = os.path.join(
config_output_dir, os.path.basename(config_file).replace(".template", ""))
self.output.info("Expanding '{0}' to '{1}'".format(
os.path.basename(config_file), build_config_file))
config_file_expanded = os.path.expandvars(
self.get_file_contents(config_file))
with open(build_config_file, "w") as config_file_build:
config_file_build.write(config_file_expanded)
self.output.line()
self.config_set = True

25
package.json Normal file
Просмотреть файл

@ -0,0 +1,25 @@
{
"name": "azure-iot-edge-dev-tool",
"version": "1.0.0",
"description": "The **Azure IoT Edge Dev Tool** greatly simplifies [Azure IoT Edge](https://azure.microsoft.com/en-us/services/iot-edge/) development down to simple CLI commands driven by Environment Variables.",
"main": "index.js",
"directories": {
"test": "tests"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/jonbgallant/azure-iot-edge-dev-tool.git"
},
"author": "",
"license": "MIT",
"bugs": {
"url": "https://github.com/jonbgallant/azure-iot-edge-dev-tool/issues"
},
"homepage": "https://github.com/jonbgallant/azure-iot-edge-dev-tool#readme",
"dependencies": {
"iothub-explorer": "^1.1.20"
}
}

5
tests/test_iotedgedev.py
Просмотреть файл

@ -9,7 +9,6 @@ import os
import shutil
from click.testing import CliRunner
from iotedgedev import iotedgedev
from iotedgedev import cli
from dotenv import load_dotenv, find_dotenv
@ -34,8 +33,8 @@ class TestIotedgedev(unittest.TestCase):
shutil.copyfile('.env', os.path.join(os.getcwd(), project, '.env'))
os.chdir(project)
except Exception as e:
print(e)
except Exception as ex:
print(str(ex))
@classmethod
def tearDownClass(self):