remote-monitoring-services-dotnet/asa-manager

Azure Stream Analytics Manager Overview

This service allows the management of Azure Stream Analytics (ASA) jobs, including setting the configuration and starting, stopping, and monitoring their statuses.

The ASA job is supported by two reference data sets: one that defines rules and one that defines device groups. Rules are stored in json format and device groups are stored in csv format.

The rules reference data is generated by the information managed by the Telemetry Microservice.

This microservice transforms telemetry rules into stream processing logic.

The device groups reference data is used to identify which group of rules to apply to an incoming telemetry message. The device groups are managed by the Configuration Microservice and use Azure IoT Hub device twin queries.

Why?

This microservice was built as part of the Azure IoT Remote Monitoring project to provide a generic implementation for an end-to-end IoT solution. More information here.

Features

• Manage ASA jobs
• Export rules defined in the Telemetry Microservice to ASA
• Export device groups defined in the Configuration Microservice for ASA consumption

How to use

Running the service with Docker

You can run the microservice and its dependencies using Docker with the instructions here.

Running the service locally

Prerequisites

1. Deploy Azure Services

This service has dependencies on the following Azure resources. Follow the instructions for Deploy the Azure services.

• Azure Stream Analytics
• Azure Blob Storage
• Azure Iot Hub
  • Note that in order for device groups mapping to update properly Iot Hub needs to be set up with routes to send device twin and lifecycle messages to Event Hub. For more information see below:
    • Device Twin Notifications
    • Device Lifecycle Notifications
• Azure Event Hub
• Azure Cosmos DB

2. Setup Dependencies

This service depends on the following repositories. Run those services from the instruction in their README's in the following order.

1. Storage Adapter Microservice
2. Telemetry Microservice
3. Iot Hub Manager Microservice
4. Configuration Microservice

3. Environment variables required to run the service

In order to run the service, some environment variables need to be created at least once. See specific instructions for IDE or command line setup below for more information. More information on environment variables here.

• PCS_TELEMETRY_DOCUMENTDB_CONNSTRING = { Azure Cosmos DB connection string }
• PCS_TELEMETRY_WEBSERVICE_URL = http://localhost:9004/v1
• PCS_CONFIG_WEBSERVICE_URL = http://localhost:9005/v1
• PCS_IOTHUBMANAGER_WEBSERVICE_URL = http://localhost:9002/v1
• PCS_ASA_DATA_AZUREBLOB_ACCOUNT = { Azure Blob Storage account name }
• PCS_ASA_DATA_AZUREBLOB_KEY = { Azure Blob Storage Key }
• PCS_ASA_DATA_AZUREBLOB_ENDPOINT_SUFFIX = { Azure Blob Storage endpoint suffix, ex. "core.windows.net" }
• PCS_EVENTHUB_CONNSTRING = { Event Hub Connection String for RootManageSharedAccessKey }
• PCS_EVENTHUB_NAME = { Name of Event Hub }
• PCS_TELEMETRY_STORAGE_TYPE = { "tsi" or "cosmosdb", default is "tsi" }

Running the service with Visual Studio

1. Install .NET Core 2.x
2. Install any recent edition of Visual Studio (Windows/MacOS) or Visual Studio Code (Windows/MacOS/Linux).
   • If you already have Visual Studio installed, then ensure you have .NET Core Tools for Visual Studio 2017 installed (Windows only).
   • If you already have VS Code installed, then ensure you have the C# for Visual Studio Code (powered by OmniSharp) extension installed.
3. Open the solution in Visual Studio or VS Code.
4. Ensure the environment variables are set as described in Prerequisites - #3. Environment variables. See Configuration and Environment variables for detailed information for setting these for your enviroment.
5. Start the WebService project (e.g. press F5).
6. Use an HTTP client such as Postman, to exercise the RESTful API.

Running the service from the command line

1. Make sure the prerequisites are set up.
2. Set the required environment variables in your system. More information on environment variables here.
3. Use the scripts in the scripts folder for many frequent tasks:
   • build: compile all the projects and run the tests.
   • compile: compile all the projects.
   • run: compile the projects and run the service. This will prompt for elevated privileges in Windows to run the web service.

Project Structure

This microservice contains the following projects:

• WebService.csproj - C# web service exposing REST interface for ASA management functionality
• WebService.Test.csproj - Unit tests for web services functionality
• Services.csproj - C# assembly containining business logic for interacting with Azure services.
• Services.Test.csproj - Unit tests for services functionality
• DeviceGroupsAgent.csproj - C# assembly containing business logic for querying device group to device id mapping and writing that mapping to Azure Blob Storage
• DeviceGroupsAgent.Test.csproj - Unit tests for device groups agent functionality
• TelemetryRulesAgent.csproj - C# assembly containing business logic for checking whether telemetry rules have changed, and exporting rules to ASA when they change
• TelemetryRulesAgent.Test.csproj - Unit tests for telemetry rules agent functionality
• Solution/scripts - Contains build scripts, docker container creation scripts, and scripts for running the microservice from the command line.
• Solution/scripts/asa - Contains ASA query, UDF functions, device group, rules samples and shell script to convert query and UDF functions into an ARM parameter object.

Updating the Docker image

The scripts folder includes a docker subfolder with the files required to package the service into a Docker image:

• Dockerfile: docker images specifications
• build: build a Docker container and store the image in the local registry
• run: run the Docker container from the image stored in the local registry
• content: a folder with files copied into the image, including the entry point script

Configuration and Environment variables

The service configuration is accessed via ASP.NET Core configuration adapters, and stored in appsettings.ini. The INI format allows to store values in a readable format, with comments.

The configuration also supports references to environment variables, e.g. to import credentials and network details. Environment variables are not mandatory though, you can for example edit appsettings.ini and write credentials directly in the file. Just be careful not sharing the changes, e.g. sending a Pull Request or checking in the changes in git.

The configuration file in the repository references some environment variables that need to be defined. Depending on the OS and the IDE used, there are several ways to manage environment variables.

1. If you're using Visual Studio or Visual Studio for Mac, the environment variables are loaded from the project settings. Right click on WebService, and select Options/Properties, and find the section with the list of env vars. See WebService/Properties/launchSettings.json.
2. Visual Studio Code loads the environment variables from .vscode/launch.json
3. When running the service with Docker or from the command line, the application will inherit environment variables values from the system.
   • This page describes how to setup env vars in Windows. We suggest to edit and execute once the env-vars-setup.cmd script included in the repository. The settings will persist across terminal sessions and reboots.
   • For Linux and MacOS, we suggest to edit and execute env-vars-setup each time, before starting the service. Depending on OS and terminal, there are ways to persist values globally, for more information these pages should help:
     • https://stackoverflow.com/questions/13046624/how-to-permanently-export-a-variable-in-linux
     • https://stackoverflow.com/questions/135688/setting-environment-variables-in-os-x
     • https://help.ubuntu.com/community/EnvironmentVariables

Contributing to the solution

Please follow our contribution guidelines. We love PRs too.

Troubleshooting

It is possible Azure Stream Analytics may crash. If this happens (rules stop evaluating), restart ASA in the portal with the following steps:

1. Navigate to your ASA instance on the Azure Portal. If you deployed an entire Remote Monitoring solution, your instance will be in a resource group with the name of your solution.
2. On the Overview ASA portal page, click "Stop" on the top of the page, then click "Yes" if prompted to confirm.
3. Wait for the job to stop. This may take a few minutes.
4. Once the job is stopped, click "Start"
   1. This will cause a flyout to appear to select your job output start time. If you do not need the job to run on previous data, click "Now", if you want to re-evaluate data that may have been missed, set the start time to the time of the crash.

Feedback

Please enter issues, bugs, or suggestions as GitHub Issues

License

Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT License.