diff --git a/_posts/2016-11-21-iceberg.md b/2016-11-21-iceberg.md similarity index 100% rename from _posts/2016-11-21-iceberg.md rename to 2016-11-21-iceberg.md diff --git a/_posts/2016-06-30-azzimov.md b/_posts/2016-06-30-azzimov.md index 1ad8784..a98b2e9 100644 --- a/_posts/2016-06-30-azzimov.md +++ b/_posts/2016-06-30-azzimov.md @@ -147,7 +147,7 @@ We used the '**echo debconf**' command to remediate to it (See the next snippet) Here is a piece of code of the shell script that we push as a custom extension in the VM : -{% highlight shell %} +{% highlight bash %} sudo add-apt-repository -y ppa:webupd8team/java sudo apt-get -y update echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections diff --git a/_posts/2016-12-07-ingenu.md b/_posts/2016-12-07-ingenu.md new file mode 100644 index 0000000..5b2db04 --- /dev/null +++ b/_posts/2016-12-07-ingenu.md @@ -0,0 +1,435 @@ +--- +layout: post +title: "Azure helps Ingenu customers push IoT device data into the cloud" +author: "Bret Stateham" +author-link: "#" +#author-image: "{{ site.baseurl }}/images/authors/bstateha.jpg" +date: 2016-12-29 +categories: [IoT, Azure App Service] +color: "blue" +#image: "{{ site.baseurl }}/images/2016-12-07-ingenu/0000-ingenu-logo.jpg" #should be ~350px tall +excerpt: Microsoft worked with Ingenu to help customers using Ingenu's RPMA-based "Machine Network" get device data into Azure. +verticals: ["Retail, Consumer Products & Services"] +language: English +--- + +A team of Microsoft and Ingenu employees worked together to create a solution that Ingenu's customers could use to move data from devices on Ingenu's RPMA-based "Machine Network" into Azure. They used existing REST APIs on Ingenu's Intellect platform to pull the data into an Azure IoT Hub using Node.js code running in an Azure Function. From there they demonstrated how Ingenu customers could use Azure Stream Analytics to alert on anomalous data using Azure Event Hubs, persist their message data longterm in an Azure SQL Database, view the data using the Web Apps feature of Azure App Service, and finally report on the data using Power BI Embedded. + +**Core team:** + +- Joshua Builta – Vice President Product Management, Ingenu +- Darrick Hatch – IoT Product Manager, Hardware, Ingenu +- Kishore Chakravadhanula – Senior Product Manager, Ingenu +- Kenneth Sinsuan – Systems Engineer, Ingenu +- Kevin Boyle – Partner Business Evangelist, Microsoft +- Bret Stateham – Senior Technical Evangelist, Microsoft + +## Customer profile + +[Ingenu](http://www.ingenu.com/) is building the first wireless Machine Network, the world’s largest IoT network dedicated to connectivity for machines. Operating on universal spectrum, the company’s RPMA® technology is a proven standard for connecting Internet of Things (IoT) and machine-to-machine (M2M) devices around the world, with more than 35 networks deployed over seven years. + +The Machine Network will have further reach, global range, and longer-lasting battery life than any existing network. It is also future-proof—enabling technology solution providers to maximize their product’s efficiency and longevity, with unparalleled control and visibility. Ingenu is led by a highly experienced team and backed by one of the strongest boards in the industry, including veterans from Verizon and Qualcomm. + +RPMA (Random Phase Multiple Access) technology provides a number of benefits over other wireless communication protocols like Sigfox, LoRa, and others in bandwidth, coverage, capacity, battery life, and more. + +![RPMA vs. the Competition]({{ site.baseurl }}/images/2016-12-07-ingenu/0010-RpmaVsCompetition.jpg) + + + +## Problem statement + +Currently, devices on the "Machine Network" transmit messages over RPMA to an access point within range. The access point then uses a backhaul network to forward those messages over the Internet to Ingenu's Intellect platform. The messages are then stored there for the customer to retrieve via either an HTTP REST API, or AMQP. + +At this time, the Intellect platform does not perform any processing of the messages, it simply stores them for retrieval by the customer. It is up to the customer to retrieve the messages from Intellect via REST or AMQP and perform any further message handling. Our goal was to produce a reference solution that Ingenu customers could use to retrieve messages from Intellect via REST, and forward those messages into an Azure IoT Hub for further processing on the Azure platform. In addition, we wanted to show how those messages could then be stored, analyzed, and reported on, using Azure services like Stream Analytics, SQL Database, Event Hubs, App Service, and Power BI Embedded. + +## Reference solution + +The overall reference solution is comprised of a Arduino as a field device with a DHT11 temperature sensor, an rACM board (Ingenu's RPMA Radio Module and Dev Board), an RPMA access point, a cellular backhaul network, Ingenu's Intellect platform, and a collection of services in Azure: + +![Architecture Diagram]({{ site.baseurl }}/images/2016-12-07-ingenu/0010-ArchitectureDiagram.jpg) + + + +## Field device and rACM + +For the sample application we used an Arduino Uno with a standard DHT11 temperature sensor. However, *any* MCU and sensor could be used. For RPMA radio communications, we connected the Arduino via software serial to the Ingenu rACM (Reference Application Communication Module). The rACM is a board used by Ingenu customers to prototype their solutions. The actual RPMA radio modules themselves can then be purchased from Ingenu, or from a growing list of third-party manufacturers for integration into the final product. + +![Arduino, Sensor and rACM]({{ site.baseurl }}/images/2016-12-07-ingenu/FieldDeviceAndRacm.jpg) + + +In this sample, every 30 seconds, the Arduino reads the current temperature and humidity values from the DHT11 sensor, and sends the value to the rACM over software serial in a string formatted as "**TT_HH**" where "**TT**" is the temperature in Fahrenheit and "**HH**" is the humidity in relative percentage. + +_Code from **[DHT11_to_rACM.ino](https://github.com/BretStateham/Ingenu2Azure/blob/master/arduino/DHT11_to_rACM/DHT11_to_rACM.ino)** Arduino Sketch_ + +```c + if(gotTemp && gotHumidity){ + serial2.print(tempFarenheit, 2); + serial2.print("_"); + serial2.println(humidityPercent, 2); + } +``` + +The rACM has preconfigured code provided by Ingenu that takes any data received via the serial connection, and first appends a `07` message type indicating that it is a "Serial" message, along with a separator, `0d000a` (carriage return, null, line feed), and finally a hex encoded version of the string received from the Arduino via serial. For example, if the Temp and Humidity string from the Arduino was `64.40_29.00`, the hex version would be `36342e34305f32392e303000`. The final hex version of the message would be: + +```text + 070d000a36342e34305f32392e303000 +``` + +Then, it base64 encodes the string: + +```text + Bw0ACjY0LjQwXzI5LjAwAA== +``` + +It sends that payload to the access point: + +![RPMA Access Point in a Box]({{ site.baseurl }}/images/2016-12-07-ingenu/RPMAAccessPointAndTower.jpg) + + +The access point then wraps the message up in JSON with some other system data, as follows: + +```JSON + { + "messageId":"eb095610-be44-11e6-b09a-4fcab088c061", + "messageType":"DatagramUplinkEvent", + "datagramUplinkEvent": + { + "nodeId":"0x00072d97", + "applicationId":24, + "timestamp":1481311380791, + "payload":"Bw0ACjY0LjQwXzI5LjAwAA==" + } + } +``` + +The message is then forwarded by the access point to Ingenu's Intellect services for retrieval by the customer via the Intellect REST or AMQP interfaces. + +## Device ID mapping + +As messages travel from device to Intellect to Azure, we need a way of tracking the device the message originated from. + +Each RPMA radio node has a unique "**MAC Address**" burned into it: + +![Architecture Diagram]({{ site.baseurl }}/images/2016-12-07-ingenu/rACMMacAddress.png) + + +That MAC Address is then used as the "**Node ID**" in the Ingenu Intellect Platform when the device is activated on the Ingenu side. Then, the same MAC Address is used as the "**Device ID**" in the Azure IoT Hub Device Identity Registry. Finally, the IoT Hub "**Device ID**" and "**Primary Connection String**" are persisted in the "**dbo.IoTHubDevicesTable**" in the Azure SQL Database. Currently, the Azure IoT Hub Device Identity creation and dbo.IoTHubDevices insert are done manually, but those steps could be automated as part of a provisioning process. In addition, the code in the Azure Function could first connect to the Azure IoT Hub using a shared access signatures (SAS) policy that had "Registry Read" permissions to read device credentials, and the Azure SQL Database dbo.IoTHubDevices table could be eliminated: + +![Device ID Mapping]({{ site.baseurl }}/images/2016-12-07-ingenu/DeviceIdMapping.png) + + +## Device-to-cloud (uplink) message flow with Azure Functions + +The field device (DHT11 + Arduino + rACM) generates a message on the RPMA network called an "Uplink," or more specifically a "DatagramUplinkEvent." This is analogous to the "device-to-cloud" or "D2C" message on the Azure IoT Hub side. The general message flow from the field device, through Intellect, via the Azure Function and into the Azure IoT Hub is diagrammed below: + +![Architecture Diagram]({{ site.baseurl }}/images/2016-12-07-ingenu/UplinkMessageFlowAnimation.png) + + +### rACM RPMA uplink message + +The access point receives the message from the rACM as described above and forwards it off to the Intellect platform as an "Uplink" message where it waits for the customer to retrieve it. The message looks something like this: + +```JSON + { + "messageId":"eb095610-be44-11e6-b09a-4fcab088c061", + "messageType":"DatagramUplinkEvent", + "datagramUplinkEvent": + { + "nodeId":"0x00072d97", + "applicationId":24, + "timestamp":1481311380791, + "payload":"Bw0ACjY0LjQwXzI5LjAwAA==" + } + } +``` + +### Intellect REST API call with Azure Functions + +The `IntellectRest2IoTHubJs` Azure Function retrieves the message from Intellect via the REST API. The function pulls the next message from the Intellect REST API. It uses the "System Data Unit" (SDU), which is really just the "messageId" of the previous message that has been persisted in the Azure SQL Database. The SDU storage and retrieval isn't shown in the diagram, but it should be understood. The rest call then uses that SDU. The credentials for the REST call and database are stored in Application Settings for the Azure Function app. The message retrieved is the same message that was sent in the previous step. Once the message is received, it converts the payload back from base64 to a hex string, then parses the hex string to retrieve the sensor data: + +_Credentials stored in App Service settings_ + +![Credentials stored in App Service Settings]({{ site.baseurl }}/images/2016-12-07-ingenu/FunctionAppSettings.jpg) + + +_REST call sample URI where `773495f0-be68-11e6-b09a-4fcab088c061` is the last SDU_ + +`https://intellect.ingenu.com/data/v1/receive//data/v1/receive/773495f0-be68-11e6-b09a-4fcab088c061?count=1` + +_Parsing the payload. Code from [IntellectRest2IoTHub/index.js](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenufunctions/IntellectRest2IoTHubJs/index.js)_ + +```JavaScript + function parsePayload(payload){ + var result = {}; + + var payloadhex = Buffer.from(payload,'base64').toString('hex'); + + var startpos = 8; + var trimlen = 2; + var msgtype = payloadhex.substr(0,2); + result.type = msgtype; + result.payloadHex = payloadhex; + switch(msgtype){ + case "08": //Alarm + debugMessage("\n!!!!!!!!!! - Alarm message received\n") + //Not going to bother parsing the alarm data at this time. + break; + case "07": //Serial + debugMessage("\n!!!!!!!!!! - Serial message received\n") + //extract the sensor values from the string. + //Not a great practice, but I'm assuming the first one is temperature in Farenheit + //And the second one is Relative Humidity % + var bodyhex = payloadhex.substr(8,payloadhex.length-(startpos+trimlen)); + var bodytext = hex2a(bodyhex); + var sensors = bodytext.split('_'); + result.bodyHex = bodyhex; + result.bodyText = bodytext; + result.temperature = parseFloat(sensors[0]); + result.humidity = parseFloat(sensors[1]) + break; + default: //Unknown + debugMessage("\n!!!!!!!!!! - Unknown message received\n") + break; + } + + return result; + + } +``` + +### Device ID lookup + +The `IntellectRest2IoTHubJs` extracts the "nodeId" value from the message received via the REST API in the previous step, and then queries the ingenudb.dbo.IoTHubDevices table in the Azure SQL Database for the corresponding Azure IoT Hub device connection string. It passes the nodeId retrieved above in as the `@deviceId` in the query: + +_SQL Query used to retrieve the device connection string. Code from [IntellectRest2IoTHub/index.js](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenufunctions/IntellectRest2IoTHubJs/index.js)_
+ +```sql + SELECT primaryConnectionString from dbo.IoTHubDevices WHERE deviceId = @deviceId +``` + +### IoT Hub connection + +The database returns the IoT Hub primary connection string for the corresponding device, and the `IntellectRest2IoTHubJs` function uses it to connect to the Azure IoT Hub as the device. + +_Connecting to the IoT Hub as the device. Code from [IntellectRest2IoTHub/index.js](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenufunctions/IntellectRest2IoTHubJs/index.js)_ + +```JavaScript + getDeviceConnectionStringFromSQL(deviceId, function (iotHubConString) { + //... + var iotHubClient = clientFromConnectionString(iotHubConString); + //... + } +``` + +### IoT Hub AMQP message + +The `IntellectRest2IoTHubJs` generates a new Azure IoT Hub Device SDK message, connects to the IoT Hub as the device using the connection string retrieved in the previous step, and sends the message to the IoT Hub via AMQP with the temperature data easily retrievable (Humidity was left off for simplicity), then sends the message to the Azure IoT Hub as the device. Finally, once the message has been processed, the function saves the "messageId" from the message as the "LastSDU" in the database for the next round of processing to start from. + +_Sending the message to the IoT Hub as the device. Code from [IntellectRest2IoTHub/index.js](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenufunctions/IntellectRest2IoTHubJs/index.js)_ + +```JavaScript + getDeviceConnectionStringFromSQL(deviceId, function (iotHubConString) { + debugMessage("iotHubConString: " + iotHubConString); + + var iotHubClient = clientFromConnectionString(iotHubConString); + + iotHubClient.open(function(err){ + if (err) { + context.log('Could not connect to IoT Hub: ' + err); + } else { + debugMessage('Connected to IoT Hub'); + var msgpaylod = { + deviceId: deviceId, + messageId: lastSDU, + timestamp: timestampDate.toISOString(), + temperature: payload.temperature + }; + + var message = new Message(JSON.stringify(msgpaylod)); + + debugMessage("Sending message: " + message.getData()); + iotHubClient.sendEvent(message, function(err,res){ + if (err) context.log('IoT Hub send error: ' + err.toString()); + if (res) debugMessage('IoT Hub send status: ' + res.constructor.name); + }); + } + }); + }); +``` + +_Sample JSON generated from the code above_ + +```JSON + { + "deviceId":"0x00072d97", + "messageId":"eb095610-be44-11e6-b09a-4fcab088c061", + "timestamp":"2016-12-09T19:23:00.791Z", + "temperature":64.4 + } +``` + +## Using Stream Analytics for hot and cold path processing + +Azure Stream Analytics is a fantastic tool for processing messages in an Azure IoT Hub. Its simple SQL-like syntax and growing support for various inputs and output make it an ideal solution. In this sample we use it for both the "Cold Path" and "Hot Path" for our data. + +The "Cold Path" implies data that is allowed to "get cold" or "old." In this case, we push *all* of the messages (well, all of the *valid* messages) that we receive into the Azure SQL Database `dbo.Measurement` table. We can then report on that data any way we want for as long as we want. The solution makes no attempt to remove "old" data so theoretically you could use it to report on data that is years old. + +The "Hot Path" implies data that is active, anomalous, interesting, or that just needs to be acted on specifically. In this sample, the Azure Stream Analytics job is pushing any messages where the temperature sensor value is greater than 80 degrees Celsius to the `ingenualerts` event hub so they can be further processed by an Azure Function (more on that later). + +Stream Analytics does all of this work for us very simply using a familiar SQL-like syntax. Here is the query our Stream Analytics job uses: + +_ingenujob Stream Analytics query from "[ingenujob query.sql](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenujob/ingenujob%20query.sql)"_ + +```SQL + -- SELECT ALL messages + -- FROM the iot hub and put them + -- INTO the sql database + SELECT + deviceId, + messageId, + [timestamp] as [timestamp], + temperature + INTO + sqldb + FROM + iothub + + -- SELECT ONLY messages + -- WHERE the temperature is > 80 + -- FROM the iot hub and put them + -- INTO the alerts event hub + SELECT + deviceId, + messageId, + [timestamp] as [timestamp], + temperature + INTO + alerts + FROM + iothub + WHERE temperature > 80 +``` + +## Cloud-to-device (downlink) message flow with Azure Functions + +The messages that are sent to the `ingenualerts` trigger the `EventHub2IntellectRestJs` function. The function pulls the deviceId and messageId from the message retrieved from the `ingenualerts` event hub and uses them to generate an XML "Downlink" message to send via Ingenu Intellect and RPMA back down to the device. The `0301870010` in the downlink message is understood by the default code on the rACM to be an **Alert** and causes the rACM to flash its onboard LED: + +_rACM responding to the alert downlink message_ + +![rACM Responding to Alert Downlink]({{ site.baseurl }}/images/2016-12-07-ingenu/rACMBlink.gif) + + +_Sending the downlink to Intellect. Code from [EventHub2IntellectRestJs/index.js](https://github.com/BretStateham/Ingenu2Azure/blob/master/ingenufunctions/EventHub2IntellectRestJS/index.js)_ + +```JavaScript +module.exports = function (context, myEventHubTrigger) { + context.log('Node.js eventhub trigger function processed work item', myEventHubTrigger); + + var nodeid = myEventHubTrigger ? myEventHubTrigger.deviceid : "0x00072d97"; + var tagid = myEventHubTrigger ? myEventHubTrigger.messageid : uuid.v4(); + + var post_data = "" + tagid + "" + nodeid + "0301870010"; + context.log("Sending\n" + post_data); + + var path = "/data/v1/send"; + var headers = { + Username: GetEnvironmentVariable("IntellectUsername"), + Password: GetEnvironmentVariable("IntellectPassword"), + "Content-type": 'application/xml' + } + var options = { + host: GetEnvironmentVariable("IntellectHost"), + port: 443, + path: path, + method: "POST", + headers: headers + }; + + var data = ''; + + var req = https.request(options, function (res) { + res.on('data', function (d) { + data += d; + context.log("data: " + data); + }); + + res.on('end', function () { + //callback(JSON.parse(data)); + context.log("Response ended."); + }) + }); + + req.on('error', function (e) { + context.log("https request error: " + e); + }) + + + req.write(post_data); + req.end(); + + context.done(); +}; +``` + +_Sample XML downlink payload_ + +```XML + + + b82536c0-be70-11e6-b09a-4fcab088c061 + 0x00072d97 + 0301870010 + + +``` + +## Web Apps + +The Azure web application is used as a "reporting" layer. In this case, it is simply retrieving the "Cold Path" data stored in the ingenudb.dbo.Measurement table and displaying it in the web application. The web app is a Node.js web application with an Angular/Bootstrap front end: + +_The source code for the web app can be found on GitHub in [ingenu2azure/ingenuweb](https://github.com/BretStateham/Ingenu2Azure/tree/master/ingenuweb)_ + +![Azure Web App]({{ site.baseurl }}/images/2016-12-07-ingenu/AzureWebApp.jpg) + + +## Power BI Embedded + +The Microsoft Power BI layer is used to demonstrate how reports, charts, and more can be integrated into Azure IoT solutions for a rich reporting experience. In this sample, it is used to generate the device temperatures chart that is rendered via the Azure web app. + +_The Power BI report and related files can be found on GitHub in [ingenu2azure/ingenucollection](https://github.com/BretStateham/Ingenu2Azure/tree/master/ingenucollection)_ + +![Azure Web App]({{ site.baseurl }}/images/2016-12-07-ingenu/PowerBIEmbeddedChart.jpg) + + +## Solution security + +Security was not a focus during this proof-of-concept project. However, it was built on existing secure infrastructures from Ingenu and Microsoft. The most likely security risk is physical access to the device in the field. That risk can only be mitigated by securing the environment the physical device is placed in. + +Ingenu's RPMA technology has a number of security-related features: + +- Message confidentiality +- Message integrity and replay protection +- Mutual authentication +- Device anonymity +- Authentic firmware upgrades +- Secure multicasts + +In addition, because RPMA is a proprietary network protocol, and specifically does not use the Internet Protocol (IP), it is impervious to IP attacks. You can learn more about RPMA security at [http://www.ingenu.com/technology/rpma/security/](http://http://www.ingenu.com/technology/rpma/security/). + +The only Azure resources that are publicly accessible are the web app and the Power BI Embedded report (via the web app). All other services either run entirely within the Azure datacenter and require authentication to access them, or in the case of the Azure IoT Hub and Azure Event Hubs, they are secured with Shared Access Policies that are kept private. Traffic from the Intellect REST API to the Azure function is secured with HTTPS and is again authenticated with credentials provided by Ingenu. The traffic between the other Azure services all occurs within the Azure datacenter and does not travel across the Internet. + +## Conclusions + +It was surprising how quickly this solution was completed. Once the rACM hardware, test RPMA access point, and Intellect accounts were provisioned, the remainder of the solution was completed very rapidly by leveraging existing sample code from both Ingenu and Microsoft. + +Azure Functions provided the ideal execution environment for code that would typically need to be run by an Ingenu customer on their own hardware. While it would have been possible for the Azure Function to place the data received from the Intellect REST API directly into the database, and to alert on anomalous data itself, we made a conscious decision to use IoT Hub as a message ingestion layer for all messages. The reasoning behind this was that it would support messaging from devices with other connectivity capabilities, not just RPMA-based devices on Ingenu's network, plus it would allow a more "PaaS" type configuration of message processing using Stream Analytics, Azure SQL Database, and Event Hubs without requiring those solutions to be manually coded. + +Finally, by leveraging Azure IoT Hub, we are positioned to carry the project further and replace the code in the Azure Functions with a formal Azure IoT Hub field gateway or protocol gateway. + +## Next steps + +Based on the success of this sample, Ingenu is now working with a third-party system integrator, Attunix, to create a field gateway or protocol gateway sample. Future integrations could also possibly include direct AMQP integration between Ingenu's Intellect platform and Azure IoT Hub as well as device provisioning integration between the Intellect and Azure IoT Hub platforms. + +## Additional resources + +[Source code on GitHub](http://aka.ms/ingenu2azure) diff --git a/images/2016-12-07-ingenu/AzureWebApp.jpg b/images/2016-12-07-ingenu/AzureWebApp.jpg new file mode 100644 index 0000000..bee01ee Binary files /dev/null and b/images/2016-12-07-ingenu/AzureWebApp.jpg differ diff --git a/images/2016-12-07-ingenu/FunctionAppSettings.jpg b/images/2016-12-07-ingenu/FunctionAppSettings.jpg new file mode 100644 index 0000000..21235de Binary files /dev/null and b/images/2016-12-07-ingenu/FunctionAppSettings.jpg differ diff --git a/images/2016-12-07-ingenu/PowerBIEmbeddedChart.jpg b/images/2016-12-07-ingenu/PowerBIEmbeddedChart.jpg new file mode 100644 index 0000000..94c3c11 Binary files /dev/null and b/images/2016-12-07-ingenu/PowerBIEmbeddedChart.jpg differ diff --git a/images/authors/bstateha.jpg b/images/authors/bstateha.jpg new file mode 100644 index 0000000..ce2b535 Binary files /dev/null and b/images/authors/bstateha.jpg differ