Document Azure Monitor-backed observability for starter kit (#783)

* Document Azure Monitor-backed observability for starter kit

* Describe conditional deployment of Log Analytics

* Describe prometheus metrics on Network Server

* Update docs/user-guide/devguide.md

Co-authored-by: Roel Fauconnier <roel.fauconnier@microsoft.com>

* Update docs/user-guide/devguide.md

Co-authored-by: Mikhail Chatillon <chmikhai@microsoft.com>

* Add link to metrics collector module

Co-authored-by: Roel Fauconnier <roel.fauconnier@microsoft.com>
Co-authored-by: Mikhail Chatillon <chmikhai@microsoft.com>

docs/quickstart.md

@@ -22,6 +22,8 @@ The template will deploy in your Azure subscription the Following resources:
 - [IoT Hub](https://azure.microsoft.com/en-us/services/iot-hub/)
 - [Azure Function](https://azure.microsoft.com/en-us/services/functions/)
 - [Redis Cache](https://azure.microsoft.com/en-us/services/cache/)
+- [Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview)
+- [Log Analytics](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-overview) (when opted in to use Azure Monitor)
 
 ## Step-by-step instructions
 
@@ -41,6 +43,7 @@ The template will deploy in your Azure subscription the Following resources:
     - **Deploy Device** - Do you want demo end devices to be already provisioned (one using OTAA and one using ABP)? If yes set this to true, the code located in the [Arduino folder](https://github.com/Azure/iotedge-lorawan-starterkit/tree/dev/Arduino) would be ready to use immediately.
     - **Reset pin** - The reset pin of your gateway (the value should be 7 for the Seed Studio LoRaWam, 25 for the IC880A)
     - **Region** - In what region are you operating your device (currently only EU868 and US915 is supported)
+    - **useAzureMonitorOnEdge** - You can opt out of using Azure Monitor services for observability on IoT Edge.
 
     The deployment would take c.a. 10 minutes to complete.
 

docs/user-guide/devguide.md

@@ -69,6 +69,7 @@ If you want to update a LoRa Gateway running a previous version fo our software
 - [Azure Container registry](https://azure.microsoft.com/en-us/services/container-registry/)
 - [Azure Functions](https://azure.microsoft.com/en-us/services/functions/)
 - [Redis Cache](https://azure.microsoft.com/en-us/services/cache/)
+- [Azure Monitor](https://docs.microsoft.com/en-us/azure/azure-monitor/overview)
 
 ### Prerequisites
 
@@ -300,6 +301,21 @@ You can even test sending Cloud-2-Device message (e.g. by VSCode right click on
 
 The Arduino example provided above will print the message on the console. Keep in mind that a [LoRaWAN Class A](https://www.thethingsnetwork.org/lorawan/) device will only receive after a transmit, in our case every 30 seconds.
 
+### Observability
+
+We support Azure Monitor for observability of the LoRaWAN starter kit. If you decide to use Azure Monitor, you will need to create an Application Insights instance and a Log Analytics workspace in your subscription. To enable observability, modify the following settings in your `.env` file:
+
+```{bash}
+APPINSIGHTS_INSTRUMENTATIONKEY={appinsight_key}
+IOT_HUB_RESOURCE_ID=/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Devices/IotHubs/{iot_hub_name}
+LOG_ANALYTICS_WORKSPACE_ID={log_analytics_workspace_id}
+LOG_ANALYTICS_SHARED_KEY={log_analytics_shared_key}
+```
+
+Generate a deployment manifest from `deployment_observability.layered.template.json` and deploy it to the edge devices for which you want to apply the observability. The template will set up the [metrics collector module](https://docs.microsoft.com/en-us/azure/iot-edge/how-to-collect-and-transport-metrics?view=iotedge-2020-11&tabs=iothub#metrics-collector-module) on the edge and connect it with your Log Analytics instance. The gateway will connect to your Application Insights instance. Make sure to set the `APPINSIGHTS_INSTRUMENTATIONKEY` before deploying the `deployment.template.lbs.json` solution, if you want to make sure that the gateway can connect to Application Insights. The Application Insights log level will always be the same as the console log level.
+
+The Network Server will always expose metrics in Prometheus format at the path `/metrics`. You can scrape these metrics using the tool of you choice.
+
 ## Debugging in Visual Studio, outside of IoT Edge and Docker
 
 <!-- markdown-link-check-disable -->