This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments
This reference implementation demonstrates how to connect to existing OPC UA servers and publishes JSON encoded telemetry data from these servers in OPC UA "Pub/Sub" format (using a JSON payload) to Azure IoT Hub. All transport protocols supported by the Azure IoTHub client SDK can be used, i.e. HTTPS, AMQP and MQTT.
This application, apart from including an OPC UA *client* for connecting to existing OPC UA servers you have on your network, also includes an OPC UA *server* on port 62222 that can be used to manage what gets published and offers IoTHub direct methods to do the same.
OPC Publisher implements a retry logic to establish connections to endpoints which have not responded to a certain number of keep alive requests, for example if the OPC UA server on this endpoint had a power outage.
For each distinct publishing interval to an OPC UA server it creates a separate subscription over which all nodes with this publishing interval are updated.
OPC Publisher supports batching of the data sent to IoTHub, to reduce network load. This batching is sending a packet to IoTHub only if the configured package size is reached.
This application uses the OPC Foundations's OPC UA reference stack as nuget packages and therefore licensing of their nuget packages apply. Visit https://opcfoundation.org/license/redistributables/1.3/ for the licensing terms.
Depending if you use Docker Linux or Docker Windows containers, there are different configuration files (Dockerfile or Dockerfile.Windows) to use for building the container.
From the root of the repository, in a console, type:
The `-f` option for `docker build` is optional and the default is to use Dockerfile. Docker also support building directly from a git repository, which means you also can build a Linux container by:
Note: if you want to have correct version information, please install [gitversion](https://gitversion.readthedocs.io/en/latest/) and run it with the following command line in the root of the repository: `gitversion . /updateassemblyinfo /ensureassemblyinfo updateassemblyinfofilename opcpublisher/AssemblyInfo.cs`
The easiest way to configure the OPC UA nodes to publish is via configuration file. The configuration file format is documented in `publishednodes.json` in this repository.
Configuration file syntax has changed over time and OPC Publisher still can read old formats, but converts them into the latest format when persisting the configuration.
An example for the format of the configuration file is:
### Configuration to publish events via configuration file
To publish OPC UA events you use the same configuration file as for data changes.
An example for publishing events which are generated by the [SimpleEvents server](https://github.com/OPCFoundation/UA-.NETStandard/tree/master/SampleApplications/Workshop/SimpleEvents/Server), which can be found in the [OPC Foundation repository](https://github.com/OPCFoundation/UA-.NETStandard)
OPC Publisher has an OPC UA Server integrated, which can be accessed on port 62222. If the hostname is `publisher`, then the URI of the endpoint is: `opc.tcp://publisher:62222/UA/Publisher`
When OPC Publisher gets notified about a value change in one of the configured published nodes, it generates a JSON formatted message, which is sent to IoTHub.
The content of this JSON formatted message can be configured via a configuration file. If no configuration file is specified via the `--tc` option a default configuration is used,
The telemetry which is put into the JSON formatted message is a selection of important properties of these objects. If you need more properties, you need to change the OPC Publisher code base.
Typically you specify the IoTHub owner connectionstring only on the first start of the application. The connectionstring will be encrypted and stored in the platforms certificiate store.
On subsequent calls it will be read from there and reused. If you specify the connectionstring on each start, the device which is created for the application in the IoTHub device registry will be removed and recreated each time.
Open the opcpublisher.sln project with Visual Studio 2017, build the solution and publish it. You can start the application in the 'Target directory' you have published to with:
OPC Publisher is ready to be used as a module to run in [Azure IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge) Microsoft's Intelligent Edge framework.
This configuration will configure IoT Edge to start a container named `publisher` with OPC Publisher as the image.
The hostname of the container's system will be set to `publisher`.
OPC Publisher is called with the following command line: `--aa`.
With this option OPC Publisher trusts those OPC UA server's certificates it is connecting to.
You can use any options in the `Cmd` array OPC Publisher supports. The only limitation is the size of the `Container Create Options`supported by IoT Edge.
* If you want to process the output of the OPC Publisher locally with another Edge module, go back to the `Set Modules` page, and go to the `Specify Routes` tab and add a new route looking like (Notice the usage of the output for the OPC publisher):
{
"routes": {
"processingModuleToIoTHub": "FROM /messages/modules/processingModule/outputs/* INTO $upstream",
"opcPublisherToProcessingModule": "FROM /messages/modules/publisher INTO BrokeredEndpoint(\"/modules/processingModule/inputs/input1\")"
* When you have started IoT Edge on your edge device and the docker container `publisher` is started, you can check out the log output of OPC Publisher either by
using `docker logs -f publisher` or by checking the logfile (in our example above `d:\iiotegde\publisher-publisher.log` content or by using the diagnostic tool [here](https://github.com/Azure-Samples/iot-edge-opc-publisher-diagnostics).
The OPC Publisher OPC UA server listens by default on port 62222. To expose this inbound port in a container, you need to use `docker run` option `-p`:
To enable name resolution from within the container to other containers, you need to create a user define docker bridge network and connect the container to this network using the `--network` option.
### Access other systems from within the container
Other containers, can be reached using the parameters described in the "Enable intercontainer nameresolution" paragraph.
If operating system on which docker is hosted is DNS enabled, then accessing all systems which are known by DNS will work..
A problems occurs in a network which does use NetBIOS name resolution. To enable access to other systems (including the one on which docker is hosted) you need to start your container using the `--add-host` option,
which effectevly is adding an entry to the containers host file.
OPC Publisher uses the hostname of the machine is running on for certificate and endpoint generation. docker chooses a random hostname if there is none set by the `-h` option. Here an example to set the internal hostname of the container to `publisher`:
The description in **Using it as a module in Azure IoT Edge** above is the simplest configuration. Very common is that you want to use the configuration files accessible in the host file system.
Here is a set of `Container Create Options`which allow you to achieve this (be aware that the following example is of a deployment using Linux Containers for Windows):
With those options OPC Publisher will read the nodes it should publish from the file `./pn.json`. The container's working directory is set to
`/appdata`at startup (see `./Dockerfile` in the repository) and thus OPC Publisher will read the file `/appdata/pn.json` inside the container to get the configuration.
Without the `--pf` option, OPC Publisher will try to read its default configuration file `./publishednodes.json`.
The log file `publisher-publisher.log` (default name) will be written to `/appdata` and the `CertificateStores` directory will also be created in this directory.
To make all those files available in the host file system the container configuration requires a bind mount volume.
The `d://iiotedge:/appdata` bind will map the directory `/appdata` (which is the current working directory on container startup) to the host directory `d://iiotedge`.
If this is not given, all file data will be not persisted when the container is started again.
If you are running Windows containers, then the syntax of the `Binds`parameter is different. At container startup the working directory is `c:\appdata`.
That means you need to specify the following mapping in the `HostConfig` if you want to put the configuration file in the directory `d:\iiotedge`on the host:
"HostConfig": {
"Binds": [
"d:/iiotedge:c:/appdata"
]
}
If you are running Linux containers on Linux, then the syntax of the `Binds`parameter is again different. At container startup the working directory is `/appdata`.
That means if you specify the following mapping in the `HostConfig`, then the configuration file should reside in the directory `/iiotedge` on the host:
As you know, OPC UA is using X.509 certificates to authenticate OPC UA client and server during establishment of a connection and
to encrypt the communication between the two parties.
OPC Publisher does use certificate stores maintained by the OPC UA stack to manage all certificates.
On startup OPC Publisher checks if there is a certificate for itself (see `InitApplicationSecurityAsync`
in `OpcApplicationConfigurationSecurity.cs`) and creates a self-signed certificate if there is none or if there is not one passed in
via command line options.
Self-signed certificates do not provide any security, since they are not signed by a trusted CA.
OPC Publisher does provide several command line options to:
* retrieve CSR information of the current application certificate used by OPC Publisher
* provision OPC Publisher with a CA signed certificate
* provision OPC Publisher with a new key pair and matching CA signed certificate
* add certificates to the trusted peer or trusted issuer cert store with certificates from an OPC UA application or from a CA
* add a CRL
* remove a certificate from the trusted peer or trusted issuers cert store
All these options allow to pass in parameters via files or base64 encoded strings.
The default store type for all cert stores is the file system. You can change that via command line options. Especially when you run OPC Publisher
in a container, then the persistency of the certificates is important, since the container does not provide persistency. You need to use docker's `-v` option
to persist the certificate stores in the host file system or a docker volume. If you are use a docker volume, you can pass in certificate relevant
data via base64 encoded strings.
If you want to see how a CA signed certificate can be used, please open an issue on this repo and we follow up.
You need to take special care how certificate stores are handled. Especially for the application certificate the runtime
environment has impact and you want to make sure that it is persisted and not created new on each start:
* Running on Windows natively, you can not use an application certificate store of type `Directory`, since the access to the private key fails. Please use the option `--at X509Store` in this case.
* Running as Linux docker container, you can map the certificate stores to the host file system by using the docker run option `-v <hostdirectory>:/appdata`. This will make the certificate persistent over starts.
* Running as Linux docker container and want to use an X509Store for the application certificate, you need to use the docker run option `-v x509certstores:/root/.dotnet/corefx/cryptography/x509stores` and the application option `--at X509Store`
Since both are interdependent and both depend on the configuration of how many nodes are configured to publish, you should ensure that the parameters you are using for:
The `--mq` parameter controls the upper bound of the capacity of the internal queue, which buffers all notifications if a value of an OPC node changes. If OPC Publisher is not able to send messages to IoTHub fast enough,
then this queue buffers those notifications. The parameter sets the number of notifications which can be buffered. If you seen the number of items in this queue increasing in your test runs, you need to:
otherwise you will loose the data values of those OPC node changes. The `--mq` parameter at the same time allows to prevent controlling the upper bound of the memory resources used by OPC Publisher.
The `--si` parameter enforces OPC Publisher to send messages to IoTHub as the specified interval. If there is an IoTHub message size specified via the `--ms` parameter (or by the default value for it),
then a message will be sent either when the message size is reached (in this case the interval is restarted) or when the specified interval time has passed. If you disable the message size by `--ms 0`, OPC Publisher
uses the maximal possible IoTHub message size of 256 kB to batch data.
The `--ms` parameter allows you to enable batching of messages sent to IoTHub. Depending on the protocol you are using, the overhead to send a message to IoTHub is high compared to the actual time of sending the payload.
Before you use OPC Publisher in production scenarios, you need to test the performance and memory under production conditions. You can use the `--di` commandline parameter to specify a interval in seconds,
The default configuration sends data to IoTHub each 10 seconds or when 256 kB of data to ingest is available. This adds a moderate latency of max 10 seconds, but has lowest probablilty of loosing data because of the large message size.
As you see in the diagnostics ouptut there are no OPC node udpates lost (`monitored item notifications enqueue failure`).
When the message size is set to 0 and there is a send interval configured (or the default of 1 second is used), then OPC Publisher does use internally batch data using the maximal supported IoTHub message size, which is 256 kB. As you see in the diagnostic output,
the average message size is 115019 byte. In this configuration we do not loose any OPC node value udpates and compared to the default it adds lower latency.
This configuration sends for each OPC node value change a message to IoTHub. You see the average message size of 234 byte is pretty small. The advantage of this configuration is that OPC Publisher does not add any latency to the ingest data path. The number of
lost OPC node value updates (`monitored item notifications enqueue failure: 44624`) is the highest of all compared configurations, which make this configuration not recommendable for use cases, when a lot of telemetry should be published.
This configuration batches as much OPC node value udpates as possible. The maximum IoTHub message size is 256 kB, which is configured here. There is no send interval requested, which makes the time when data is ingested
completely controlled by the data itself. This configuration has the least probability of loosing any OPC node values and can be used for publishing a high number of nodes.
When using this configuration you need to ensure, that your scenario does not have conditions where high latency is introduced (because the message size of 256 kB is not reached).
If you need to access the OPC UA server in the OPC Publisher, you should ensure that the firewall setting allow access to the port the server is listening on (default: 62222).
As already mentioned above the configuration of nodes to be published can be configured via IoTHub direct methods.
Beyond this OPC Publisher implements a few IoTHub direct method calls, which allow to read:
* general Information
* diagnostic information on OPC sessions, subscriptions and monitored items
* diagnostic information on IoTHub messages and events
* the startup log
* the last 100 lines of the log
In addition to this is implements a direct method to exit the application.
In the following github repos there are tools to [configure the nodes to publish](https://github.com/Azure-Samples/iot-edge-opc-publisher-nodeconfiguration)
and [read the diagnostic information](https://github.com/Azure-Samples/iot-edge-opc-publisher-diagnostics). Both tools are also available as containers in Docker Hub.
# As OPC UA server to start
If you do not have a real OPC UA server, you can use this [sample OPC UA PLC](https://github.com/Azure-Samples/iot-edge-opc-plc) to start. This sample PLC is also available on Docker Hub.
It implements a couple of tags, which generate random data and tags with anomalies and can be extended easily if you need to simulate any tag values.
