- This PR adds an event listener to log SDK events. The event listener is guarded by the env var and otherwise should have no effect.
- Also this PR fixes the devguide to specify .net 8.
Local testing is in progress...
## Azure IoT Edge PR checklist:
In preparation for v1.5, this change updates our C# code to .NET 8.0. It also updates our dependency on the IoT SDKs to the latest LTS versions (Microsoft.Azure.Devices.Client 1.36.10, Microsoft.Azure.Devices 1.31.6).
I had to upgrade our dependency on Azure.Storage.Blobs because the device SDK upgraded theirs. This required that I upgrade the code in Edge Agent and some test modules to use newer storage blob libraries that aren't deprecated. I did local testing of the Edge Agent code to ensure that the new calls work as expected.
I updated the buildBranch.sh and runTests.sh scripts. buildBranch.sh was building two different ways: (1) it was doing a standard build of all .NET components from the repo root (`dotnet build -c Release`), and (2) it was publishing each "app" in a manner suitable for including in Docker images. I simplified it to just focus on the second scenario. For the first, I updated dotnet.yaml to call `dotnet test` directly. I simplified runTests.sh to use `dotnet test` instead of `dotnet vstest` (which is being deprecated) because `dotnet test` now supports running pre-built binaries instead of always building the binaries itself. Note that `dotnet test` apparently doesn't support some of the parallelization features that vstest did, but the time difference is minimal and the script logic is vastly simplified.
I updated EdgeHubTriggerCsharp.csproj to specify its own TargetFramework property, rather than including netcoreappVersion.props, because it has dependedencies that don't yet support .NET 8.
I successfully ran the CI Build pipeline with these changes. To test, I ran the end-to-end tests, nested end-to-end tests, connectivity tests, and ISA-95 smoke tests against the new bits.
## Azure IoT Edge PR checklist:
- Updates .NET Standard modules to .NET 6 (required for TLS 1.3 support)
- Removes support for TLS 1.0 and 1.1 and adds support for TLS 1.3. The default setting for edgeHub is to enable both TLS 1.2 and 1.3.
- Removes the unused `min_tls_version` setting from aziot-edged.
Although removing TLS 1.0 and 1.1 is a breaking change for users, these versions are deprecated everywhere and should be removed. Furthermore, current versions of edgeHub already do not support TLS 1.0/1.1 due to being built against a version of openssl that disables TLS 1.0/1.1 support, so in reality edgeHub has already dropped support for TLS 1.0/1.1.
Ubuntu 18.04 went out of support in May 2023. This change updates our build/release pipelines so that they won't produce Ubuntu 18.04 packages. It also removes Ubuntu 18.04 references generally.
_Note:_ We can merge this change to main at any point, but we shouldn't make this change in the release/1.4 branch until after November 30, 2023, which is when we documented that we'll stop producing Ubuntu 18.04 packages.
To test, I ran the CI Build pipeline to confirm that 18.04 packages are not built. Then I ran the end-to-end, nested end-to-end, and ISA-95 smoke test pipelines, and confirmed they run and pass.
## Azure IoT Edge PR checklist:
We are missing instructions on how to set environment variables for Edge Agent and Edge Hub. PR adds portal instructions and link to docs for more information about deployment manifests. The instructions improve the reference link in IoT Edge documentation table of contents and this PR https://github.com/MicrosoftDocs/azure-docs-pr/pull/248363
## Azure IoT Edge PR checklist:
Add a flag in EdgeHub for the interval after which EdgeHub will check for server cert expiry.
Manual tests done:
- Configured device to not use NTP, but set the correct time
- Configured IoT Edge to have a delayed start.
- Deployed EdgeHub with MaxCheckCertExpiryInMs=120s.
- Also deployed SimulatedTemperatureSensor
- Started IoT Edge. EdgeAgent, EdgeHub, SimulatedTemperatureSensor came up and was working fine.
- Restarted device
- When device restarted, set the time to 2018, before IoT Edge started.
- IoT Edge started, and EdgeHub came up. The server cert it obtained had an expiry date in 2018.
- Set the correct time to 2023. EdgeHub detected the expired cert in 120s, and restarted, got the correct cert.
- Simulated temperature sensor was able to continue.
We ship six IoT Edge modules as Docker images: Edge Agent, Edge Hub, API Proxy (for nested edge scenarios), Metrics Collector, simulated temperature sensor (a sample), and Diagnostics (not really a module, used internally by the `iotedge check` command). We also maintain a number of images internally as test modules. Until recently, the structure of our Docker images looked like this:
```
1.x.x-linux-<arch>
+-----------+
| linux/ |
+-| amd64 |
| |[BuildInfo]|
| +-----------+
1.x.x |
+--------+ | +-----------+
| Multi- | | | linux/ |
|platform|--+-| arm64 |
| | | |[BuildInfo]|
+--------+ | +-----------+
|
| +-----------+
| | linux/ |
+-| arm/v7 |
|[BuildInfo]|
+-----------+
```
The "BuildInfo" metadata embedded in the platform-specific images was added more recently. It contained information about the base image to help us detect when a newer version becomes available.
Docker recently replaced the BuildInfo metadata with [provenance attestations](https://docs.docker.com/build/attestations/slsa-provenance/). We updated our builds to begin generating and consuming provenance attestations (#6878, #6922) and, using the recommended docker tooling (specifically, `buildx imagetools create`), it modified the structure of our images:
```
+-------------+
1.x.x-linux-<arch> +--| linux/amd64 |
+--------+ | +-------------+
| Multi- |------+
+-|platform| | +-------------+
| | | +--| Provenance |
| +--------+ +-------------+
|
1.x.x | +-------------+
+--------+ | +--------+ +--| linux/arm64 |
| Multi- | | | Multi- | | +-------------+
|platform|--+-|platform|------+
| | | | | | +-------------+
+--------+ | +--------+ +--| Provenance |
| +-------------+
|
| +--------+ +-------------+
| | Multi- | +--| linux/arm64 |
+-|platform| | +-------------+
| |------+
+--------+ | +-------------+
+--| Provenance |
+-------------+
```
While the change should be transparent to users, it could potentially present a problem if people are relying on specific details of our image structure in their pipelines. We haven't seen any evidence of this, but we did run into some minor problems with our internal tooling for publishing images to mcr.microsoft.com.
Crafting our images more carefully to give us a structure that (1) supports the new provenance attestations and (2) more closely resembles the original structure will provide the most seamless experience. This change updates our build pipelines to produce images with the following structure:
```
1.x.x-linux-<arch>
+--------+
| linux/ |
+-| amd64 |
| | |
| +--------+ +------------+
+-------------| Provenance |
1.x.x | +------------+
+--------+ | +--------+
| Multi- | | | linux/ |
|platform|--+-| arm64 |
| | | | |
+--------+ | +--------+ +------------+
+-------------| Provenance |
| +------------+
| +--------+
| | linux/ |
+-| arm/v7 |
| | |
| +--------+ +------------+
+-------------| Provenance |
+------------+
```
To test, I ran my changes through the CI Build pipeline and confirmed that the resulting images passed the end-to-end tests.
I also set up the release pipelines in a test environment, and confirmed that the pipelines run and produce the expected images:
- Metrics Collector stage/publish
- API Proxy stage/publish
- Core images stage/publish
- Metrics Collector auto-refresh
- Core images auto-refresh
Docker recently introduced [provenance attestation](https://docs.docker.com/build/attestations/slsa-provenance/) in buildx 0.10.0, which broke our multi-arch image builds. To build our multi-arch manifests, we use a really old tool that isn't compatible with the buildx changes. To work around the problem, we disabled provenance attestation in our builds (see f82a7d94f3).
With this change, we re-enable provenance attestation and upgrade our multi-arch build tooling to support it.
I also took this opportunity to refactor our Dockerfiles into a single Dockerfile per module except API Proxy (which structures its x64 and arm images very differently). I initially did this refactoring because I planned to upgrade our pipelines/scripts to the canonical method for building Docker images: passing a comma-separated list of platforms to `docker buildx build` and letting it build the images _and_ the multi-arch manifest at once. This method would have required a single Dockerfile for all architectures. However, since we provide per-architecture tags for our images, it ended up being easier to keep our current pattern (build each single-arch image, then create the multi-arch manifest separately) and just upgrade the tools.
Other changes:
- In cases where I was already updating a pipeline script task (e.g., to add/change a parameter passed to a script), I converted the task syntax to use the newer 'script' alias for consistency with other pipelines.
- In our API Proxy pipelines, there were three different places where we installed qemu and binfmt to prepare for cross-compiling the code, but the agent already has those installed so I removed it.
- Also in our API Proxy pipelines, there was a place where we did some docker buildx setup even though the job doesn't use docker buildx. Maybe it used to? Anyway, I removed the setup task.
- Removed the explicit bin_dir argument from all image-linux.yaml calls, since it is redundant with that parameter's default value.
- Deleted all manifest.yaml.template files, since they were required for the old manifest tool and are no longer used.
- In some of our release pipelines we have a default value for the 'tags' parameter that is given in not-quite-JSON format (an array with a single string value in _single_ quotes). I changed it to valid JSON so I can use jq to merge it with other tags when building the multi-arch image.
- Removed the unused '--postfix' parameter from buildRocksDb.sh.
- Removed the redundant and misnamed '--image-name' parameter from buildApiProxy.sh.
- Cleaned up script variables and args parsing for consistency, and in a few cases to fix minor bugs.
To test, I ran the CI build, which exercises the key YAML templates and all the scripts. I also ran the end-to-end tests and nested end-to-end tests to verify that the images still work as expected.
## Azure IoT Edge PR checklist:
This change removes `edge-hub/watchdog` and most of `mqtt/` from the codebase and removes the experimental feature flag in Edge Hub. It also removes the test module `generic-mqtt-tester`, which is no longer needed. Finally, it removes several unused build/test pipelines, most of which were related to the broker.
To test, I ran the following pipelines:
- Build Images
- Edgelet Packages (pretty sure this one was unnecessary)
- Single-node End-to-end
- Single-node Connectivity
- Nested End-to-end
## Azure IoT Edge PR checklist:
This PR removes:
- the manifest trust sample
- manifest trust related docs
- code coverage settings and project reference related to manifest trust
- [X] I have read the [contribution guidelines](https://github.com/azure/iotedge#contributing).
Currently, the edge agent encodes some runtime information (e.g. EdgeAgent and .NET versions) as part of the product information connection parameter, which can then be used to examine device characteristics from management interfaces. Some users desire the ability to add system-level properties--such as kernel version--to the product information. This PR adds a configuration option for custom product information strings as well as an automated option which augments the existing runtime information with the following system parameters:
- kernel name, release, architecture
- OS name, release
- system name, version, vendor
Generated files:
- edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2021_12_07/generatedCode/EdgeletHttpClient.cs
Largely duplicated files:
- edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2021_12_07/ModuleManagementHttpClient.cs
- edgelet/api/managementVersion_2021_12_07.yaml
### General Guidelines and Best Practices
- [x] I have read the [contribution guidelines](https://github.com/azure/iotedge#contributing).
- [x] Title of the pull request is clear and informative.
- [x] Description of the pull request includes a concise summary of the enhancement or bug fix.
### Testing Guidelines
- [x] Pull request includes test coverage for the included changes.
- Description of the pull request includes
- [ ] concise summary of tests added/modified
- [x] local testing done.
This change updates references to this repo's default branch ("master" to "main") in YAML pipelines, markdown documents, scripts, and code comments in this repo. It also updates references to other repos where the default branch has been renamed.
The devguide specified a set of commands to build EdgeHub locally. These stopped working when we made the change to build the rocksdb `.so` file every build.
To fix this I created two scripts:
1. buildRocksDb.sh: This is the current bash logic in the devops template which builds rocksdb and copies out the binary
2. buildLocalEdgeHub.sh: This uses buildRocksDB.sh, along with another set of scripts to build a local EdgeHub image.
Passing build (waiting for results):
https://msazure.visualstudio.com/One/_build/results?buildId=51582888&view=results
***Please replace this line with your PR description and read PR checklist below***
## Azure IoT Edge PR checklist:
E2E content trust setup contains the following changes for enablement
1. Set up of content trust registry containing a notary signed image and an unsigned image
2. Enable Edge Daemon with content trust feature by configuring the edged and certd toml files with its certificates
3. Addition of content trust azure container registry and its credentials are updated to both E2E checkin and E2E core test pipelines along with the registry Root CA
4. Notary installation in /usr/bin in the pipeline infrastructure
5. Deploy the signed and unsigned container images modules
6. Update in the document to use the right Notary version and also configuring in the super-config.toml
Add a script for setting up the Visual Studio debugger in an Edge container/module. Also, update the Dev Guide to document how to use the script and VSCode to attach a debugger to the edgeAgent container/module.
This script and documented steps have been tested on an Ubuntu18.04 dev machine running a local build of aziot-edge v1.2. In this test, VSCode was running on the dev machine and was able to successfully attach its debugger to the Edge Agent container.
## Azure IoT Edge PR checklist:
@veyalla After noticing that the CACertificates readme had some outdated links, I did a scrub of links to docs.microsoft.com in the other markdown files to make sure they were still pointing to the correct info/not redirecting/not 404'ing.
I also replaced the X.509 DPS content with a 'deprecated' note and a link to the doc that was published at GA. I can delete the file, but didn't know if anything linked to it still.
I don't know why the quotation marks and apostrophes changed in reported_states_during_deployment.md -- it could have been done by the docs extension since those Word-style punctuation marks often render odd on the docs site. I can undo that if you have any concerns.
Port 50027ff39e to master
It is better to use Unpooled buffers in dotnetty by default, as it is more stable, and should provide comparable performance. Adding a new flag "mqttSettings__UsePooledBuffers" to control buffer pooling a OptimizeForPerformance affects other things and we want to be able to control pooled/unpooled buffers separately.
The RocksDB MANIFEST file stores the history of operations. When it reaches
the configured max size, RocksDB rolls over to a new file and marks
the previous file as obsolete, so that it will be deleted in
the next compaction.
Since the default max size in RocksDB is 1 GiB which might be too high for
some users, this env var allows those users to set it lower.
Created a new document that shows how edgeAgent's twin properties change when a new deployment is applied to an IoT Edge device. This gives a user an idea how to detect the progress of the deployment on the edge device. This document is primarily aimed towards at-scale deployments.
Add metrics to edgeAgent and edgeHub for manifest trust feature:
edgeAgent_manifest_integrity_flag/edgehub_manifest_integrity_flag
`signing_with_ca_enabled` (true/false)<br>`signing_with_integrity_enabled` (true/false)
The value is 1 if manifest integrity is present or 0 if not present, tags indicate which integrity components are present.
Gauge
edgeAgent_twin_signature_check_count/edgehub_twin_signature_check_count
`result` (Success/Failure)<br>`algorithm`
The number of twin signature checks, `result` is "Success" or "Failure".
Counter
edgeAgent_twin_signaturs_check_seconds/edgehub_twin_signaturs_check_seconds
The amount of time it took to verify twin signature.
Summary
In this PR I'm trying to consolidate Broker and Bridge settings together in `mqtt-edgehub` crate. Even if the Bridge settings structs are defined in `mqtt-bridge`, we need to manage "reading" of the settings in `mqtt-edgehub`, so we can control how we read (file vs env vars) and what we read (which edgehub-specific env vars can override settings).
Changes in this PR:
- Include Bridge settings into Edgehub settings.
- Move and fix tests.
- Restrict bridge settings to the following variables:
- `UsePersistentStorage` (existing var) - defines whether to use disk or in memory storage for the bridge.
- `MqttBridge__StorageMaxFileSize` - defines maximum size (in bytes) of the MQTT Bridge storage. This storage is separate from edgehub messages / twins storage.
- `MqttBridge__StorageMaxMessages` - defines max in-memory queued messages in the MQTT Bridge.
- `MqttBridge__StorageFlushOptions` - for now only two: "AfterEachWrite" and "Off". **There is a bug in reading "AfterXWrites(u64)" variants from env var, I'd like to fix that in a separate PR**
- `StorageFolder` - path where to write storage file(s)
- Make sure `$upstream` bridge settings can still be read from env vars
Replace some remaining references to Ubuntu 16.04 with 18.04.
Note our release pipeline for docker images (`images-release.yaml`) calls `install.sh` with the `--package-arm` argument, which hasn't been updated for 18.04. That's a bigger effort that would need to be resolved before this can be merged (or handled in a separate PR).
ApiProxy -> EH traffic currently goes over port 443, specifically traffic dealing with device authentication. This introduces a security risk in the scenarios where the proxy is not present, and a rogue offbox connection masquerades as the proxy trying to authenticate as a downstream device.
To close this gap, this change makes EdgeHub listen on port 8080, but it's not mappable to the host. Thus the port can only be reached by connections originating from the internal Docker vNet (i.e. the ApiProxy module). We then direct all communications between the proxy and Edgehub be through this internal port.
There are few issues with env vars that we have and below are the improvements I suggest.
- Not consistent naming - I suggest to land on PascalCase, since it's already most popular format for our existing vars. Please note this is pure documentation update. Our configuration implementation will be case insensitive.
- We use `:` in many settings where it is not well-supported on all platforms. Specifically it is hard to use on Linux - I suggest to move to `__` as a separator. This is a supported separator for dotnet core config provider and is back-compatible with `:`. So this, again, can be just documentation change, since both `:` and `__` will continue working in dotnet. For rust (broker code) we will also support both `:` and `__`.
- Some variables in broker needs to be changed from snake_case to PascalCase, while we have time before release. Those vars are:
- `MqttBroker__MaxQueuedMessages`
- `MqttBroker__MaxQueuedBytes`
- `MqttBroker__MaxInflightMessages`
- `MqttBroker__WhenFull`
Some other changes:
- Added support for `StorageFolder` env for Broker, so EH and Broker will store their state to the same folder.
- Updated `EnvironmentVariables.md`
Adding build logic for the mqtt test module. I added some base images for arm, which I don't think we will need to update at all so I pushed some base images with the tag below. If we need to start updating these we should conform with the branch-consistent versioning scheme for our dotnet base images.
```
edgebuilds.azurecr.io/microsoft/azureiotedge-module-base-rust:1.0.0-[ARCH]
```
While I was touching the build logic I also refactored the edgehub build logic. The problem before was that the logic that moved the artifacts around was in a bunch of yamls, which was not conducive to local builds. This new way will allow us to run a series of scripts to build edgehub locally. Something like this:
```
scripts/linux/buildBranch.sh
scripts/linux/cross-platform-rust-build.sh --os alpine --arch amd64 --build-path mqtt/mqttd
scripts/linux/cross-platform-rust-build.sh --os alpine --arch amd64 --build-path edgehub/watchdog
scripts/linux/consolidate-build-artifacts.sh --artifact-name "edge-hub"
scripts/linux/buildImage.sh -r "$(registry.address)" -u "$(registry.user)" -p "$(registry.password)" -i "${{ parameters.imageName }}" -n "${{ parameters.namespace }}" -P "${{ parameters.project }}" -v "${{ parameters.version }}"
```
Config.yaml previously had a mapping of registry hostname and root CA path. With new runtime, it would be a mapping of registry hostname and the cert ID of its corresponding root CA.
In certd.toml, the mapping of Cert of the root CA and its file path must be configured.
Vadim Kovalyov
Damon Barry
Gordon Wang
Pat Altimore
Varun Puranik
Anca Antochi
Micah Lewis
Andrew Smith
vipeller
Gaurav Kulkarni
onalante-msft
gaya
Noel Campbell
Nishant
Kelly Gremban
Arnav Singh
Lee Fitchett
Sanjid
David Robson
Isabel B
hugues bouvier
Vadim Kovalyov
richma-ms