Removing docker mode from EdgeAgent. Synced with @marianan to confirm docker mode not used in our tooling. It certainly isn't used by the product.
I have removed docker command factory and related test. Reasoning:
> we have this docker command factory here:
https://github.com/Azure/iotedge/blob/main/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerCommandFactory.cs
> This is not used for anything except for this E2E test:
https://github.com/Azure/iotedge/blob/main/edge-agent/test/Microsoft.Azure.Devices.Edge.Agent.Docker.E2E.Test/AgentTests.cs
> The E2E test doesn't make much sense to keep around for me, since the benefit of it on top of existing integration tests is to test whether the plan being executed manifests in docker containers state being altered. However it is not using the command factory we actually use, but a different one. In my opinion this makes it pointless.
> I'd like to remove both these files so I can fully remove DockerCommandFactory from EdgeAgent. This will also make it easier for the work we are doing now because we won't have to worry about altering DockerCommandFacotry.
## Azure IoT Edge PR checklist:
This change removes `edge-hub/watchdog` and most of `mqtt/` from the codebase and removes the experimental feature flag in Edge Hub. It also removes the test module `generic-mqtt-tester`, which is no longer needed. Finally, it removes several unused build/test pipelines, most of which were related to the broker.
To test, I ran the following pipelines:
- Build Images
- Edgelet Packages (pretty sure this one was unnecessary)
- Single-node End-to-end
- Single-node Connectivity
- Nested End-to-end
## Azure IoT Edge PR checklist:
This PR removes:
- the manifest trust sample
- manifest trust related docs
- code coverage settings and project reference related to manifest trust
- [X] I have read the [contribution guidelines](https://github.com/azure/iotedge#contributing).
Move test metrics collector into test directory for clarity and consistency w/ 1.1 branch.
These changes were tested by running a build images pipeline followed by E2E and Connectivity Test pipelines.
## Azure IoT Edge PR checklist:
The K8s client is currently requiring a library (BouncyCastle) we are
attempting to remove from the project. Since the existing Edge on K8s
feature is tied to 1.1, I am removing it from main and 1.2 branches.
## Azure IoT Edge PR checklist:
In nested environment, the diagnostic image extracts the trust bundle before establishing the communication with upstream.
Improved error message with a suggestion for our user.
Before it was just trusting the upstream CA.
Removed unused publishing of connectivity test.
Test:
Put a dummy CA inside the trustbundle. Confirmed it fails
Put good CA, confirmed it passes
Checked in regular, non nested scenario for regression.
Manifest Signing Feature flow in Edge:
1. The deployment manifest is signed using Manifest signer client (exists in the sample folder of the solution). It adds integrity section for each module in this format below. Signature is computed based on the algorithm used and the hash of the content of the each module's desired properties.
```json
"$edgeAgent": {
"properties.desired": {
"modules": {},
"runtime": {},
"schemaVersion": "1.0",
"systemModules": {},
"integrity": {
"header": {
"signercert": ["signercert1stpart", "signercert2ndpart"],
"intermediatecacert": ["intermediatecacert1stpart", "intermediatecacert2ndpart"]
},
"signature": {
"bytes": "signaturebytes",
"algorithm": "ES256"
}
}
}
},
"$edgeHub": {
"properties.desired": {
"routes": {},
"schemaVersion": "1.0",
"storeAndForwardConfiguration": {},
"integrity": {
"header": {
"signercert": ["signercert1stpart", "signercert2ndpart"],
"intermediatecacert": ["intermediatecacert1stpart", "intermediatecacert2ndpart"]
},
"signature": {
"bytes": "signaturebytes",
"algorithm": "ES256"
}
}
}
},
```
2. A signer cert is used to signed the deployment manifest is also added to the integrity section
3. When a signed deployment is deployed, the twins received by the modules are checked for valid signature before consuming the twin.
What is this PR?
- Twin Signature validation checks on receiving full twin and patches for EA and EH
- Schema change for EA deployment config
- Schema change for Edge hub config
- Added Test cases
- Updated Manifest Signer client by removing manifest version
- Manifest Signer client added as a sample( removed from tools) and now shares the JSON canonicalization from Edge Util
- Modified code coverage settings to exclude the open source code of JSON canonicalization
What is not in this PR?
- iotedged workload API for manifest trust bundle and modules using the API for chaining verification
* GetModuleLogs returns an error message on too large payload instead of using tail 500
* remove flag
* ping test
* name
* timeout
* modulename
* fix expected
* get ea logs
* Make number logger
* uncomment setup
* test getmodule logs
* uploadtest
* fix file header
* check restart dm
* update dockerfile
* fix parsing response
* parse logs response
* log returns array
* correct expected
* add \n
* add delay
* more delay
* fix get logs expected
* test longer timeout
* increase timeout to test pi
* add other tests back
* Undo spacing changes
* codecov settings
* undo accidental merge changes
* fix status check for upload tests
* update image to close when requested
* reduce upload amount
* fix number logger delay code
MQTT Broker is running as an external process to EdgeHub, but relies on it to authenticate connecting clients. An HTTP endpoint is added to EdgeHub, which can receive different type of credentials (SAS token, certificates) and responses with yes/no, reflecting the authentication result.
The endpoint is designed for internal (in-container) use.
The main file is AuthAgentListener.cs, which handles the HTTP calls and dispatches the queries to the already implemented authentication machinery.
Overall changes:
- Introduce EdgeHubRestartTester module
- Introduce edgehub_restart_deployment.template.json
- Introduce EdgeHubRestartTester to Image Build pipeline
- Introduce EdgeHubRestartTester to Container Registry publish
- Make deployment file for connectivity test configurable from VSTS pipeline
- Muffled soon-to-be a new report type for EdgeHubRestartTester
The EdgeHub Restart Test does the following:
1. The EdgeHubRestartTester sends an edgeHub-restart command to edgeAgent to restart the edgeHub module.
2. The EdgeHubRestartTester spins up parallel tasks (configurable) to
2.1 Send messages until a message is sent successfully
2.2 Send directMethods until a directMethod is sent successfully
3. Record a time period between restart until either message or direct method succeed
4. Generate a test result for each of the sent methods { message, or directMethod }
5. For each corresponding result, trigger an appropriate TestResultCoordinator endpoint.
The change enables code coverage analysis for our .NET Core code in the checkin build, and (similar to our iotedged checkin build) fails the build if % line coverage is below a certain threshold. Current coverage is measuring at 61% so I set the threshold to 60%.
Note this change relies on the built-in code coverage functionality that has recently been added in .NET Core. Downsides are: it's currently only available on Windows, and it produces a .coverage file which has to be opened in Visual Studio. Upsides are: this is the "officially supported" option for Azure DevOps, so while support is minimal now, future improvements to analytics in DevOps would likely come for free. Also, because this option works out-of-the-box in the dotnet tooling, I was able to remove a **bunch** of gunk from our project files.
The other option would be to use the 3rd-party [Coverlet](https://github.com/tonerdo/coverlet) tool, which would allow us to run on Linux (might shave a couple minutes off job time) and outputs in Cobertura format just like our Rust code coverage. More details [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops#collect-code-coverage-metrics-with-coverlet). So we'd have a more consistent experience between languages. But we'd have to add back a bunch of project references in our test projects for arguably minimal benefit since this change seems to give us everything we need, at least for now.
This change also adds the "Unit" category to 132 unit tests which were not categorized.
* Mapping the pod status to the module status for Pending/Failed/Unknown
* Created Kubernetes Environment & its Provider to handle the module status reports to the IOT hub
* Updated the unit tests for handling the change in pod status to module status
* Removed blank line
* Pod status checks for container status before converting it to module status
* Unit test to reflect the new model of determining module status
* Update edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Kubernetes/KubernetesEnvironment.cs
Co-Authored-By: Denis Molokanov <dmolokanov@users.noreply.github.com>
* Updated unit tests
* fixing the braces
* fixed braces 2
* Fix build errors
* Debug mode changes to test code flow
* Updated status in debug mode
* changed moduleruntime status instead of dockerruntimestatus which gets reported to IOT
* Removed debug logs
* Resolve conflicts
* fix typo
* Fix typo in unit test
* More typo fix
* Simplified unit test
* Updated DockerModule priority parameter changes for k8s environment
* Fix typo in unit test names
* Updated the test cases with single responsbility approach
* start time format changes
* Refractoring
Co-authored-by: Denis Molokanov <dmolokanov@users.noreply.github.com>
With this change, a new property ‘Priority’ will be added to the deployment configuration of each module. This value will be used by Edge Agent to determine the order in which all other modules are to be created.
The value of ‘Priority’ will be positive and zero-based with 0 being the highest priority value. A higher numeric ‘Priority’ value would indicate a lower priority assigned to that module. The maximum value of this property will be UInt32.MaxValue (4294967295).
Modules that possess the same priority will be created at the same time and will have no deterministic startup order imposed amongst themselves. Modules with higher priority values will be created and started first and once Edge Agent has made an attempt to create and start them (only the modules with a desired state of ‘Running’ will be started), it will proceed with the creation (and optionally startup) of other lower priority modules in order.
Edge Agent will not support the ‘Priority’ property and will by default have the highest priority of zero (0).
Modules that do not have a ‘Priority’ value specified will be started in a non-deterministic order (as is the case today) and will be assigned a priority of UInt32.MaxValue which indicates the lowest possible priority. Edge Agent will trigger the creation of these modules after all modules with a higher priority have been created.
Module boot ordering will initially not be supported for the Kubernetes execution mode of IoT Edge.
* Clean up TestOperationResult; and AnalyzerClient and TRC Client.
* Rename ClientTestResultOperation to TestResultOperationDto
* Update descriptions in swagger files.
- Remove the old Direct Method Cloud Sender module
- Modify Long Haul pipeline to handle a new deployment for the new Direct Method Sender module
- Set the default value of the Direct Method Sender to be compatible with the old E2E pipeline
- Fix a broken reporting endpoint for Long Haul pipeline
* add metrics collector module to build pipeline
* add new metrics publishers for log analytics and eventhub
* fix edgehub crash when using experimental features by switching instance number file to guid
- Add integration tests for k8s environment operator against kind cluster
- Install kind in ci pipeline
- Run integration tests against k8s cluster in ci pipeline on Linux
- Disable k8s integrations tests in ci pipeline on windows
* Add secret type for ImagePullSecret in k8s
* Get rid of fullname in the resourse definitions
* Add Kubernetes Mode to edgeagent env vars.
* Integrate Kubernetes into Edge Agent (#1556)
* Allow Kubernetes mode to be activated
* Manage service accounts from edge agent
* Add back module start
* Make iotedged have cluster role to allow token review.
* Add trust bundle for each module (#1607)
* [k8s] Add missed env variables for EA (#1623)
* Added missing env vars to edgeagent deployment
* Remove service account env for edgeagent
* Fix formating errors after stub change
* Namespace those custom objects.
* Set management and workload api urls as default for k8s
Update edgeagent role
* Planner needs namespace, not gatewayhostname
* Add edgedeployments rbac for iotedged
* add edgedeployment rbac to iotedged, too.
* Add strict serialization rules (#1644)
* Use KUBECONFIG to get kubernetes config first
* [k8s] Refactor edge deployment serde (#1686)
* Do not deleted service accounts if deloyment is updated, fix hubname in
iotedged.
* Update edgelet/edgelet-kube/src/convert/mod.rs
Co-Authored-By: Denis Molokanov <dmolokanov@users.noreply.github.com>
* Update edgelet/edgelet-kube/src/convert/mod.rs
Co-Authored-By: Denis Molokanov <dmolokanov@users.noreply.github.com>
* Missed a few places to use dns_domain.
* Config map names given to EdgeAgent (#1703)
* Service account names match original module id in annotations
* [k8s] Refactor CrdWatcher (#1708)
- Renamed CrdWatchOperator to EdgeDeploymentOperator
- Renamed CrdWatcher to EdgeDeploymentController
- EdgeDeploymentOperator only
1. watch CRD and delegates modules deployment on k8s to EdgeDeploymentController
2. stores current modules
- EdgeDeploymentController make all checks against k8s api and deploy modules with services and service accounts
* add check for config map names
* [k8s] Refactor runtime info provider (#1714)
* Add resource name preconditions
* Rename CRD -> EdgeDeployment
* Start operators in Program.cs
* Get k8s systeminfo from edgelet (#1717)
* [k8s] Refactor RBAC for iotedged (#1726)
- Refactor iotedged ClusterRole to several small scoped roles
- Added doc describing RBAC we are using
* [k8s] Refactor docker to k8s primitives conversion (#1733)
* [k8s] Combined docker config comparison. (#1734)
* [k8s] Lifecycle manager for kubernetes needs to return all active modules. (#1748)
* Make the internal workings a virtual function,
so I can override in k8s mode.
* Move "proxy" constants to k8s constants.
* [k8s] Spec: Kubernetes experimental extensions for createOptions (#1753)
* Add spec to k8s create options extensions (volumes, nodeSelector, resources)
* KubeUtils and test
* System Info test complete
* Successful happy path (add 3 modules)
* Added a modified and a deleted pod test.
* Some fixes and formatting improvements.
* KubeUtils produces errors if names are too long.
* add null and empty checks
* Remove KubeUtils.K8sNamespace, pass in namespace instead.
* KubeUtils doesn't need a constructor now.
* Convert array check to use hashset.
* Remove length parameter from SanitizeDNSValue as it's no longer needed.
The change adds the module-to-module direct method test to our growing suite of end-to-end tests that will replace IotEdgeQuickstart. The test is parameterized on the module-to-edge protocol, so there are four tests in all (AMQP, MQTT, AMQP-WS & MQTT-WS).
Now that there's more than one test in the suite, I've updated the infrastructure so that we run setup (i.e., create a new device identity and install IoT Edge daemon on the test device) and teardown once for the _suite_, instead of per test. To do that I replaced the console app with a regular test framework. Our existing unit/integration tests use Xunit, but I chose NUnit for the end-to-end tests because it can output log messages to the console _during_ the test run, instead of just at the end. Since the tests run for several minutes each and we're impatient, it's nice to be able to look at the console (or build dashboard) and see that things are moving along.
I've refactored a few things in the common library as needed for this change, but there will certainly be more as we add more tests. I still think each test should be shorter and easier to read, but we need to add a wider range of tests (like e.g. the transparent gateway test) before I know how best to refactor them.
* create a new ModuleLib project and fix some warnings in Leaf device
* Add Load Gen and Message Analyzer in images build
* Update NuGet for loadgen
* add a space in function sample name
* remove module user from direct method receiver docker file
* Create a new module DirectMethodCloudSender to test Direct Method from Cloud to Module and it will be added to long hual test
* Refactor to reuse commond logic.
* create a new ModuleUtil project and fix some warnings in Leaf device
* make sure all module use ShutdownHandler.
* Add Load Gen and Message Analyzer in Windows images build
* Update NuGet for loadgen
* add a space in function sample name
* remove module user from direct method receiver docker file
* Adding the Leaf Device Client for E2E Tests.
* Add message Validation based on GUID, so we make sure the message we are getting is unique.
* Using Device Connection String, instead of iotHub Connection String.
* Fix small typo on LeafDevice Program.cs
* Clean up solution.
1. remove script folder from solution since scripts should not be referenced by solution.
2. remove certificate folder that doesn't exist from Microsoft.Azure.Devices.Edge.Util project.
This code implements a module that can be used to generate telemetry and twin update loads on Edge Hub. It supports the following configuration options that can be set via environment variables (all of them are optional):
- `messageFrequency` - Defines the frequency at which messages should be generated specified as a string that can be parsed into a `TimeSpan`.
- `messageSizeInBytes` - Defines how big the data (which are randomly generated) in each telemetry message should be in bytes.
- `jitterFactor` - Specifies the jitter in frequency, i.e., how much randomness to add with respect to the frequency of telemetry and twin updates. This must be a value between `0.0` and `1.0`.
- `transportType` - Defines what transport to use. Can specify any of the values that the `TransportType` enumeration can be deserialized into.
- `outputName` - The output to which events should be published.
- `twinUpdateFrequency` - Defines the frequency at which twin updates should be sent.
A typical message generated by this module would look like this:
```json
{
"sequenceNumber": 42,
"data": [random byte array of length messageSizeInBytes],
"hash": [SHA256 hash of 'data']
}
```
A typical twin reported property update would look like this:
```json
{
messagesSent: [count of messages sent so far]
}
```
Andrew Smith
Damon Barry
Gaurav Kulkarni
Noel Campbell
David Robson
hugues bouvier
gaya
Lee Fitchett
Varun Puranik
vipeller
yophilav
dylanbronson
Shantanu Singh
Philip Lin
Anca Antochi
Denis Molokanov
Philip Lin
Angelo Ribeiro
Anca Antochi
Philip Lin
Raj Vengalil