**Major:**
1. Backoff in case of d2c messaging throttling, stopping telemetry and twin operations
2. Improve perf reducing load on garbage collector and disposing IoT SDK resources
3. Improve device twin management, e.g. try to avoid twin writes when data is not changed, handle more error scenarios
4. Add "development" feature flag to enable/disable expensive runtime checks
5. Change partitions size from 1000 to 5000 devices to reduce the load on storage
**Minor:**
1. Backoff in case of device count throttling (e.g. when reaching 8000 devices in the free SKU)
2. Remove unused daily counter for telemetry
3. Improve perf reducing the number of no-op async tasks
4. Update IoT SDK and other dependencies
5. Clean up logging code, reimplement log filtering (removed in past PRs)
6. Print SDK version at startup
7. Add some scripts for development, see /scripts/development (create/delete simulation, start storage adapter)
**Bug fixes:**
1. Handle and recover from exceptions in the partitioning agent
2. Fix some swallowed errors/exceptions and unnecessary try/catch
3. Share script interpreter between methods and state to ensure device state consistency
4. Fix logged throughput in case of no traffic, i.e. show 0.0 msg/sec, and round value to 3 decimals
5. Fix logging from ConfigData
6. Change the dev endpoint used to delete simulations: don't delete devices (the endpoint was not working) - no user impact, dev only
* Refactor code writing twin tags for better performance
* Refactor device properties code for better performance
* Refactor simulation runner for better performance
* Improve connections performance
* Reduce time to start simulation, by creating devices using the same conn string and fetching device from registry only if the authentication fails.
* Limit the number of pending threads attempting to deliver telemetry
* Change all stock device models to use AMQP
* Move SDK timeout setting and other hard coded magic numbers to config file
* Use default timeout value from SDK instead of hardcoding it
* Improve documentation in the configuration file
* Upgrade IoT SDK
* Improve logging in device client
* Reduce noise in the "Info" log level
* When building, remove "deleted files" left over from previous builds
* Improve error detection when creating devices with batch operations
* Recreate device client when the internal AMQP client is disposed (lib bug workaround)
* VS code config update
* Remove simulation version, not used anywhere
* Catch and log when a device model doesn't exist
* Log missing config settings
* Make sure the simulation object is not lost in case of errors while checking for simulation changes
* Limit async tests duration
* Remove IoT hub registry workaround
* Fix boolean conf values parsing
* Remove "en-us" from links in the comments
* Fix invalid parsing of environment variables
* Before checking all the env vars, detect whether secrets, usually encrypted, are available or not. Secrets are not available when building a pull request, so the script will not check for those.
* Output env info during Docker image build
* comment out latest changes to Dockerfile
* Revert changes and remove symlink to /app/data
* undo changes to build script
* undo changes to build script
* trigger build
* Rewrite to use fewer threads and increase throughput
* Expose simulation metrics in the status endpoint
* When available, return a URL to the Azure Portal metrics of the pre-provisioned IoT Hub
* Allow to run a simulation against a custom IoT Hub, passing the connection string
* Allow to schedule a simulation, defining start and end time
* Improve security by running the service as a non-root user
* Support multiple behavior scripts in a device model
* Support custom device with custom sensors, behavior and frequency
* Allow to override the initial device state when creating a simulation
* When a simulation starts, create all the devices in batches
* When a simulation is deleted, delete also the devices
* Refactor timers to start sending telemetry as soon as possible
* Refactor and improve how simulation scripts access state and properties
* Change stock models to use AMQP (#189)
* Overall improvements to exceptions handling
* Disable SDK retry, and change timeout from default (4 minutes) to 10 seconds
* Do not retry sending telemetry on failure, skip message
* Use IoT Hub S2 SKU limits by default
* Upgrade to .NET Core 2.0.3
* Upgrade Azure IoT SDK: Devices 1.4.1 to 1.6.0, Devices Client 1.5.2 to 1.7.0
* Run simulation engine in the same process used for the web service
* Move docs, move wiki, new API specs, fix scripts for Windows Bash
* Fix the spelling of “ETag” and allow ETag=*
* Add internal scripts for increasing/decreasing/random telemetry
* Add env vars documentation ENVIRONMENT_VARIABLES.md
* Add more optional logging by device and by actor
* Use logging level and other logging settings from configuration
* Adjust unit test precision to mitigate flaky test
* Add system properties to telemetry messages
* Removing the squash flag from Docker scripts
* Use env vars in launchSettings.json
* New configuration settings for rate limiting
* Rate connections and messaging frequency
* Don’t read the twin on creation
* reate GitHub issues tracking the TODO notes
* Change the timer to run only once and support scenarios where it is being disposed
* Pass the cancellation through where it can be used by IoT SDK
* Fix web service returning the wrong ETag when editing a simulation
* Remove all the timeouts incompatible with throttling (i.e. throttling will slow down operations and a timeout wouldn’t make sense)
* Improve the Actor stop logic to be resilient to errors and make all the required steps regardless of exceptions
* Use async/await where possible to avoid the use of .Wait() and .Result
* Log throughput in debug mode
* Complete seed data, complete all the device models as per specs
* Rename “device type” to “device model”
* Clean JS syntax, check with JS Lint
* Generate device IDs from device model ID, instead of device model Name
* Complete health check endpoint
* Store simulations using the storage adapter
* Implement device bootstrap scenario: when a new device is created, mark it as a simulated device and report some properties like device type, messages schema, and initial location.
* Remove dependency on IoT Hub manager and access IoT Hub directly
* Refactor state machine to reduce complexity and reuse code
* Add launch settings for Visual Studio
* Remove env var used for the web service TCP port
* Improve logging of exceptions to avoid log flooding
* Fix messages format, to always use the “_unit” convention
* Add JSON config checks
* When a new device is created, mark it as a simulated device and
* When a device connects report some properties like device type, messages schema, and initial location. Use MQTT for twin operations
* Remove dependency on IoT Hub manager and access IoT Hub directly
* Refactor state machine to reduce complexity and reuse code
* Add launch settings for Visual Studio
* Remove env var used for the web service TCP port
* Improve logging of exceptions to avoid log flooding
* Fix messages format, to always use the “_unit” convention
* Bump version to 0.1.4
* Allow to inject custom device models
* Improve parsing of Jint response to support dictionaries and objects
* Sync scripts with template
* Change Truck geolocation format
* Update docker scripts
Use external Javascript functions to generate the telemetry for simulated devices. The path of folder that contain the Javascript files can be configured, so it's possible to mount JS files from anywhere. Files are loaded at runtime. The name of the files is specified in the Device Type JSON files, which are managed in the same way, with a configurable path and loaded at runtime.
Javascript files are interpreted using Jint https://github.com/sebastienros/jint .
Documentation here: https://github.com/Azure/device-simulation-dotnet/wiki/Device-Types
Remove Akka and use .NET Thread Pool to manage multiple threads sending telemetry. The lack of Akka means some extra work that we'll need in order to distribute the simulation over multiple servers. For the time being, we assume that the simulation runs in one instance only. If Akka's move to .NET Core is ready we can move back to Akka, otherwise we need some partitioning logic to distribute the load over multiple machines.
Upgrade the project to the latest template, e.g. scripts and documentations. Minor fixes for cross-platform development (VS2017 and Rider).
Complete the /status endpoint to check if IoT Hub Manager is available. In order to run the simulation, IoT Hub Manager is required. Without this check, it was too difficult to debug whether the service could connect to IoT Hub Manager or not, particularly when using Docker Compose.
Added Docker Compose file, to bootstrap this service and IoT Hub Manager containers. Add script to publish to Docker Hub. Move Docker image version/tag to a version file, so that the version is not hard coded in multiple places.
Fix an issue with the logger using too much memory when serializing exceptions.
Improve handling of exceptions caused by unreachable external dependencies when using the HTTP client.
Upgrade source code, project files and scripts to use .NET Core. Mono is not required anymore, and also the Docker image will now use only .NET Core.
The configuration format has changed to be in line with ASP.NET Core configuration providers.
The simulation runner is temporarily disabled, because Akka.NET is not ready for .NET Core, and I'm in the middle of removing Akka from the project as well.
Due to the move from .NET Core, some internal APIs are different, for instance you might notice changes in the web service exception filter, in the Autofac dependency injection setup, in the use of Kestrel as a HTTP server.
Since the microservice is now built and packaged with the `dotnet` command, some paths have changed, and scripts have been updated accordingly. I tested the scripts in Windows and MacOS, hopefully catching all the edge cases. The scripts are now actually simpler, which is a nice and unexpected result of the migration.
* Remove Akka HOCON from web service, use app.confg for settings - use HOCON only when using Akka
* Improve config parsing
* Add Moq to all test projects
* Add env var for IoTHubMan web service
* Update env vars management and correlated scripts
* implement simulation agent using Akka actors
* update Docker scripts to package the web service and the simulation agent
* some DI refactoring to align web service and simulation agent
* detect and manage content negotiation errors
* Change ERRORLEVEL code in Windows scripts
* Change windows scripts error redirection
* Fix compilation warnings and convert some *nix scripts to ISO
* Document the exceptions and apply filters on controllers (rather than globally)
This adds a new endpoint to list all the supported device types. Device types are configured via JSON files external to the assembly, so that users can add custom types without rebuilding the service. The files include a README describing the format of these files.
I was going to use HOCON for these files too, for a cleaner look, but I'm actually reconsidering that approach and the use of Newtonsoft greatly reduces the code required to parse these files (at least in .NET).
Peter Felts 🍺
Avani Patel
Devis Lucato
Peter Felts
Jill Bender
Zhang Hua
Parvez
Devis Lucato
Tim Laverty