When a node starts it generates a unique ID to identify itself, and writes the ID into the Nodes collection. The ID expires unless the process keeps updating the record, as a result the nodes collection contains an eventually consistent list of active nodes, which can collaborate to run simulations. The list size for instance is used to calculate the throttling limits.
To avoid contentions, only one node takes care of cleaning up the nodes collection, and in order to decide which node, there is a master-node election process. If the master node crashes another node eventually becomes master. The master election process will be used also to decide which node creates devices and partitions.
Add agent for creating and partitioning devices
1. Add a new agent to the solution, which will be responsible for creating
the devices in IoT hub and for partitioning the list of devices, so
that multiple VMs can each simulate a different set of devices.
The agent is not started yet because the new storage library is not
ready.
2. Refactor concurrency settings and add clustering settings
3. Temporary add retry logic to a couple of flaky tests
* Refactor code writing twin tags for better performance
* Refactor device properties code for better performance
* Refactor simulation runner for better performance
* Improve connections performance
* Reduce time to start simulation, by creating devices using the same conn string and fetching device from registry only if the authentication fails.
* Limit the number of pending threads attempting to deliver telemetry
* Change all stock device models to use AMQP
* Move SDK timeout setting and other hard coded magic numbers to config file
* Use default timeout value from SDK instead of hardcoding it
* Improve documentation in the configuration file
* Upgrade IoT SDK
* Improve logging in device client
* Reduce noise in the "Info" log level
* When building, remove "deleted files" left over from previous builds
* Improve error detection when creating devices with batch operations
* Recreate device client when the internal AMQP client is disposed (lib bug workaround)
* VS code config update
* Remove simulation version, not used anywhere
* Catch and log when a device model doesn't exist
* Log missing config settings
* Make sure the simulation object is not lost in case of errors while checking for simulation changes
* Limit async tests duration
* Remove IoT hub registry workaround
* Fix boolean conf values parsing
* Remove "en-us" from links in the comments
* Fix invalid parsing of environment variables
* New endpoint to CRUD custom device models, stored via storage adapter
* Note: stock models are read-only
* Some changes to internal classes used for device models, e.g. use Enum instead of Strings
* Fix incorrect exception used in the Simulations service class: when the ETag doesn't match use ResourceOutOfDateException, not ConflictingResourceException (no API change, same HTTP status code)
* Simplify the device models, no methods and twins (#121)
* Remove unused models, scripts, methods, properties
* Upgrade to .NET Core 2.0
* Add some more logging
* Improve simulation runner startup to handle exceptions
* Update CI conf
* move docs, move wiki, new API specs, fix scripts for Windows Bash (#123)
* move docs, move wiki, new API specs, add copyright, fix for Windows Bash
* Disable twin read write by default and add config option to enable (#125)
* add config setting to turn of twin read/write
* add logic to only call twin reads/writes if enabled
* change twin read/write config value to an optional env var
* Allow to schedule a simulation, defining start and end time (#126)
* Support simulation duration and scheduling
* Fix the spelling of “ETag”
* Allow ETag=*
* adjust unit test precision to mitigate flaky test
* Upgrade Docker image to 2.0.3 (#132)
* Fix nullref exception
* Copy agent settings into web service
* Fix status endpoint
* Use S2 limits and increase async thread pool size
* Run a simulation against a hub provisioned by the user (#133)
* Add IoTHub with connstring to Api Model, pass "default" to use the configured hub
* Add private helpers to redact key from IotHubConnString
* Store non redacted entire connection string in a file
* Update API specs documentation
* Reduce the number of threads and improve overall performance (#137)
* Rewrite the simulation engine to use fewer threads
* Start sending telemetry as soon as possible
* Run simulation engine in the same process used for the web service
* Add workaround for registry manager becoming unresponsive
* Add more optional logging by device and by actor
* Use logging level and other logging settings from configuration
* New web service APIs (#138)
New web service APIs
* New API to support multiple scripts
* New API to support custom sensors
* New API to support custom telemetry frequency
* New API to expose whether a hub is configured, i.e. provisioned and the connection string is in the config file
* Fix exception occurring when there is no connection string
* Simulation input validation (#140)
* Use custom frequency for telemetry and device state (#141)
* Apply the customized telemetry and state frequencies + fix bug for models with multiple messages
* Fix exception on bad request
* Improved Azure IotHub Connection String Validation (#143)
* additional validation for iothub connection string
* check permissions with calls to registry manager instead of string comparison
* Refactor permissions validation into separate methods. Make sure test device created is deleted, create with a regognizable test name in case of error.
* remove await from Assert.ThrowsAsync
* added retry logic to delete test device
* Logging cleanup
* Internal scripts for increasing/decreasing/random (#146)
* Update travis CI config to use .NET core 2.0.3
* Fix log level parsing from the config, to be case insensitive
* Update simulation model parsing/serialization. Add script Params in the service layer
* Complete the device model override logic
* Add internal scripts
* Allow to override the initial state. Add “online” key where missing.
* Fix null reference exception in DeviceModelsGeneration (#147)
* Use env var to define log level (#150)
* Use env var to define log level
* Add env vars documentation ENVIRONMENT_VARIABLES.md
* bugfix: Etag mismatch now returns 409 Conflict instead of 400 Bad Request (#151)
* When available, return a URL to the metrics of the pre-provisioned hub (#153)
When available and when the simulation is running, return a URL to the metrics of the pre-provisioned hub. Also some changes to the /status endpoint
* Use PCS_LOG_LEVEL env var for log level, instead of LOG_LEVEL
* Create devices using batch operations (#154)
* When a simulation is deleted, delete also the devices
* When a simulation starts, create all the devices in batches
* Throw error on all-whitespace connection string & validate default hub (#155)
* Check for whitespace in custom connection string. Check that default IotHub exists and is valid.
* Update invalid connection string message
* fix log level for GetiotHubConnectionString()
* Map “Custom” device model id used by the UI to the internal ID
* Fix code style
* Fix Decrementing function
* Add system properties to the messages
* Remove old code used to detect simulation stop via cancellation token (#162)
* Remove old code used to detect simulation stop via cancellation token
* Rethrow timeout exceptions
* Fix for override stock device model with null value (#163)
* add null check for PUT payload properties
* add null check test
* update test arrangement
* uncomment UpdateTwinAsync
* add folder for DeviceTwinActor
* Fix exceptions when override stock device model (#164)
* Add null check for MessageSchema
* Add support classes for DeviceTwinActor to update twin -- uses empty methods
* remove Properties from DeviceStateActor class
* Add InternalDeviceState Class and ability to update twin via callback from javascript
* add placeholders for twin update logic for future pr
* remove eslint jsdoc updates and update javascriptinterpreter tests
* Add new thread for device properties updates
* return null for not implemented methods
* Add active device count for status API (#169)
* Register SimulationRunner as a singleton type
* Allow DeviceStatusActor report active status
* Add active device count to status API
* Add Unit Tests for Internal Device State
* Update existing tests with new InternalDeviceState class
* Update scripts with instructions on device property updates
* Fix errors with JS device scripts and readonly dictionary
* revert changes to DeviceTwin class
* Update log messages for readability with timestamp
* Rename logging methods for consistency
* Separate properties and state and remove unused lines
* update comments for twin update branch
* Revert UpdateTwinAsync signature in DeviceClient
* revert UpdateState and JSValueToDictionary to private
* update comment on restoreState in chiller js script
* Update InternalDevicePropertiesTest names
* set sending telemetry retry policy to No Retry (#175)
* Consolidated to SmartDictionary class
* remove DevicePropertiesActor, revert DeviceTwin, change terminology from sensors to state
* revert actors logger
* revert simulation runner
* revert simulation runner test
* revert deletion of UpdateReportedProperties.cs
* fix spacing in SimulationRunner
* fix spacing in device twin
* consolidate restore state javascript methods
* add properties to internal script method call
* variable/method naming, whitespace cleanup, add missing method in elevator state script
* Add Device Properties logic to internal simulation (#173)
* uncomment UpdateTwinAsync
* add folder for DeviceTwinActor
* Add support classes for DeviceTwinActor to update twin -- uses empty methods
* remove Properties from DeviceStateActor class
* Add InternalDeviceState Class and ability to update twin via callback from javascript
* add placeholders for twin update logic for future pr
* remove eslint jsdoc updates and update javascriptinterpreter tests
* Add new thread for device properties updates
* return null for not implemented methods
* Add Unit Tests for Internal Device State
* Update existing tests with new InternalDeviceState class
* Update scripts with instructions on device property updates
* Fix errors with JS device scripts and readonly dictionary
* revert changes to DeviceTwin class
* Update log messages for readability with timestamp
* Rename logging methods for consistency
* Separate properties and state and remove unused lines
* update comments for twin update branch
* Revert UpdateTwinAsync signature in DeviceClient
* revert UpdateState and JSValueToDictionary to private
* update comment on restoreState in chiller js script
* Update InternalDevicePropertiesTest names
* Consolidated to SmartDictionary class
* remove DevicePropertiesActor, revert DeviceTwin, change terminology from sensors to state
* revert actors logger
* revert simulation runner
* revert simulation runner test
* revert deletion of UpdateReportedProperties.cs
* fix spacing in SimulationRunner
* fix spacing in device twin
* consolidate restore state javascript methods
* add properties to internal script method call
* variable/method naming, whitespace cleanup, add missing method in elevator state script
* Revert "Add Device Properties logic to internal simulation (#173)"
This reverts commit 37a720d044.
* [Updated] Add DevicePropertiesActor and push reported properties updates to the IoT Hub (#181)
* Initial commit with placeholders for sending twin updates
* Add skeleton DevicePropertiesActor and integrate with SimulationRunner
* Add mock devicePropertiesActor for SimulationRunnerTest
* added Device Properties Actor Setup
* add device properties actor tests
* Add Device Properties logic to internal simulation (#173)
* uncomment UpdateTwinAsync
* add folder for DeviceTwinActor
* Add support classes for DeviceTwinActor to update twin -- uses empty methods
* remove Properties from DeviceStateActor class
* Add InternalDeviceState Class and ability to update twin via callback from javascript
* add placeholders for twin update logic for future pr
* remove eslint jsdoc updates and update javascriptinterpreter tests
* Add new thread for device properties updates
* return null for not implemented methods
* Add Unit Tests for Internal Device State
* Update existing tests with new InternalDeviceState class
* Update scripts with instructions on device property updates
* Fix errors with JS device scripts and readonly dictionary
* revert changes to DeviceTwin class
* Update log messages for readability with timestamp
* Rename logging methods for consistency
* Separate properties and state and remove unused lines
* update comments for twin update branch
* Revert UpdateTwinAsync signature in DeviceClient
* revert UpdateState and JSValueToDictionary to private
* update comment on restoreState in chiller js script
* Update InternalDevicePropertiesTest names
* Consolidated to SmartDictionary class
* remove DevicePropertiesActor, revert DeviceTwin, change terminology from sensors to state
* revert actors logger
* revert simulation runner
* revert simulation runner test
* revert deletion of UpdateReportedProperties.cs
* fix spacing in SimulationRunner
* fix spacing in device twin
* consolidate restore state javascript methods
* add properties to internal script method call
* variable/method naming, whitespace cleanup, add missing method in elevator state script
* Revert "Add Device Properties logic to internal simulation (#173)"
This reverts commit 37a720d044.
* Add State Machine for device properties updates
* Add Device Property Update Logic and Remove Unused Device Twin Model
* delete old commented out DeviceConnection/UpdateReportedProperties.cs
* Add tests for UpdateReportedProperties
* Add properties to device models & clean up UpdatePropertiesAsync
* remove unused usings
* remove unnecessary lock in JavascriptInterpreter
* Parvezp/add methods (#182)
* Initial commit with placeholders for sending twin updates
* Add skeleton DevicePropertiesActor and integrate with SimulationRunner
* Add mock devicePropertiesActor for SimulationRunnerTest
* added Device Properties Actor Setup
* add device properties actor tests
* Initial check-in for adding methods
* Updating the tabs
* Adding more log parameters
* Fixing build break
Sorting using
* Adding all the simulated devices,state and methods (#183)
* Initial commit with placeholders for sending twin updates
* Add skeleton DevicePropertiesActor and integrate with SimulationRunner
* Add mock devicePropertiesActor for SimulationRunnerTest
* added Device Properties Actor Setup
* add device properties actor tests
* Initial check-in for adding methods
* Updating the tabs
* Adding more log parameters
* Fixing build break
Sorting using
* * Adding all simulated devices
* Adding all of their state javascript files
* Adding all of the javascript methods for those simulated devices
* Adding unit tests for the new files
* Addressing comments
* Removing extra space that i missed
* Update FirmwareUpdate method to use correct parameters in the callback from js to c# (#186)
* Feature/telemetry metrics (#170)
* Add simulation status reporting messages throughput and errors counters
* Do not retry sending telemetry in case of errors
* Change Azure SDK timeout to 10 seconds (default is 4 minutes)
* Improve SDK exceptions handling
* Fix rare exception due to contention on the `state` variable
* Change stock models to use AMQP (#189)
* Upgrade Azure IoT SDK: Devices 1.4.1 to 1.6.0 (#187)
Upgrade Azure IoT SDK: Devices Client 1.5.2 to 1.7.0
Upgrade test dependencies
* Register for Desired Property State Changes (#184)
Adds functionality for receiving notifications of Desired Property state changes from the IoT Hub. When the hub detects a state change it will send a notification via the DeviceClient.SetDesiredPropertyUpdateCallbackAsync registration. This change allows the simulation service to subscribe to the change notifications and update the internal simulation state to report the desired property to the hub when executed.
Changes:
* add callback for desired properties updates to Device Client
* add DevicePropertiesRequestTest
* add property for SupportedMethods
* Update DeviceConnectionActorTest with correct constructor for DeviceConnectionActor
* Update DEVICE_MODELS.md
* Add waiting state for properties update
* Update SetAll in SmartDictionar to set a new ConurrentDictionary
* Refactor properties actor
* Rename DeviceProperties.cs & methods, make callback private, update tests
* remove unused property from DevicePropertiesActor
* UpdateReportedPropertiesTest changes to reflect desired functionality
* Update DeviceConnectionActor state machine to tag simulated devices that were created via a list creation in Devices.cs
* Add Device Methods and Properties Functionality (#193)
* Add new state machine to manage changes to devices' "reported properties"
* Add callback to manage changes to devices's "desired properties"
* Add callbacks to manage method calls
* Refactor and improve how simulation scripts access state and properties
* move device tagging to DevicePropertiesActor and update tests
* Fix DevicePropertiesActor reference to twin failure count
* remove whitespace
* Fix merge conflicts between azure-iot-pcs-simulation and master (#196)
* Fix#122 + backport fix for flaky test (#128)
* Upgrade to .NET Core 2.0 and upgrade dependencies (#130)
* Upgrade to .NET Core 2.0 and upgrade dependencies
* Update comments and docs
* Change Docker base image to use .NET Core 2.0.3 (#131)
* Change Docker base image to use .NET Core 2.0.3
* use jessie
* Use absolute instead of relative links to wiki (#161)
Fix for:
https://github.com/Azure/device-simulation-dotnet/issues/39
The relative links don't resolve to the wiki page ending up in a 404. This update changes the URL links to be the full relative link to the documentation page.
* Remove squash (#166)
* Removing the squash flag
* Fixing extra space
* Use env vars in launchSettings.json (#168)
* Docker container excessively privileged running root user (#178)
* Add default non-root user 'pcsuser'
* Run service as 'pcsuser'
PBI[2211778]
* remove whitespace
* revert SimulationAgent.csproj changes
* Adjust whitespace in SimulationAgent.csproj
* remove newline
* Move FailedDeviceTwinUpdatesCount logic to SimulationRunner
* rename Tag to SetDeviceTag
* rename twin to device
* Clean up comments
* Upgrade travis.yml dotnet core version
* revert .Net core upgrade for travis
* Update .travis.yml
* Fix build
* fixing build...
* tag devices with ContinueWith
* fix typos
* Rewrite to use fewer threads and increase throughput
* Expose simulation metrics in the status endpoint
* When available, return a URL to the Azure Portal metrics of the pre-provisioned IoT Hub
* Allow to run a simulation against a custom IoT Hub, passing the connection string
* Allow to schedule a simulation, defining start and end time
* Improve security by running the service as a non-root user
* Support multiple behavior scripts in a device model
* Support custom device with custom sensors, behavior and frequency
* Allow to override the initial device state when creating a simulation
* When a simulation starts, create all the devices in batches
* When a simulation is deleted, delete also the devices
* Refactor timers to start sending telemetry as soon as possible
* Refactor and improve how simulation scripts access state and properties
* Change stock models to use AMQP (#189)
* Overall improvements to exceptions handling
* Disable SDK retry, and change timeout from default (4 minutes) to 10 seconds
* Do not retry sending telemetry on failure, skip message
* Use IoT Hub S2 SKU limits by default
* Upgrade to .NET Core 2.0.3
* Upgrade Azure IoT SDK: Devices 1.4.1 to 1.6.0, Devices Client 1.5.2 to 1.7.0
* Run simulation engine in the same process used for the web service
* Move docs, move wiki, new API specs, fix scripts for Windows Bash
* Fix the spelling of “ETag” and allow ETag=*
* Add internal scripts for increasing/decreasing/random telemetry
* Add env vars documentation ENVIRONMENT_VARIABLES.md
* Add more optional logging by device and by actor
* Use logging level and other logging settings from configuration
* Adjust unit test precision to mitigate flaky test
* Add system properties to telemetry messages
* Removing the squash flag from Docker scripts
* Use env vars in launchSettings.json
* New configuration settings for rate limiting
* Rate connections and messaging frequency
* Don’t read the twin on creation
* reate GitHub issues tracking the TODO notes
* Change the timer to run only once and support scenarios where it is being disposed
* Pass the cancellation through where it can be used by IoT SDK
* Fix web service returning the wrong ETag when editing a simulation
* Remove all the timeouts incompatible with throttling (i.e. throttling will slow down operations and a timeout wouldn’t make sense)
* Improve the Actor stop logic to be resilient to errors and make all the required steps regardless of exceptions
* Use async/await where possible to avoid the use of .Wait() and .Result
* Log throughput in debug mode
This PR provides method support for chillers in the default simulation: reboot, firmware update, increase pressure, and decrease pressure. It also keeps reported properties in sync with desired properties (by polling for value changes).
* DeviceMethod implmentation class
* First round of methods, added: 1) infra for registering methods for devices, 2) walking avail. methods for each device. Main functionality missing is allowing specifation and then running the method script.
* remove todo from actor
* nits to code style
* Move methods ownership from devicebootstrap to deviceclient.
* error handling and todos.
* retry method registration, minor fixes.
* nits, logging, removed exception handling for registering methods (it's handled by the parent method)
* bumped version
* code style nits
* Methods use scripts.
Desired properties are polled in UpdateDeviceState and delta is copied into Reported property values.
Telemetry is not sent if the device is marked other than online == "True"
* code style/cleanup changes.
change get query for simulations to linq query rather than iterating the list.
* update version
* Exception handling for updatedevicestate
nit for rename method execution endpoint to include Async suffix.
* nit fix comment code style
* Inject scriptinterpreter. refactoring.
* nits
* state update bug
* no need for scriptengine null check
* Bootstrap individual methods, implement decreasepressure aka EMergencyValveRelease
* remove connection string :(
* fixed threading problem w/ methods, implemented & tested first four methods
* Implement switch to turn random telemetry off; e.g. when increasepressure is called, wait until decreasepressure is called to turn it back on.
* remove clearing of DeviceMethodStatus
* Code style changes + fix some jslint errors
* Fix js methods files
* DeviceMethods code style changes
* Devices.cs code style
* JavascriptInterpreter.cs code style
* script changes for messages, docs
* Cleanup up Javascript functions, create Issue to track implementing functions not yet done.
* Halve the amount of telemetry sent
* Address PR comments.
* issue for script interpreter todo
* todos, nits, pr feedback
* Complete seed data, complete all the device models as per specs
* Rename “device type” to “device model”
* Clean JS syntax, check with JS Lint
* Generate device IDs from device model ID, instead of device model Name
* Complete health check endpoint
* Store simulations using the storage adapter
* Implement device bootstrap scenario: when a new device is created, mark it as a simulated device and report some properties like device type, messages schema, and initial location.
* Remove dependency on IoT Hub manager and access IoT Hub directly
* Refactor state machine to reduce complexity and reuse code
* Add launch settings for Visual Studio
* Remove env var used for the web service TCP port
* Improve logging of exceptions to avoid log flooding
* Fix messages format, to always use the “_unit” convention
* Add JSON config checks
* When a new device is created, mark it as a simulated device and
* When a device connects report some properties like device type, messages schema, and initial location. Use MQTT for twin operations
* Remove dependency on IoT Hub manager and access IoT Hub directly
* Refactor state machine to reduce complexity and reuse code
* Add launch settings for Visual Studio
* Remove env var used for the web service TCP port
* Improve logging of exceptions to avoid log flooding
* Fix messages format, to always use the “_unit” convention
* Bump version to 0.1.4
* Add Configuration settings for CORS whitelist
* Check for null values generated in deserializing incorrect CORS whitelist
* Hard code CORS whitelist
* Remove reference to environment variable PCS_DEVICESIMULATION_CORS_WHITELIST
Hugh Xiong
sasathy
Avani Patel
Peter Felts
Devis Lucato
Harleen Thind
Isaac
Jill Bender
Mary Ellen Chaffin
Devis Lucato
Tim Laverty
Xiangzhi Sheng