Scale In/Out
All the services in notification provider can be scaled in/out independently.
-
Notification Handler: This service exposes endpoints to be consumed and is forefront for most of the actions such sending asynchronous notifications, managing templates, getting notifications history etc. As per the load requirement this service can be scaled in/out. The important point to note here is, whenever this service is scaled, the other services Notification queue processor (Function App) and Notification Service (Azure App Service) must also be scaled.
-
Notification Service: This service is consumed by Function App to process queued notifications. This service can be scaled in/out as per load requirement to send notifications. This service can be used to send synchronous notifications in some scenarios which requires notifications be sent immediately.
-
Notification Queue Processor: This service consumes the notification message from queue and gets those processed with Notification Service. This service can be scaled in/out without any issues. No duplicate message will be processed even if this service has multiple instance running.
Performance Improvement
The default batch-size for message consumption from queue using Queue Triggered Function Apps (Notification Queue Processor) is 1. We can set the default batch-size of message consumption using configuration key AzureFunctionsJobHost_extensions_queues_batchSize. This setting will improve the notification message processing performance drastically. Please see the configuration highlighted in yellow in below image.
Performance Metrics with respect to configuration value
| Notification Provider | Configuration Value (AzureFunctionsJobHost_extensions_queues_batchSize) | Message/Min |
|---|---|---|
| Graph | Default (1) | 65+ |
| Graph | 8 | 160+ |
| Graph | 16 | 350+ |
| Graph | 32 | 800+ |
| DirectSend | Default (1) | 65+ |
| DirectSend | 8 | 300+ |
| DirectSend | 16 | 650+ |
| DirectSend | 32 | 2000+ |
Performance Limitations
-
Notification Send for some notifications(~20-30 messages) may fail on higher load ( more than 500 message per min) with exception Application is over its MailboxConcurrency limit.This exception is because of throttling limit from Graph Apis. The failed messages will be successful when retried with resend. It is advisable to keep the configuration value with provider type Graph must not exceed 16 with single instance of Notification Service.
-
More instances of Notification Service may also cause Application is over its MailboxConcurrency limit exception, as more instance will try creating more number of concurrent connections.