From cc5f323ee6d4b8458ae62e6d6367fc7208b9abe0 Mon Sep 17 00:00:00 2001 From: Michael Cooper Date: Mon, 3 May 2021 09:50:31 -0700 Subject: [PATCH] Add retrospective ADR for usage of remote settings This may help when reviewing recent problems with the signing infrastructure, and to better undrest and Normandy's history. Fixes #2309 --- docs/adrs/adr-0002-remote-settings.rst | 131 +++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/adrs/adr-0002-remote-settings.rst diff --git a/docs/adrs/adr-0002-remote-settings.rst b/docs/adrs/adr-0002-remote-settings.rst new file mode 100644 index 00000000..92531ec1 --- /dev/null +++ b/docs/adrs/adr-0002-remote-settings.rst @@ -0,0 +1,131 @@ +ADR 0002 - Syncing data to Firefox with Remote Settings +======================================================= + +* Status: retrospective +* Last Updated: 2021-05-03 + +.. note:: + + This is a *retrospective* ADR, that documents a past decision well after + the time the decision was actually made. It's accuracy may be limited by + the distance in time between the decision and the time of record. + +Context and Problem Statement +----------------------------- + +Normandy maintains a collection of recipes, and Firefox needs to access this +collection periodically to determine if any of the recipes are applicable to +that instance of Firefox. How should Firefox sync the collection of recipes +from Normandy Server? + +Decision Drivers +---------------- + +* The collection of recipes changes slowly, and so refetching the entire + collection every time is wasteful. +* It is important to have up-to-date versions of the recipes, so the update + interval should be relatively short. +* Updates to the collection of recipes should be secure. + +Considered Options +------------------ + +1. Fetch all recipes from Normandy via an HTTP request each run +2. Implement a smart syncing system that would only download changed recipes +3. Process recipe filters on the server, reducing the data that clients download +4. Use Remote Settings to sync recipes to Firefox + +Decision Outcome +---------------- + +Chosen option: "option 4", Use Remote Settings. This option provides +efficient updates without sacrificing update frequency or user privacy, and +is a wise use of our limited development resources, since it re-uses existing +tools. Ultimately it provided a better product, after some additional +engineering work. + +Positive Consequences +~~~~~~~~~~~~~~~~~~~~~ + +As a result of using Remote Settings to synchronize data to clients, +Normandy's servers have lighter load, clients have less to download, and +uptake is faster thanks to Remote Settings's push notification support. +Additionally, Normandy helped design some changes to Remote Settings that +make server-to-server work loads better. Additionally, Normandy served as an +example for other Remote Settings integration, including as an inspiration +for Nimbus's sync model. + +Negative Consequences +~~~~~~~~~~~~~~~~~~~~~ + +Switching to Remote Settings was not without its growing pains. There were +several problems relating to the push notification system. When clients +updated recipes in relation to a push service, they would often overwhelm +various services, including Normandy's Classify-Client (which got rewritten +in Rust for performance), Telemetry (rate limiting was added to mitigate +this), and Remote Settings itself (caching bugs were revealed and fixed). + +Additionally, this adds another layer of indirection that can make tracking +down issues more complicated. When unexpected behavior is found, there is +another system to consider. + +Pros and Cons of the Options +---------------------------- + +1. Fetch all recipes from Normandy via an HTTP request each run +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is the behavior that Normandy was originally designed, built, and ran +for several years with. All active recipes are listed with the needed details +in a single API endpoint that clients can fetch, and each client calls this +endpoint every time it processes Normandy recipes. + +* Good, because it is proven to work in real world uses cases for several years. +* Good, because it gives Normandy more control over its syncing process. +* Bad, because it produces higher load on the servers than needed. +* Bad, because clients needlessly redownload data multiple times. +* Bad, because a service interruption causes client side issues, ranging from + wasted resources and retries to experiment failure. + +2. Implement a smart syncing system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Clients could keep track of a last modified timestamp, or some other system +that could allow them to communicate their sync state to the server. The +server would then send them only changed recipes. + +* Good, because it reduces the amount of data transferred. +* Good, because Normandy don't have to rely on a new outside services. +* Bad, because it is a complicated system to implement on both server and client side. +* Bad, because it re-invents a system already accessible in Firefox. + +3. Process recipes on the server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Instead of downloading all recipes and filtering them locally, clients would +assemble a filtering context and then send it to the server. The server would +evaluate the filter rules of each recipe, and then reply to the client with +only the relevant recipes. + +* Good, because it reduces the amount of data transferred (though doesn't + entirely deduplicate those transfers). +* Good, because it simplifies the execution model of Normandy, giving us more + insight and control into the system. +* Bad, because it either reduces our filtering capabilities, or requires + sending more sensitive data to Normandy. +* Bad, because it is more expensive to host. It requires increased + per-request-processing and responses are not very cacheable with current + filter criteria. + +4. Use Remote Settings to sync recipes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Normandy server would maintain a Remote Settings collection with all active +recipes. Clients would use the existing Remote Settings protocol to +efficiently synchronize data between the server and a local store. + +* Good, because less data is transferred. +* Good, because it uses pre-existing components that are already well tested. +* Good, because at the time, Remote Settings and Normandy were managed by the same team. +* Good, because we can push-notifications. +* Bad, because it increases the complexity of the system.