Document the concepts behind SHIELD and a full example.

docs/dev/concepts.rst

@@ -0,0 +1,177 @@
+Concepts
+========
+Normandy is a component in a larger system called SHIELD. The end goal of
+SHIELD is to allow Firefox to perform a variety of actions to aid the user and
+gather feedback to guide Mozilla in developing a better browser for them. To do
+this, we need a way to instruct Firefox to perform certain actions without
+having to download an update or wait for a new release.
+
+SHIELD is split into three main concepts: Actions, Recipes, and the runtime.
+This document describes those concepts in detail.
+
+.. _actions:
+
+Actions
+-------
+An action is JavaScript code that describes a process to perform within a user's
+browser. The code is designed to run in a sandbox that has no privileged access
+to the rest of the browser except through a :doc:`driver object </dev/driver>`
+that has a constrained API for performing sensitive actions, such as showing a
+survey to a user or getting information about the browser itself.
+
+Actions can accept a configuration object that is specified within the Normandy
+control interface. This configuration, which is part of a
+:ref:`recipe <recipes>`, contains static data required by the action, such as
+translated strings or boolean options for controlling behavior.
+
+For example, a survey action would include code to check whether we have showed
+a survey to the user recently, to determine which variant of a survey to show,
+and the call to the driver to actually display the survey itself. The
+configuration for the survey action would contain the text of the survey prompt
+and the URL to redirect the user to after answering.
+
+Actions are stored in the Normandy code repository and the
+:ref:`runtime <runtime>` fetches the actions from the Normandy service during
+execution. Here's an example of an action that logs a message to the console:
+
+.. code-block:: javascript
+
+   export default class ConsoleLogAction {
+     constructor(normandy, recipe) {
+       this.normandy = normandy; // The driver object
+       this.recipe = recipe; // Contains recipe arguments and other data
+     }
+
+     // Actions should return a Promise when executed
+     async execute() {
+       this.normandy.log(this.recipe.arguments.message, 'info');
+     }
+   }
+
+   // registerAction is in the sandbox's global object
+   registerAction('console-log', ConsoleLogAction);
+
+.. _recipes:
+
+Recipes
+-------
+A recipe is a model that represents a single instance of an action we want to
+perform. Recipes contain three important pieces of information:
+
+1. Client filtering data that determines which users the recipe should be run
+   for. This takes the form of a JEXL_ expression that is evaluated on the
+   client and has access to several pieces of data about the client, such as the
+   data collected by Telemetry_.
+2. A pointer to the :ref:`action <actions>` that this recipe should perform when
+   run.
+3. The configuration data to pass to the Action code when it is executed, called
+   the recipe's **arguments**.
+
+Recipes are stored in the Normandy database and the :ref:`runtime <runtime>`
+fetches them from the service during execution. Here's an example of a recipe in
+JSON format as returned by Normandy:
+
+.. code-block:: json
+
+   {
+      "id": 1,
+      "name": "Console Log Test",
+      "revision_id": 12,
+      "action": {
+         "name": "console-log",
+         "implementation_url": "https://self-repair.mozilla.org/api/v1/action/console-log/implementation/8ee8e7621fc08574f854972ee77be2a5280fb546/",
+         "arguments_schema": {
+            "$schema": "http://json-schema.org/draft-04/schema#",
+            "description": "Log a message to the console",
+            "type": "object",
+            "properties": {
+               "message": {
+                  "default": "",
+                  "description": "Message to log to the console",
+                  "type": "string",
+               }
+            },
+            "required": [
+               "message"
+            ],
+         }
+      },
+      "arguments": {
+         "message": "It works!"
+      }
+   }
+
+Note that the recipe as returned by the API also contains data about the action,
+including a schema describing the arguments field.
+
+.. _JEXL: https://github.com/TechnologyAdvice/Jexl
+.. _Telemetry: https://wiki.mozilla.org/Telemetry
+
+.. _runtime:
+
+Runtime
+-------
+The runtime is an execution environment that can affect a running instance of
+Firefox. Upon activation (typically a few moments after Firefox launches), the
+runtime:
+
+1. Downloads :ref:`recipes <recipes>` from the Normandy service.
+2. Verifies the signature of the recipes.
+3. Evaluates the recipe filters and filters out recipes that do not match the
+   client the runtime is installed within.
+4. Downloads the :ref:`actions <actions>` for the remaining recipes.
+5. Executes the action code for each recipe in a sandbox, passing in the
+   arguments from the recipe, and a :doc:`driver object </dev/driver>` containing
+   methods that can perform privileged actions.
+
+The runtime is implemented as a system add-on in the normandy-addon_ repository.
+
+.. _normandy-addon: https://github.com/mozilla/normandy-addon
+
+Threat Model
+------------
+Since the goal of SHIELD is to allow Mozilla to perform certain privileged
+actions quickly without shipping full updates to Firefox, it is a tempting
+target for compromising Firefox users. Normandy includes several security
+controls to help mitigate this risk.
+
+Action and Recipe Signing
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Actions and recipes that are downloaded by the runtime are signed according to
+the Content-Signature protocol as provided by the autograph_ service. The
+runtime verifies the signature upon downloading the recipes and actions,
+ensuring that the runtime only executes recipes that have been signed with a
+Mozilla-controlled key.
+
+This helps prevent Man-in-the-Middle attacks where an adversary pretends to be
+the remote Normandy service.
+
+.. _autograph: https://github.com/mozilla-services/autograph
+
+Peer Approval
+^^^^^^^^^^^^^
+Recipes cannot be enabled in the Normandy admin interface without going through
+an approval process. One user must submit the recipe for approval, and a
+separate user must approve the recipe before it can be distributed by the
+service.
+
+This helps prevent compromise of a single account from compromising the entire
+service, since two accounts need to be compromised to publish a recipe.
+
+Action Sandbox
+^^^^^^^^^^^^^^
+Actions are executed within a JavaScript sandbox by the runtime. The sandbox
+limits the access of the JavaScript to prevent it from modifying Firefox in ways
+that haven't been reviewed and approved beforehand.
+
+To perform actions that JavaScript normally can't (such as displaying a
+Heartbeat survey), the action in the sandbox is passed a
+:doc:`driver object </dev/driver>`, which contains methods that can modify the
+client or trigger other privileged behavior.
+
+Configurable Admin
+^^^^^^^^^^^^^^^^^^
+The admin interface for Normandy can be disabled via a Django setting, which
+allows for disabling the admin interface on public-facing web servers and
+running them with read-only privileges. The writable admin interface is then
+deployed behind a VPN to restrict access to authorized users.

docs/dev/self-repair.rst

@@ -1,29 +1,7 @@
-Architecture
-============
-Normandy is a component in a larger system called SHIELD. The end goal of
-SHIELD is to allow Firefox to perform a variety of actions to aid the user and
-gather feedback to guide us in developing a better browser for them. To do
-this, we need a way to instruct Firefox to perform certain actions without
-having to download an update or wait for a new release.
-
-To accomplish this goal, the SHIELD system consists of:
-
-Normandy
-   Service that acts as the source of actions that we wish to perform, and
-   determines which users receive which recipes.
-Self-Repair Client
-   Ships with Firefox and handles fetching recipes and action implementations
-   from Normandy and executing them within Firefox.
-
 Self-Repair Protocol
---------------------
+====================
 The following section describes how Firefox retrieves and executes recipes from
-Normandy.
-
-.. image:: /resources/self-repair-sequence.png
-   :width: 460
-   :height: 535
-   :align: center
+Normandy via the self-repair iframe.
 
 When Firefox is launched by a user, it makes the following series of requests:
 

docs/index.rst

@@ -10,7 +10,9 @@ JavaScript to various clients based on certain rules.
 
    dev/install
    dev/workflow
-   dev/architecture
+   dev/concepts
+   qa/example
+   dev/self-repair
    dev/driver
    dev/troubleshooting
    qa/docker

docs/qa/example.rst

@@ -0,0 +1,154 @@
+Example
+=======
+This document describes the end-to-end process of developing, shipping, and
+executing a new action with SHIELD, in order to illustrate in detail the
+mechanics of the system.
+
+In this example, we want to add the ability to SHIELD to prompt users with a
+pop-up notification with some configurable text, and only prompts users once.
+
+1. Adding a Function to the Driver
+----------------------------------
+The first step is to add a new function to the :doc:`driver </dev/driver>` that
+will trigger the notification on the client. The driver is implemented in the
+`system add-on <normandy-addon>`_, so we update the add-on to implement the
+function:
+
+.. js:function:: showNotification(message)
+
+   Display a pop-up notification to the user.
+
+   :param message: Message to show in the pop-up.
+
+The change has to be merged into the system add-on and released before it will
+be available to actions to use.
+
+2. Write a Notification Action
+------------------------------
+Next, we must develop an :ref:`action <actions>` that uses ``showNotification``
+to display the notification given in the action's arguments. This includes the
+logic for only showing the notification if the user hasn't seen it before:
+
+.. code-block:: javascript
+
+   export default class NotificationAction {
+     constructor(normandy, recipe) {
+       this.normandy = normandy; // The driver object
+       this.recipe = recipe; // Contains recipe arguments and other data
+
+       // Persistent data store on the client.
+       this.storage = normandy.createStorage(recipe.id);
+     }
+
+     async execute() {
+       // Check if we've shown the notification previously, and show it if we
+       // have not.
+       const haveShownPreviously = await this.storage.getItem('shownPreviously');
+       if (!haveShownPreviously) {
+         this.storage.setItem('shownPreviously', true);
+         this.normandy.showNotification(this.recipe.arguments.message);
+       }
+     }
+   }
+
+   registerAction('notification', ConsoleLogAction);
+
+.. note:: In addition to the code above, actions have to define a `JSON Schema`_
+   as well as a form for the arguments. These are used in the control interface
+   in Normandy as well as in the system add-on to validate arguments.
+
+After creating the action, it must be deployed to the Normandy service before it
+can be used in a recipe.
+
+.. _JSON Schema: http://json-schema.org/
+
+3. Create a Recipe
+------------------
+The next step is to create a :ref:`recipe <recipes>` that uses the
+``notification`` action that we created above. This is done via the control
+interface on the Normandy service itself. Important fields to input via the
+web interface include:
+
+- **Action**: The ``notification`` action.
+- **Filter Expression**: A JEXL_ statement to filter the recipe so that only
+  certain users see it. For testing purposes, the expression ``true`` will match
+  all users.
+- **Arguments**: A single ``message`` field containing the message to display.
+
+Once created, the recipe will need to be submitted for peer review and approved
+by another users before it is enabled within the service.
+
+.. _JEXL: https://github.com/TechnologyAdvice/Jexl
+
+4. Delivery
+-----------
+Once the recipe is enabled, the service will include it in queries to the
+recipe API. The system add-on performs the following steps to fetch and execute
+our new recipe:
+
+1. Upon activation, send a ``POST`` request to ``/api/v1/recipe/?enabled=true``
+   to retrieve all currently-enabled recipes. This returns a JSON response that
+   looks similar:
+
+   .. code-block:: json
+
+      [
+         {
+              "id": 1,
+              "name": "Notification",
+              "enabled": true,
+              "revision_id": 1,
+              "action_name": "notification",
+              "arguments": {
+                  "message": "Notification message!"
+              },
+              "filter_expression": "true"
+         }
+      ]
+
+   .. note:: Some fields were removed from the response above for readability.
+
+2. For each recipe, evaluate its ``filter_expression`` field as a JEXL_
+   expression against a context containing information about the client and
+   environment that it is running in. If the expression returns true, then the
+   recipe matches the client and will be run. Otherwise, the recipe is
+   discarded.
+
+   The ``/api/v1/classify_client/`` API endpoint is used to populate the context
+   with the current server time and the country the user is located in via IP
+   address geolocation.
+3. For each matching recipe, download the action specified in the recipe if it
+   hasn't been downloaded yet. Actions served from URLs of the form
+   ``/api/v1/action/notification/`` and return a response that looks like:
+
+   .. code-block:: json
+
+      {
+         "name": "show-heartbeat",
+         "implementation_url": "https://self-repair.mozilla.org/api/v1/action/notification/implementation/4574dbc126af07cd031a0da29d625a11365403ea/",
+         "arguments_schema": {
+               "$schema": "http://json-schema.org/draft-04/schema#",
+               "title": "Display a pop-up notification",
+               "type": "object",
+               "required": [
+                   "message"
+               ],
+               "properties": {
+                   "message": {
+                       "description": "Message to show in the notification",
+                       "type": "string",
+                       "default": ""
+                   }
+               }
+           }
+      }
+
+   In addition, the JavaScript code for the action is downloaded via the URL in
+   the ``implementation_url`` property of the response above.
+4. For each matching recipe, execute the action associated with it in a sandbox,
+   passing in information about the recipe (including its arguments) and the
+   driver object.
+
+After these steps, the ``notification`` action and recipe that we created will
+have been downloaded and executed, and the user will see a notification pop up.
+Future runs of that specific recipe will not show a notification.