From 7c4a70049ef3d3e8f6cea71d0484e4c8290cc359 Mon Sep 17 00:00:00 2001 From: Julien Vehent Date: Tue, 6 Oct 2015 09:29:38 -0400 Subject: [PATCH] [doc] tons of new docs, with some CSS changes --- doc/agent.rst | 336 ++++++++++++++- doc/agent.rst.html | 235 ++++++++++- doc/api.rst | 363 ++++++++--------- doc/api.rst.html | 236 +++++------ doc/cheatsheet.rst.html | 6 +- doc/command_line.rst | 34 ++ doc/command_line.rst.html | 247 ++++++++++++ doc/concepts.rst | 329 +-------------- doc/concepts.rst.html | 249 +----------- doc/configuration.rst | 1 - doc/configuration.rst.html | 8 +- doc/console.rst | 495 +++++++++++++++++++++++ doc/console.rst.html | 605 ++++++++++++++++++++++++++++ doc/data.rst | 1 - doc/data.rst.html | 8 +- doc/docstyle.css | 6 +- doc/mig-worker-agent-intel.html | 6 +- doc/mig-worker-compliance-item.html | 6 +- doc/module_file.html | 6 +- doc/module_memory.html | 6 +- doc/module_ping.html | 6 +- doc/module_pkg.html | 6 +- doc/module_scribe.html | 6 +- doc/module_timedrift.html | 6 +- doc/modules.rst | 1 - doc/modules.rst.html | 8 +- doc/runner.rst | 1 - doc/runner.rst.html | 293 ++++++++++++++ 28 files changed, 2607 insertions(+), 903 deletions(-) create mode 100644 doc/command_line.rst create mode 100644 doc/command_line.rst.html create mode 100644 doc/console.rst create mode 100644 doc/console.rst.html create mode 100644 doc/runner.rst.html diff --git a/doc/agent.rst b/doc/agent.rst index 8bc08030..3ba0b0b2 100644 --- a/doc/agent.rst +++ b/doc/agent.rst @@ -1,10 +1,234 @@ -Agent -===== +====================== +MIG Agent Architecture +====================== -The agent accepts different classes of inputs on stdin, as one-line JSON objects. The most common one is the -``parameters`` class, but it could also receive a ``stop`` input that -indicates that the module should stop its execution immediately. The format of -module input messages is defined by ``modules.Message``. +.. sectnum:: +.. contents:: Table of Contents + +Initialization process +---------------------------- +The agent tries to be as autonomous as possible. One of the goal is to ship +agents without requiring external provisioning tools, such as Chef or Puppet. +Therefore, the agent attempts to install itself as a service, and also supports +a builtin upgrade protocol (described in the next section). + +As a portable binary, the agent needs to detect the type of operating system +and init method that is used by an endpoint. Depending on the endpoint, +different initialization methods are used. The diagram below explains the +decision process followed by the agent. + +.. image:: .files/mig-agent-initialization-process.png + +Go does not provide support for running programs in the backgroud. On endpoints +that run upstart, systemd (linux) or launchd (darwin), this is not an issue +because the init daemon takes care of running the agent in the background, +rerouting its file descriptors and restarting on crash. On Windows and System-V, +however, the agent daemonizes by forking itself into `foreground` mode, and +re-forking itself on error (such as loss of connectivity to the relay). +On Windows and System-V, if the agent is killed, it will not be restarted +automatically. + +Registration process +~~~~~~~~~~~~~~~~~~~~ + +The initialization process goes through several environment detection steps +which are used to select the proper init method. Once started, the agent will +send a heartbeat to the public relay, and also store that heartbeat in its +`run` directory. The location of the `run` directory is platform specific. + +* windows: C:\Windows\ +* darwin: /Library/Preferences/mig/ +* linux: /var/run/mig/ + +Below is a sample heartbeat message from a linux agent stored in +`/var/run/mig/mig-agent.ok`. + +.. code:: json + + { + "destructiontime": "0001-01-01T00:00:00Z", + "environment": { + "arch": "amd64", + "ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)", + "init": "upstart" + }, + "heartbeatts": "2014-07-31T14:00:20.00442837-07:00", + "name": "someserver.example.net", + "os": "linux", + "pid": 26256, + "queueloc": "linux.someserver.example.net.5hsa811oda", + "starttime": "2014-07-30T21:34:48.525449401-07:00", + "version": "201407310027+bcbdd94.prod" + } + +Check-In mode +~~~~~~~~~~~~~ +In infrastructure where running the agent as a permanent process is not +acceptable, it is possible to run the agent as a cron job. By starting the +agent with the flag **-m agent-checkin**, the agent will connect to the +configured relay, retrieve and run outstanding commands, and exit after 10 +seconds of inactivity. + +Communication with modules +-------------------------- + +Upon processing of an action, the scheduler will retrieve a list of agents to +send the action to. One action is then derived into multiple commands and sent +to agents. + +An agent receives a command from the scheduler on its personal AMQP queue (1). +It parses the command (2) and extracts all of the operations to perform. +Operations are passed to modules and executed in parallel (3). Rather than +maintaining a state of the running command, the agent create a goroutine and a +channel tasked with receiving the results from the modules. Each modules +published its results inside that channel (4). The result parsing goroutine +receives them, and when it has received all of them, populates the `results` (5) +array of the command with the results from each module, and send the command +back to the scheduler(6). + +When the agent is done running the command, both the channel and the goroutine +are destroyed. + + :: + + +-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ] + |command|+---->(listener) + +-------+ |(2) + ^ V + |(1) (parser) + | + [ m o d u l e s ] + +---------+ | (3)|----------> op1 +----------------+ + |SCHEDULER|+---+ |------------> op2 +--------------| + | |<---+ |--------------> op3 +------------| + +---------+ | +----------------> op4 +----------+ + | V(4) + |(6) (receiver) + | | + | V(5) + + (publisher) + +-------+ / + |results|<-----------------------------------------' + +-------+ + +The command received by the agent is composed of a copy of the action described +previously, but signed with the private key of a trusted investigator. It also +contains additional parameters that are specific to the targetted agent, such as +command processing timestamps, name of the agent queue on the message broker, +action and command unique IDs and status and results of the command. Below is an +command derived from the root password checking action, and ran on the host named +'host1.example.net'. + +.. code:: json + + { + "id": 1.427392971126604e+18, + "action": { ... SIGNED COPY OF THE ACTION ... }, + "agent": { + "id": 1.4271760437936648e+18, + "name": "host1.example.net", + "queueloc": "linux.host1.example.net.981alsd19aos1984", + "mode": "daemon", + "version": "20150324+0d0f88c.prod" + }, + "status": "success", + "results": [ + { + "foundanything": true, + "success": true, + "elements": { + "root_passwd_hashed_or_disabled": [ + { + "file": "/etc/shadow", + "fileinfo": { + "lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC", + "mode": "----------", + "size": 1684 + }, + "search": { + "contents": [ + "root:(\\*|!|\\$(1|2a|5|6)\\$).+" + ], + "options": { + "matchall": false, + "matchlimit": 0, + "maxdepth": 0 + }, + "paths": [ + "/etc" + ] + } + } + ] + }, + "statistics": { + "exectime": "2.017849ms", + "filescount": 1, + "openfailed": 0, + "totalhits": 1 + }, + "errors": null + } + ], + "starttime": "2015-03-26T18:02:51.126605Z", + "finishtime": "2015-03-26T18:03:00.671232Z" + } + +The results of the command show that the file '/etc/shadow' has matched, and +thus "FoundAnything" returned "True". + +The invocation of the file module has completed successfully, which is +represented by **results->0->success=true**. In our example, there is only one +operation in the **action->operations** array, so only one result is present. +When multiple operations are performed, each has its results listed in a +corresponding entry of the results array (operations[0] is in results[0], +operations[1] in results[1], etc...). + +Finally, the agent has performed all operations in the operations array +successfully, and returned **status=success**. Had a failure happened on the +agent, the returned status would be one of "failed", "timeout" or "cancelled". + +Command expiration & timeouts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To prevent abuse of resources, agents will kill long-running modules after a +given period of time. That timeout is hardcoded in the agent configuration +at compile time and defaults to 5 minutes. + +.. code:: go + + // timeout after which a module run is killed + var MODULETIMEOUT time.Duration = 300 * time.Second + +That timeout represents the **maximum** execution time of a single operation. If +an action contains 3 operations, each operation gets its own timeout. But because +operations run in parallel in the agent, the maximum runtime of an action should +be very close to the value of MODULETIMEOUT. + +In a typical deployment, it is safe to increase MODULETIMEOUT to allow for +longer operations. A value of 20 minutes is usual. Make sure to fine tune this +to your environment, and get the approval of your ops team because mig-agent +may end up consuming resources (but never more than 50% of the cpu available on +a system). + +Oftentimes, an investigator will want a timeout that is much shorter than the value +of MODULETIMEOUT. In the MIG command line, the flag `-e` controls the +expiration. It defaults to 5 minutes but can be set to 30 seconds for simple +investigations. When that happens, the agent will calculate an appropriate expiration +for the operations being run. If the expiration set on the action is set to 30 seconds, +the agent will kill operations that run for more than 30 seconds. + +If the expiration is larger than the value of MODULETIMEOUT (for example, 2 +hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to +allow agents that only check in periodically to pick up actions long after they +are launched. + +Agent/Modules message format +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The agent accepts different classes of inputs on stdin, as one-line JSON objects. +The most common one is the ``parameters`` class, but it could also receive a +``stop`` input that indicates that the module should stop its execution immediately. +The format of module input messages is defined by ``modules.Message``. .. code:: go @@ -25,4 +249,104 @@ and copies them into ``Message.Parameters``. It then sets ``Message.Class`` to ``modules.MsgClassParameters``, marshals the struct into JSON, and pass the resulting ``[]byte`` to the module as an IO stream. +Agent upgrade process +--------------------- +MIG supports upgrading agents in the wild. The upgrade protocol is designed with +security in mind. The flow diagram below presents a high-level view: + :: + + Investigator Scheduler Agent NewAgent FileServer + +-----------+ +-------+ +---+ +------+ +--------+ + | | | | | + | 1.initiate | | | | + |------------------>| | | | + | | 2.send command | | | + | |------------------>| 3.verify | | + | | |--------+ | | + | | | | | | + | | | | | | + | | |<-------+ | | + | | | | | + | | | 4.download | | + | | |-------------------------------------->| + | | | | | + | | | 5.checksum | | + | | |--------+ | | + | | | | | | + | | | | | | + | | |<-------+ | | + | | | | | + | | | 6.exec | | + | | |------------------>| | + | | 7.return own PID | | | + | |<------------------| | | + | | | | | + | |------+ 8.mark | | | + | | | agent as | | | + | | | upgraded | | | + | |<-----+ | | | + | | | | | + | | 9.register | | | + | |<--------------------------------------| | + | | | | | + | |------+10.find dup | | | + | | |agents in | | | + | | |registrations | | + | |<-----+ | | | + | | | | | + | | 11.send command to kill PID old agt| | + | |-------------------------------------->| | + | | | | | + | | 12.acknowledge | | | + | |<--------------------------------------| | + +All upgrade operations are initiated by an investigator (1). The upgrade is +triggered by an action to the upgrade module with the following parameters: + +.. code:: json + + "Operations": [ + { + "Module": "upgrade", + "Parameters": { + "linux/amd64": { + "to_version": "16eb58b-201404021544", + "location": "http://localhost/mig/bin/linux/amd64/mig-agent", + "checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03" + } + } + } + ], + +* Each OS family and architecture have their own parameters (ex: "linux/amd64", + "darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have: +* to_version is the version an agent should upgrade to +* location points to a HTTPS address that contains the agent binary +* checksum is a SHA256 hash of the agent binary to be verified after download + +The parameters above are signed using a standard PGP action signature. + +The upgrade action is forwarded to agents (2) like any other action. The action +signature is verified by the agent (3), and the upgrade module is called. The +module downloads the new binary (4), verifies the version and checksum (5) and +installs itself on the system. + +Assuming everything checks in, the old agent executes the binary of the new +agent (6). At that point, two agents are running on the same machine, and the +rest of the protocol is designed to shut down the old agent, and clean up. + +After executing the new agent, the old agent returns a successful result to the +scheduler, and includes its own PID in the results. +The new agent starts by registering with the scheduler (7). This tells the +scheduler that two agents are running on the same node, and one of them must +terminate. The scheduler sends a kill action to both agents with the PID of the +old agent (8). The kill action may be executed twice, but that doesn't matter. +When the scheduler receives the kill results (9), it sends a new action to check +for `mig-agent` processes (10). Only one should be found in the results (11), +and if that is the case, the scheduler tells the agent to remove the binary of +the old agent (12). When the agent returns (13), the upgrade protocol is done. + +If the PID of the old agent lingers on the system, an error is logged for the +investigator to decide what to do next. The scheduler does not attempt to clean +up the situation. diff --git a/doc/agent.rst.html b/doc/agent.rst.html index 142ff561..0c49f473 100644 --- a/doc/agent.rst.html +++ b/doc/agent.rst.html @@ -1,6 +1,6 @@

Agent

The agent accepts different classes of inputs on stdin, as one-line JSON objects. The most common one is the -parameters class, but it could also receive a stop input that -indicates that the module should stop its execution immediately. The format of -module input messages is defined by modules.Message.

// Message defines the input messages received by modules.
+
MIG Agent Architecture
1   Initialization process
The agent tries to be as autonomous as possible. One of the goal is to ship
+agents without requiring external provisioning tools, such as Chef or Puppet.
+Therefore, the agent attempts to install itself as a service, and also supports
+a builtin upgrade protocol (described in the next section).
As a portable binary, the agent needs to detect the type of operating system
+and init method that is used by an endpoint. Depending on the endpoint,
+different initialization methods are used. The diagram below explains the
+decision process followed by the agent.
.files/mig-agent-initialization-process.png
Go does not provide support for running programs in the backgroud. On endpoints
+that run upstart, systemd (linux) or launchd (darwin), this is not an issue
+because the init daemon takes care of running the agent in the background,
+rerouting its file descriptors and restarting on crash. On Windows and System-V,
+however, the agent daemonizes by forking itself into foreground mode, and
+re-forking itself on error (such as loss of connectivity to the relay).
+On Windows and System-V, if the agent is killed, it will not be restarted
+automatically.
1.1   Registration process
The initialization process goes through several environment detection steps
+which are used to select the proper init method. Once started, the agent will
+send a heartbeat to the public relay, and also store that heartbeat in its
+run directory. The location of the run directory is platform specific.
  • windows: C:Windows
  • darwin: /Library/Preferences/mig/
  • linux: /var/run/mig/
Below is a sample heartbeat message from a linux agent stored in
+/var/run/mig/mig-agent.ok.
{
+        "destructiontime": "0001-01-01T00:00:00Z",
+        "environment": {
+                "arch": "amd64",
+                "ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)",
+                "init": "upstart"
+        },
+        "heartbeatts": "2014-07-31T14:00:20.00442837-07:00",
+        "name": "someserver.example.net",
+        "os": "linux",
+        "pid": 26256,
+        "queueloc": "linux.someserver.example.net.5hsa811oda",
+        "starttime": "2014-07-30T21:34:48.525449401-07:00",
+        "version": "201407310027+bcbdd94.prod"
+}
1.2   Check-In mode
In infrastructure where running the agent as a permanent process is not
+acceptable, it is possible to run the agent as a cron job. By starting the
+agent with the flag -m agent-checkin, the agent will connect to the
+configured relay, retrieve and run outstanding commands, and exit after 10
+seconds of inactivity.
2   Communication with modules
Upon processing of an action, the scheduler will retrieve a list of agents to
+send the action to. One action is then derived into multiple commands and sent
+to agents.
An agent receives a command from the scheduler on its personal AMQP queue (1).
+It parses the command (2) and extracts all of the operations to perform.
+Operations are passed to modules and executed in parallel (3). Rather than
+maintaining a state of the running command, the agent create a goroutine and a
+channel tasked with receiving the results from the modules. Each modules
+published its results inside that channel (4). The result parsing goroutine
+receives them, and when it has received all of them, populates the results (5)
+array of the command with the results from each module, and send the command
+back to the scheduler(6).
When the agent is done running the command, both the channel and the goroutine
+are destroyed.
             +-------+   [ - - - - - - A G E N T - - - - - - - - - - - - ]
+             |command|+---->(listener)
+             +-------+          |(2)
+               ^                V
+               |(1)         (parser)
+               |               +       [ m o d u l e s ]
++---------+    |            (3)|----------> op1 +----------------+
+|SCHEDULER|+---+               |------------> op2 +--------------|
+|         |<---+               |--------------> op3 +------------|
++---------+    |               +----------------> op4 +----------+
+               |                                                 V(4)
+               |(6)                                         (receiver)
+               |                                                 |
+               |                                                 V(5)
+               +                                             (publisher)
+             +-------+                                           /
+             |results|<-----------------------------------------'
+             +-------+
The command received by the agent is composed of a copy of the action described
+previously, but signed with the private key of a trusted investigator. It also
+contains additional parameters that are specific to the targetted agent, such as
+command processing timestamps, name of the agent queue on the message broker,
+action and command unique IDs and status and results of the command. Below is an
+command derived from the root password checking action, and ran on the host named
+'host1.example.net'.
{
+  "id": 1.427392971126604e+18,
+  "action": { ... SIGNED COPY OF THE ACTION ... },
+  "agent": {
+        "id": 1.4271760437936648e+18,
+        "name": "host1.example.net",
+        "queueloc": "linux.host1.example.net.981alsd19aos1984",
+        "mode": "daemon",
+        "version": "20150324+0d0f88c.prod"
+  },
+  "status": "success",
+  "results": [
+        {
+          "foundanything": true,
+          "success": true,
+          "elements": {
+                "root_passwd_hashed_or_disabled": [
+                  {
+                        "file": "/etc/shadow",
+                        "fileinfo": {
+                          "lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC",
+                          "mode": "----------",
+                          "size": 1684
+                        },
+                        "search": {
+                          "contents": [
+                                "root:(\\*|!|\\$(1|2a|5|6)\\$).+"
+                          ],
+                          "options": {
+                                "matchall": false,
+                                "matchlimit": 0,
+                                "maxdepth": 0
+                          },
+                          "paths": [
+                                "/etc"
+                          ]
+                        }
+                  }
+                ]
+          },
+          "statistics": {
+                "exectime": "2.017849ms",
+                "filescount": 1,
+                "openfailed": 0,
+                "totalhits": 1
+          },
+          "errors": null
+        }
+  ],
+  "starttime": "2015-03-26T18:02:51.126605Z",
+  "finishtime": "2015-03-26T18:03:00.671232Z"
+}
The results of the command show that the file '/etc/shadow' has matched, and
+thus "FoundAnything" returned "True".
The invocation of the file module has completed successfully, which is
+represented by results->0->success=true. In our example, there is only one
+operation in the action->operations array, so only one result is present.
+When multiple operations are performed, each has its results listed in a
+corresponding entry of the results array (operations[0] is in results[0],
+operations[1] in results[1], etc...).
Finally, the agent has performed all operations in the operations array
+successfully, and returned status=success. Had a failure happened on the
+agent, the returned status would be one of "failed", "timeout" or "cancelled".
2.1   Command expiration & timeouts
To prevent abuse of resources, agents will kill long-running modules after a
+given period of time. That timeout is hardcoded in the agent configuration
+at compile time and defaults to 5 minutes.
// timeout after which a module run is killed
+var MODULETIMEOUT time.Duration = 300 * time.Second
That timeout represents the maximum execution time of a single operation. If
+an action contains 3 operations, each operation gets its own timeout. But because
+operations run in parallel in the agent, the maximum runtime of an action should
+be very close to the value of MODULETIMEOUT.
In a typical deployment, it is safe to increase MODULETIMEOUT to allow for
+longer operations. A value of 20 minutes is usual. Make sure to fine tune this
+to your environment, and get the approval of your ops team because mig-agent
+may end up consuming resources (but never more than 50% of the cpu available on
+a system).
Oftentimes, an investigator will want a timeout that is much shorter than the value
+of MODULETIMEOUT. In the MIG command line, the flag -e controls the
+expiration. It defaults to 5 minutes but can be set to 30 seconds for simple
+investigations. When that happens, the agent will calculate an appropriate expiration
+for the operations being run. If the expiration set on the action is set to 30 seconds,
+the agent will kill operations that run for more than 30 seconds.
If the expiration is larger than the value of MODULETIMEOUT (for example, 2
+hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to
+allow agents that only check in periodically to pick up actions long after they
+are launched.
2.2   Agent/Modules message format
The agent accepts different classes of inputs on stdin, as one-line JSON objects.
+The most common one is the parameters class, but it could also receive a
+stop input that indicates that the module should stop its execution immediately.
+The format of module input messages is defined by modules.Message.
// Message defines the input messages received by modules.
 type Message struct {
         Class      string      // represent the type of message being passed to the module
         Parameters interface{} // for `parameters` class, this interface contains the module parameters
@@ -243,4 +391,77 @@ module input messages is defined by modules.Message.
Command.Action.Operations[N].Parameters
 and copies them into Message.Parameters. It then sets Message.Class to
 modules.MsgClassParameters, marshals the struct into JSON, and pass the
-resulting []byte to the module as an IO stream.

\ No newline at end of file
+resulting []byte to the module as an IO stream.
3   Agent upgrade process
MIG supports upgrading agents in the wild. The upgrade protocol is designed with
+security in mind. The flow diagram below presents a high-level view:
Investigator          Scheduler             Agent             NewAgent           FileServer
++-----------+         +-------+             +---+             +------+           +--------+
+          |                   |                   |                   |                   |
+          |    1.initiate     |                   |                   |                   |
+          |------------------>|                   |                   |                   |
+          |                   |  2.send command   |                   |                   |
+          |                   |------------------>| 3.verify          |                   |
+          |                   |                   |--------+          |                   |
+          |                   |                   |        |          |                   |
+          |                   |                   |        |          |                   |
+          |                   |                   |<-------+          |                   |
+          |                   |                   |                   |                   |
+          |                   |                   |    4.download     |                   |
+          |                   |                   |-------------------------------------->|
+          |                   |                   |                   |                   |
+          |                   |                   | 5.checksum        |                   |
+          |                   |                   |--------+          |                   |
+          |                   |                   |        |          |                   |
+          |                   |                   |        |          |                   |
+          |                   |                   |<-------+          |                   |
+          |                   |                   |                   |                   |
+          |                   |                   |      6.exec       |                   |
+          |                   |                   |------------------>|                   |
+          |                   |  7.return own PID |                   |                   |
+          |                   |<------------------|                   |                   |
+          |                   |                   |                   |                   |
+          |                   |------+ 8.mark     |                   |                   |
+          |                   |      | agent as   |                   |                   |
+          |                   |      | upgraded   |                   |                   |
+          |                   |<-----+            |                   |                   |
+          |                   |                   |                   |                   |
+          |                   |    9.register     |                   |                   |
+          |                   |<--------------------------------------|                   |
+          |                   |                   |                   |                   |
+          |                   |------+10.find dup |                   |                   |
+          |                   |      |agents in   |                   |                   |
+          |                   |      |registrations                   |                   |
+          |                   |<-----+            |                   |                   |
+          |                   |                   |                   |                   |
+          |                   |    11.send command to kill PID old agt|                   |
+          |                   |-------------------------------------->|                   |
+          |                   |                   |                   |                   |
+          |                   |  12.acknowledge   |                   |                   |
+          |                   |<--------------------------------------|                   |
All upgrade operations are initiated by an investigator (1). The upgrade is
+triggered by an action to the upgrade module with the following parameters:
"Operations": [
+    {
+        "Module": "upgrade",
+        "Parameters": {
+            "linux/amd64": {
+                "to_version": "16eb58b-201404021544",
+                "location": "http://localhost/mig/bin/linux/amd64/mig-agent",
+                "checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"
+            }
+        }
+    }
+],
  • Each OS family and architecture have their own parameters (ex: "linux/amd64",
+"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:
  • to_version is the version an agent should upgrade to
  • location points to a HTTPS address that contains the agent binary
  • checksum is a SHA256 hash of the agent binary to be verified after download
The parameters above are signed using a standard PGP action signature.
The upgrade action is forwarded to agents (2) like any other action. The action
+signature is verified by the agent (3), and the upgrade module is called. The
+module downloads the new binary (4), verifies the version and checksum (5) and
+installs itself on the system.
Assuming everything checks in, the old agent executes the binary of the new
+agent (6). At that point, two agents are running on the same machine, and the
+rest of the protocol is designed to shut down the old agent, and clean up.
After executing the new agent, the old agent returns a successful result to the
+scheduler, and includes its own PID in the results.
+The new agent starts by registering with the scheduler (7). This tells the
+scheduler that two agents are running on the same node, and one of them must
+terminate. The scheduler sends a kill action to both agents with the PID of the
+old agent (8). The kill action may be executed twice, but that doesn't matter.
+When the scheduler receives the kill results (9), it sends a new action to check
+for mig-agent processes (10). Only one should be found in the results (11),
+and if that is the case, the scheduler tells the agent to remove the binary of
+the old agent (12). When the agent returns (13), the upgrade protocol is done.
If the PID of the old agent lingers on the system, an error is logged for the
+investigator to decide what to do next. The scheduler does not attempt to clean
+up the situation.

\ No newline at end of file
diff --git a/doc/api.rst b/doc/api.rst
index c286341b..e3731bea 100644
--- a/doc/api.rst
+++ b/doc/api.rst
@@ -1,7 +1,6 @@
 =======
 MIG API
 =======
-:Author: Julien Vehent 
 
 .. sectnum::
 .. contents:: Table of Contents
@@ -14,186 +13,8 @@ The API follows the core principles of REST, and provides discoverable
 endpoints. API responses follows the **cljs** format defined in
 `Collection+JSON - Hypermedia Type `_.
 
-Authentication with X-PGPAUTHORIZATION version 1
-------------------------------------------------
-
-Authenticating against the MIG API requires sending a PGP signed token in the
-request header named `X-PGPAUTHORIZATION`. The key that signs the token must
-belong to an active investigator. Construction of the token works as follows:
-
-1. make a string named **str** composed of a version, a UTC timestamp in RFC3339 format
-   and a random nonce, each separated by semicolons. The current version is **1**
-   and may be upgraded in the future. The nonce value must be a positive integer.
-
-   **str=;;**
-
-   UTC is a hard requirement. The timestamp must end with the suffix **Z**
-   which indicates the UTC timezone. In bash, a correct timestamp can be
-   generated with the command `$ date -u +%Y-%m-%dT%H:%M:%SZ`.
-
-   An example string would look like: `1;2006-01-02T15:04:05Z;1825922807490630059`
-
-   The string must be terminated by a newline character, hexadecimal code `0x0a`.
-
-.. code:: bash
-
-	$ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
-	00000000  31 3b 32 30 30 36 2d 30  31 2d 30 32 54 31 35 3a  |1;2006-01-02T15:|
-	00000010  30 34 3a 30 35 5a 3b 31  38 32 35 39 32 32 38 30  |04:05Z;182592280|
-	00000020  37 34 39 30 36 33 30 30  35 39 0a                 |7490630059.|
-	0000002b
-
-2. PGP sign **str** with the private key of the investigator. Armor and detach
-   the signature into **armoredSig**::
-
-	$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
-
-	-----BEGIN PGP SIGNATURE-----
-	Version: GnuPG v1
-
-	iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
-	JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
-	kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
-	b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
-	PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
-	MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
-	=4nrR
-	-----END PGP SIGNATURE-----
-
-3. Create **sig** by taking **armoredSig** and removing the PGP headers, footers,
-   empty lines and newlines.
-
-	example: `iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT`
-
-4. Create **token** by concatenating **str**, a semicolon, and **sig**.
-   **token=;**
-   example: `1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...`
-
-5. Send **token** in the header named **X-PGPAUTHORIZATION** with the request::
-
-	$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...' localhost:12345/api/v1/
-
-6. The API verifies the version and validity period of the timestamp. By default, a
-   token will be rejected if its timestamp deviates from the server time by more
-   than 10 minutes. Administrators can configure this value. In effect, this
-   means a timestamp is valid for twice the duration of the window. By default,
-   that's 10 minutes before current server time, and 10 minutes after current
-   server time.
-
-7. If the timestamp is valid, the API next verifies the signature against the data
-   and authenticates the user. Failure to verify the signature returns an error
-   with the HTTP code 401 Unauthorized.
-
-8. The user is authorized, the API processes and answer the request.
-
-Security implications
-~~~~~~~~~~~~~~~~~~~~~
-
-1. A token can be used an unlimited number of times within its validity period.
-   There is no check to guarantee that a token is only used once. It is
-   assumed that the token is transmitted over a secure channel such as HTTPS to
-   prevent token theft by a malicious user.
-
-2. API clients and servers must use proper time synchronization for the timestamp
-   verification to work. A client or a server that has inaccurate time may not be
-   able to establish connections. We believe this requirement to be reasonable
-   considering the sensitivity of the API.
-
-Example 1: invalid timestamp
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The signature is valid but the timestamp is beyond the acceptable time window.
-
-.. code:: bash
-
-	$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...' http://localhost:12345/api/v1/
-
-	{
-		"collection": {
-			"error": {
-				"code": "6077873045059431424",
-				"message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
-			},
-			"href": "http://localhost:12345/api/v1/",
-			"template": {},
-			"version": "1.0"
-		}
-	}
-
-Example 2: invalid signature
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The signature is not valid, or is signed by a key that the API does not
-recognize.
-
-.. code:: bash
-
-	$ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...' http://localhost:12345/api/v1/
-
-	{
-		"collection": {
-			"error": {
-				"code": "6077875007260332032",
-				"message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
-			},
-			"href": "http://localhost:12345/api/v1/",
-			"template": {},
-			"version": "1.0"
-		}
-	}
-
-Generating a token in Bash
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code::
-
-	$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
-	sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
-	| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
-
-	X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....
-
-Generating a token in Python
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code:: python
-
-	#!/usr/bin/env python
-	import os
-	import gnupg
-	from time import gmtime, strftime
-	import random
-	import requests
-	import json
-
-	def makeToken(gpghome, keyid):
-		gpg = gnupg.GPG(gnupghome=gpghome)
-		version = "1"
-		timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
-		nonce = str(random.randint(10000, 18446744073709551616))
-		token = version + ";" + timestamp + ";" + nonce
-		sig = gpg.sign(token + "\n",
-			keyid=keyid,
-			detach=True, clearsign=True)
-		token += ";"
-		linectr=0
-		for line in iter(str(sig).splitlines()):
-			linectr+=1
-			if linectr < 4 or line.startswith('-') or not line:
-				continue
-			token += line
-		return token
-
-	if __name__ == '__main__':
-		token = makeToken("/home/ulfr/.gnupg",
-			"E60892BB9BD89A69F759A1A0A3D652173B763E8F")
-		r = requests.get("http://localhost:12345/api/v1/dashboard",
-			headers={'X-PGPAUTHORIZATION': token})
-		print token
-		print r.text
-
-API endpoints
--------------
+Endpoints
+---------
 
 The API root is at `/api/v1` by defualt. All the endpoints described below are
 reachable behind the root. If you change the location of the API root, update
@@ -1125,3 +946,183 @@ Geolocations are returned as CLJS items in this format:
 
 When using the parameter `&report=geolocations`, the `search` endpoint of the
 API will generate a list of geolocations from the results of the search.
+
+Authentication with X-PGPAUTHORIZATION version 1
+------------------------------------------------
+
+Authenticating against the MIG API requires sending a PGP signed token in the
+request header named `X-PGPAUTHORIZATION`. The key that signs the token must
+belong to an active investigator. Construction of the token works as follows:
+
+1. make a string named **str** composed of a version, a UTC timestamp in RFC3339 format
+   and a random nonce, each separated by semicolons. The current version is **1**
+   and may be upgraded in the future. The nonce value must be a positive integer.
+
+   **str=;;**
+
+   UTC is a hard requirement. The timestamp must end with the suffix **Z**
+   which indicates the UTC timezone. In bash, a correct timestamp can be
+   generated with the command `$ date -u +%Y-%m-%dT%H:%M:%SZ`.
+
+   An example string would look like: `1;2006-01-02T15:04:05Z;1825922807490630059`
+
+   The string must be terminated by a newline character, hexadecimal code `0x0a`.
+
+.. code:: bash
+
+	$ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
+	00000000  31 3b 32 30 30 36 2d 30  31 2d 30 32 54 31 35 3a  |1;2006-01-02T15:|
+	00000010  30 34 3a 30 35 5a 3b 31  38 32 35 39 32 32 38 30  |04:05Z;182592280|
+	00000020  37 34 39 30 36 33 30 30  35 39 0a                 |7490630059.|
+	0000002b
+
+2. PGP sign **str** with the private key of the investigator. Armor and detach
+   the signature into **armoredSig**::
+
+	$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
+
+	-----BEGIN PGP SIGNATURE-----
+	Version: GnuPG v1
+
+	iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
+	JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
+	kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
+	b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
+	PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
+	MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
+	=4nrR
+	-----END PGP SIGNATURE-----
+
+3. Create **sig** by taking **armoredSig** and removing the PGP headers, footers,
+   empty lines and newlines.
+
+	example: `iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT`
+
+4. Create **token** by concatenating **str**, a semicolon, and **sig**.
+   **token=;**
+   example: `1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...`
+
+5. Send **token** in the header named **X-PGPAUTHORIZATION** with the request::
+
+	$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...' localhost:12345/api/v1/
+
+6. The API verifies the version and validity period of the timestamp. By default, a
+   token will be rejected if its timestamp deviates from the server time by more
+   than 10 minutes. Administrators can configure this value. In effect, this
+   means a timestamp is valid for twice the duration of the window. By default,
+   that's 10 minutes before current server time, and 10 minutes after current
+   server time.
+
+7. If the timestamp is valid, the API next verifies the signature against the data
+   and authenticates the user. Failure to verify the signature returns an error
+   with the HTTP code 401 Unauthorized.
+
+8. The user is authorized, the API processes and answer the request.
+
+Security implications
+~~~~~~~~~~~~~~~~~~~~~
+
+1. A token can be used an unlimited number of times within its validity period.
+   There is no check to guarantee that a token is only used once. It is
+   assumed that the token is transmitted over a secure channel such as HTTPS to
+   prevent token theft by a malicious user.
+
+2. API clients and servers must use proper time synchronization for the timestamp
+   verification to work. A client or a server that has inaccurate time may not be
+   able to establish connections. We believe this requirement to be reasonable
+   considering the sensitivity of the API.
+
+Example 1: invalid timestamp
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The signature is valid but the timestamp is beyond the acceptable time window.
+
+.. code:: bash
+
+	$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...' http://localhost:12345/api/v1/
+
+	{
+		"collection": {
+			"error": {
+				"code": "6077873045059431424",
+				"message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
+			},
+			"href": "http://localhost:12345/api/v1/",
+			"template": {},
+			"version": "1.0"
+		}
+	}
+
+Example 2: invalid signature
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The signature is not valid, or is signed by a key that the API does not
+recognize.
+
+.. code:: bash
+
+	$ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...' http://localhost:12345/api/v1/
+
+	{
+		"collection": {
+			"error": {
+				"code": "6077875007260332032",
+				"message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
+			},
+			"href": "http://localhost:12345/api/v1/",
+			"template": {},
+			"version": "1.0"
+		}
+	}
+
+Generating a token in Bash
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code::
+
+	$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
+	sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
+	| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
+
+	X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....
+
+Generating a token in Python
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: python
+
+	#!/usr/bin/env python
+	import os
+	import gnupg
+	from time import gmtime, strftime
+	import random
+	import requests
+	import json
+
+	def makeToken(gpghome, keyid):
+		gpg = gnupg.GPG(gnupghome=gpghome)
+		version = "1"
+		timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
+		nonce = str(random.randint(10000, 18446744073709551616))
+		token = version + ";" + timestamp + ";" + nonce
+		sig = gpg.sign(token + "\n",
+			keyid=keyid,
+			detach=True, clearsign=True)
+		token += ";"
+		linectr=0
+		for line in iter(str(sig).splitlines()):
+			linectr+=1
+			if linectr < 4 or line.startswith('-') or not line:
+				continue
+			token += line
+		return token
+
+	if __name__ == '__main__':
+		token = makeToken("/home/ulfr/.gnupg",
+			"E60892BB9BD89A69F759A1A0A3D652173B763E8F")
+		r = requests.get("http://localhost:12345/api/v1/dashboard",
+			headers={'X-PGPAUTHORIZATION': token})
+		print token
+		print r.text
+
+
diff --git a/doc/api.rst.html b/doc/api.rst.html
index d227a348..0dc8bd5d 100644
--- a/doc/api.rst.html
+++ b/doc/api.rst.html
@@ -1,6 +1,6 @@
 
MIG API
AuthorJulien Vehent <jvehent@mozilla.com>
Interactions between an investigator (a human being) and the MIG platform are
+
MIG API
Interactions between an investigator (a human being) and the MIG platform are
 performed through a REST API. The API exposes functions to create actions,
 retrieve results, and generally monitor the activity of the agents.
The API follows the core principles of REST, and provides discoverable
 endpoints. API responses follows the cljs format defined in
-Collection+JSON - Hypermedia Type.
1   Authentication with X-PGPAUTHORIZATION version 1
Authenticating against the MIG API requires sending a PGP signed token in the
-request header named X-PGPAUTHORIZATION. The key that signs the token must
-belong to an active investigator. Construction of the token works as follows:
  1. make a string named str composed of a version, a UTC timestamp in RFC3339 format
-and a random nonce, each separated by semicolons. The current version is 1
-and may be upgraded in the future. The nonce value must be a positive integer.
    str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE>
    UTC is a hard requirement. The timestamp must end with the suffix Z
-which indicates the UTC timezone. In bash, a correct timestamp can be
-generated with the command $ date -u +%Y-%m-%dT%H:%M:%SZ.
    An example string would look like: 1;2006-01-02T15:04:05Z;1825922807490630059
    The string must be terminated by a newline character, hexadecimal code 0x0a.
$ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
-00000000  31 3b 32 30 30 36 2d 30  31 2d 30 32 54 31 35 3a  |1;2006-01-02T15:|
-00000010  30 34 3a 30 35 5a 3b 31  38 32 35 39 32 32 38 30  |04:05Z;182592280|
-00000020  37 34 39 30 36 33 30 30  35 39 0a                 |7490630059.|
-0000002b
  1. PGP sign str with the private key of the investigator. Armor and detach
-the signature into armoredSig:
    $ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
-
------BEGIN PGP SIGNATURE-----
-Version: GnuPG v1
-
-iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
-JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
-kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
-b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
-PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
-MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
-=4nrR
------END PGP SIGNATURE-----
  2. Create sig by taking armoredSig and removing the PGP headers, footers,
-empty lines and newlines.
    example: iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT
  3. Create token by concatenating str, a semicolon, and sig.
-token=<str>;<sig>
-example: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated>
  4. Send token in the header named X-PGPAUTHORIZATION with the request:
    $ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/
  5. The API verifies the version and validity period of the timestamp. By default, a
-token will be rejected if its timestamp deviates from the server time by more
-than 10 minutes. Administrators can configure this value. In effect, this
-means a timestamp is valid for twice the duration of the window. By default,
-that's 10 minutes before current server time, and 10 minutes after current
-server time.
  6. If the timestamp is valid, the API next verifies the signature against the data
-and authenticates the user. Failure to verify the signature returns an error
-with the HTTP code 401 Unauthorized.
  7. The user is authorized, the API processes and answer the request.
1.1   Security implications
  1. A token can be used an unlimited number of times within its validity period.
-There is no check to guarantee that a token is only used once. It is
-assumed that the token is transmitted over a secure channel such as HTTPS to
-prevent token theft by a malicious user.
  2. API clients and servers must use proper time synchronization for the timestamp
-verification to work. A client or a server that has inaccurate time may not be
-able to establish connections. We believe this requirement to be reasonable
-considering the sensitivity of the API.
1.2   Example 1: invalid timestamp
The signature is valid but the timestamp is beyond the acceptable time window.
$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>' http://localhost:12345/api/v1/
-
-{
-        "collection": {
-                "error": {
-                        "code": "6077873045059431424",
-                        "message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
-                },
-                "href": "http://localhost:12345/api/v1/",
-                "template": {},
-                "version": "1.0"
-        }
-}
1.3   Example 2: invalid signature
The signature is not valid, or is signed by a key that the API does not
-recognize.
$ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>' http://localhost:12345/api/v1/
-
-{
-        "collection": {
-                "error": {
-                        "code": "6077875007260332032",
-                        "message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
-                },
-                "href": "http://localhost:12345/api/v1/",
-                "template": {},
-                "version": "1.0"
-        }
-}
1.4   Generating a token in Bash
$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
-sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
-| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
-
-X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....
1.5   Generating a token in Python
#!/usr/bin/env python
-import os
-import gnupg
-from time import gmtime, strftime
-import random
-import requests
-import json
-
-def makeToken(gpghome, keyid):
-        gpg = gnupg.GPG(gnupghome=gpghome)
-        version = "1"
-        timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
-        nonce = str(random.randint(10000, 18446744073709551616))
-        token = version + ";" + timestamp + ";" + nonce
-        sig = gpg.sign(token + "\n",
-                keyid=keyid,
-                detach=True, clearsign=True)
-        token += ";"
-        linectr=0
-        for line in iter(str(sig).splitlines()):
-                linectr+=1
-                if linectr < 4 or line.startswith('-') or not line:
-                        continue
-                token += line
-        return token
-
-if __name__ == '__main__':
-        token = makeToken("/home/ulfr/.gnupg",
-                "E60892BB9BD89A69F759A1A0A3D652173B763E8F")
-        r = requests.get("http://localhost:12345/api/v1/dashboard",
-                headers={'X-PGPAUTHORIZATION': token})
-        print token
-        print r.text
2   API endpoints
The API root is at /api/v1 by defualt. All the endpoints described below are
+Collection+JSON - Hypermedia Type.
1   Endpoints
The API root is at /api/v1 by defualt. All the endpoints described below are
 reachable behind the root. If you change the location of the API root, update
-the query paths accordingly.
2.1   GET /api/v1/heartbeat
  • Description: basic endpoint that returns a HTTP 200
  • Parameters: none
  • Authentication: none
  • Response Code: 200 OK
  • Reponse: Collection+JSON
{
+the query paths accordingly.
1.1   GET /api/v1/heartbeat
  • Description: basic endpoint that returns a HTTP 200
  • Parameters: none
  • Authentication: none
  • Response Code: 200 OK
  • Reponse: Collection+JSON
{
         "collection": {
                 "error": {},
                 "href": "https://api.mig.mozilla.org/api/v1/heartbeat",
@@ -351,9 +253,9 @@ the query paths accordingly.
<
                 "template": {},
                 "version": "1.0"
         }
-}
2.2   GET /api/v1/ip
  • Description: basic endpoint that returns the public IP of the caller. If the
+}
1.2   GET /api/v1/ip
  • Description: basic endpoint that returns the public IP of the caller. If the
 API is behind a load balancer, it returns the value of X-Forwarded-For.
  • Parameters: none
  • Authentication: none
  • Response Code: 200 OK
  • Response: Text
$ curl https://api.mig.mozilla.org/api/v1/ip
-108.36.248.44
2.3   GET /api/v1/dashboard
  • Description: returns a status dashboard with counters of active and idle
+108.36.248.44

1.3   GET /api/v1/dashboard

{
   "collection": {
         "error": {},
@@ -526,7 +428,7 @@ agents, and a list of the last 10 actions ran.
  • Parameters: none"template": {},
         "version": "1.0"
   }
-}

    • 2.4   GET /api/v1/action

    {
+}

    1.4   GET /api/v1/action

    {
   "collection": {
         "error": {},
         "href": "https://api.mig.example.net/api/v1/action?actionid=6115472790658567168",
@@ -639,7 +541,7 @@ agents, and a list of the last 10 actions ran.
  • Parameters: none"template": {},
         "version": "1.0"
   }
-}

    • 2.5   POST /api/v1/action/create/

    2.6   GET /api/v1/agent

    {
+}

    1.5   POST /api/v1/action/create/

    1.6   GET /api/v1/agent

    {
   "collection": {
         "error": {},
         "href": "https://api.mig.example.net/api/v1/agent?agentid=1423779015943326976",
@@ -681,7 +583,7 @@ agents, and a list of the last 10 actions ran.
  • Parameters: none"template": {},
         "version": "1.0"
   }
-}

    • 2.7   GET /api/v1/command

    {
+}

    1.7   GET /api/v1/command

    {
   "collection": {
         "error": {},
         "href": "https://api.mig.example.net/api/v1/command?commandid=1424700180901330688",
@@ -896,7 +798,7 @@ agents, and a list of the last 10 actions ran.
  • Parameters: nonetemplate": {},
         "version": "1.0"
   }
-}

    • 2.8   GET /api/v1/investigator

    1.8   GET /api/v1/investigator

    {
   "collection": {
         "error": {},
@@ -929,8 +831,8 @@ investigator's action history.
  • Authentication: X-PGPAUTHORIZATION
         "template": {},
         "version": "1.0"
   }
-}

    • 2.9   POST /api/v1/investigator/create/

    $ gpg --export -a --export-options export-minimal bob_kelso@example.net > /tmp/bobpubkey
-$ curl -iv -F "name=Bob Kelso" -F publickey=@/tmp/pubkey https://api.mig.example.net/api/v1/investigator/create/

    2.10   POST /api/v1/investigator/update/

    $ curl -iv -X POST -d id=1234 -d status=disabled https://api.mig.example.net/api/v1/investigator/update/

    1.9   POST /api/v1/investigator/create/

    $ gpg --export -a --export-options export-minimal bob_kelso@example.net > /tmp/bobpubkey
+$ curl -iv -F "name=Bob Kelso" -F publickey=@/tmp/pubkey https://api.mig.example.net/api/v1/investigator/create/

    1.10   POST /api/v1/investigator/update/

    $ curl -iv -X POST -d id=1234 -d status=disabled https://api.mig.example.net/api/v1/investigator/update/

    3   Data transformation

    The API implements several data transformation functions between the base -format of action and command, and reporting formats.

    3.1   Compliance Items

    The compliance item format is used to measure the compliance of a target with +&after=2014-05-01T00:00:00-00:00&before=2014-05-30T00:00:00-00:00

    Find the last 10 commands signed by an investigator identified by name.

    /api/v1/search?investigatorname=%25bob%25smith%25&limit=10&type=command

    2   Data transformation

    The API implements several data transformation functions between the base +format of action and command, and reporting formats.

    2.1   Compliance Items

    The compliance item format is used to measure the compliance of a target with particular requirement. A single compliance item represent the compliance of one target (host) with one check (test + value).

    In MIG, an action can contain compliance checks. An action creates one command per agent. Upon completion, the agent stores the results in the @@ -980,7 +882,7 @@ commands could, in turn, generate thousands of compliance items.

    The forma "name": "attemptstoaltercrontab_user_config" } }

    When using the parameter &report=complianceitems, the search endpoint of the API -will generate a list of compliance items from the results of the search.

    3.2   Geolocations

    The geolocations format transforms command results into an array of geolocated +will generate a list of compliance items from the results of the search.

    2.2   Geolocations

    The geolocations format transforms command results into an array of geolocated endpoints for consumption by a map, like Google Maps. The format discards results details, and only stores the value of FoundAnything.

    This feature requires using MaxMind's GeoIP2-City database. The database must be configured in the API as follow:

    [maxmind]
@@ -994,4 +896,104 @@ must be configured in the API as follow:
    [maxmind]
         "latitude": 39.4284,
         "longitude": -74.4957
 }
    When using the parameter &report=geolocations, the search endpoint of the
-API will generate a list of geolocations from the results of the search.
    \ No newline at end of file +API will generate a list of geolocations from the results of the search.

    3   Authentication with X-PGPAUTHORIZATION version 1

    Authenticating against the MIG API requires sending a PGP signed token in the +request header named X-PGPAUTHORIZATION. The key that signs the token must +belong to an active investigator. Construction of the token works as follows:

    1. make a string named str composed of a version, a UTC timestamp in RFC3339 format +and a random nonce, each separated by semicolons. The current version is 1 +and may be upgraded in the future. The nonce value must be a positive integer.

      str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE>

      UTC is a hard requirement. The timestamp must end with the suffix Z +which indicates the UTC timezone. In bash, a correct timestamp can be +generated with the command $ date -u +%Y-%m-%dT%H:%M:%SZ.

      An example string would look like: 1;2006-01-02T15:04:05Z;1825922807490630059

      The string must be terminated by a newline character, hexadecimal code 0x0a.

    $ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
+00000000  31 3b 32 30 30 36 2d 30  31 2d 30 32 54 31 35 3a  |1;2006-01-02T15:|
+00000010  30 34 3a 30 35 5a 3b 31  38 32 35 39 32 32 38 30  |04:05Z;182592280|
+00000020  37 34 39 30 36 33 30 30  35 39 0a                 |7490630059.|
+0000002b

    1. PGP sign str with the private key of the investigator. Armor and detach +the signature into armoredSig:

      $ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1
+
+iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
+JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
+kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
+b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
+PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
+MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
+=4nrR
+-----END PGP SIGNATURE-----

    2. Create sig by taking armoredSig and removing the PGP headers, footers, +empty lines and newlines.

      example: iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT

    3. Create token by concatenating str, a semicolon, and sig. +token=<str>;<sig> +example: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated>

    4. Send token in the header named X-PGPAUTHORIZATION with the request:

      $ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/

    5. The API verifies the version and validity period of the timestamp. By default, a +token will be rejected if its timestamp deviates from the server time by more +than 10 minutes. Administrators can configure this value. In effect, this +means a timestamp is valid for twice the duration of the window. By default, +that's 10 minutes before current server time, and 10 minutes after current +server time.

    6. If the timestamp is valid, the API next verifies the signature against the data +and authenticates the user. Failure to verify the signature returns an error +with the HTTP code 401 Unauthorized.

    7. The user is authorized, the API processes and answer the request.

    3.1   Security implications

    1. A token can be used an unlimited number of times within its validity period. +There is no check to guarantee that a token is only used once. It is +assumed that the token is transmitted over a secure channel such as HTTPS to +prevent token theft by a malicious user.

    2. API clients and servers must use proper time synchronization for the timestamp +verification to work. A client or a server that has inaccurate time may not be +able to establish connections. We believe this requirement to be reasonable +considering the sensitivity of the API.

    3.2   Example 1: invalid timestamp

    The signature is valid but the timestamp is beyond the acceptable time window.

    $ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>' http://localhost:12345/api/v1/
+
+{
+        "collection": {
+                "error": {
+                        "code": "6077873045059431424",
+                        "message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
+                },
+                "href": "http://localhost:12345/api/v1/",
+                "template": {},
+                "version": "1.0"
+        }
+}

    3.3   Example 2: invalid signature

    The signature is not valid, or is signed by a key that the API does not +recognize.

    $ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>' http://localhost:12345/api/v1/
+
+{
+        "collection": {
+                "error": {
+                        "code": "6077875007260332032",
+                        "message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
+                },
+                "href": "http://localhost:12345/api/v1/",
+                "template": {},
+                "version": "1.0"
+        }
+}

    3.4   Generating a token in Bash

    $ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
+sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
+| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
+
+X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....

    3.5   Generating a token in Python

    #!/usr/bin/env python
+import os
+import gnupg
+from time import gmtime, strftime
+import random
+import requests
+import json
+
+def makeToken(gpghome, keyid):
+        gpg = gnupg.GPG(gnupghome=gpghome)
+        version = "1"
+        timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
+        nonce = str(random.randint(10000, 18446744073709551616))
+        token = version + ";" + timestamp + ";" + nonce
+        sig = gpg.sign(token + "\n",
+                keyid=keyid,
+                detach=True, clearsign=True)
+        token += ";"
+        linectr=0
+        for line in iter(str(sig).splitlines()):
+                linectr+=1
+                if linectr < 4 or line.startswith('-') or not line:
+                        continue
+                token += line
+        return token
+
+if __name__ == '__main__':
+        token = makeToken("/home/ulfr/.gnupg",
+                "E60892BB9BD89A69F759A1A0A3D652173B763E8F")
+        r = requests.get("http://localhost:12345/api/v1/dashboard",
+                headers={'X-PGPAUTHORIZATION': token})
+        print token
+        print r.text
    \ No newline at end of file diff --git a/doc/cheatsheet.rst.html b/doc/cheatsheet.rst.html index 2ba54696..e49c75db 100644 --- a/doc/cheatsheet.rst.html +++ b/doc/cheatsheet.rst.html @@ -1,6 +1,6 @@

    MIG command line

    The MIG command line is a terminal-based client interface that provides fast +interaction with the MIG API to launch actions and receive results. Its goal is +to be simple and fast to use for basic investigations. Unlike the mig-console, +it does not provide access to previous actions, search or investigators +management.

    To install the MIG command line, use go get mig.ninja/mig/client/mig. This +command will place a binary called mig under $GOPATH/bin/mig. On first run, it +will invite you to create a configuration:

    $ mig file -path /etc -name "^passwd$" -content "jvehent"
+no configuration file found at /home/jvehent/.migrc
+would you like to generate a new configuration file at /home/jvehent/.migrc? Y/n> Y
+found key 'Julien Vehent (ulfr) <jvehent@mozilla.com>' with fingerprint 'E60892BB9BD89A69F759A1A0A3D652173B763E8F'.
+use this key? Y/n> Y
+using key E60892BB9BD89A69F759A1A0A3D652173B763E8F
+what is the location of the API? (ex: https://mig.example.net/api/v1/) > https://api.mig.example.net/api/v1/
+MIG client configuration generated at /home/jvehent/.migrc
+
+1226 agents will be targeted. ctrl+c to cancel. launching in 5 4....

    Subsequent runs of mig will read the configuration from $HOME/.migrc directly.

    For more examples on how to use the mig command line, see the cheatsheet.

    \ No newline at end of file diff --git a/doc/concepts.rst b/doc/concepts.rst index 68ec7914..a611f3de 100644 --- a/doc/concepts.rst +++ b/doc/concepts.rst @@ -1,7 +1,6 @@ =================================================== Mozilla InvestiGator Concepts & Internal Components =================================================== -:Author: Julien Vehent .. sectnum:: .. contents:: Table of Contents @@ -76,11 +75,8 @@ Below is a high-level view of the architecture: .. image:: .files/MIG-Arch-Diagram.png -Actions and Commands --------------------- - Actions -~~~~~~~ +------- Actions are JSON files created by investigator to perform tasks on agents. @@ -168,8 +164,8 @@ Upon generation, additional fields are appended to the action: * **validfrom** and **expireafter**: two dates that constrains the validity of the action to a UTC time window. -Action/Commands workflow ------------------------- +Investigation workflow +----------------------- The diagram below represents the full workflow from the launch of an action by an investigation, to the retrieval of results from the database. The steps are explained in the legend of the diagram, and map to various components of MIG. @@ -184,159 +180,6 @@ View `full size diagram`_. .. image:: .files/action_command_flow.svg -Command execution flow in Agent and Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Upon processing of an action, the scheduler will retrieve a list of agents to -send the action to. One action is then derived into multiple commands and sent -to agents. - -An agent receives a command from the scheduler on its personal AMQP queue (1). -It parses the command (2) and extracts all of the operations to perform. -Operations are passed to modules and executed in parallel (3). Rather than -maintaining a state of the running command, the agent create a goroutine and a -channel tasked with receiving the results from the modules. Each modules -published its results inside that channel (4). The result parsing goroutine -receives them, and when it has received all of them, populates the `results` (5) -array of the command with the results from each module, and send the command -back to the scheduler(6). - -When the agent is done running the command, both the channel and the goroutine -are destroyed. - - :: - - +-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ] - |command|+---->(listener) - +-------+ |(2) - ^ V - |(1) (parser) - | + [ m o d u l e s ] - +---------+ | (3)|----------> op1 +----------------+ - |SCHEDULER|+---+ |------------> op2 +--------------| - | |<---+ |--------------> op3 +------------| - +---------+ | +----------------> op4 +----------+ - | V(4) - |(6) (receiver) - | | - | V(5) - + (publisher) - +-------+ / - |results|<-----------------------------------------' - +-------+ - -The command received by the agent is composed of a copy of the action described -previously, but signed with the private key of a trusted investigator. It also -contains additional parameters that are specific to the targetted agent, such as -command processing timestamps, name of the agent queue on the message broker, -action and command unique IDs and status and results of the command. Below is an -command derived from the root password checking action, and ran on the host named -'host1.example.net'. - -.. code:: json - - { - "id": 1.427392971126604e+18, - "action": { ... SIGNED COPY OF THE ACTION ... }, - "agent": { - "id": 1.4271760437936648e+18, - "name": "host1.example.net", - "queueloc": "linux.host1.example.net.981alsd19aos1984", - "mode": "daemon", - "version": "20150324+0d0f88c.prod" - }, - "status": "success", - "results": [ - { - "foundanything": true, - "success": true, - "elements": { - "root_passwd_hashed_or_disabled": [ - { - "file": "/etc/shadow", - "fileinfo": { - "lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC", - "mode": "----------", - "size": 1684 - }, - "search": { - "contents": [ - "root:(\\*|!|\\$(1|2a|5|6)\\$).+" - ], - "options": { - "matchall": false, - "matchlimit": 0, - "maxdepth": 0 - }, - "paths": [ - "/etc" - ] - } - } - ] - }, - "statistics": { - "exectime": "2.017849ms", - "filescount": 1, - "openfailed": 0, - "totalhits": 1 - }, - "errors": null - } - ], - "starttime": "2015-03-26T18:02:51.126605Z", - "finishtime": "2015-03-26T18:03:00.671232Z" - } - -The results of the command show that the file '/etc/shadow' has matched, and -thus "FoundAnything" returned "True". - -The invocation of the file module has completed successfully, which is -represented by **results->0->success=true**. In our example, there is only one -operation in the **action->operations** array, so only one result is present. -When multiple operations are performed, each has its results listed in a -corresponding entry of the results array (operations[0] is in results[0], -operations[1] in results[1], etc...). - -Finally, the agent has performed all operations in the operations array -successfully, and returned **status=success**. Had a failure happened on the -agent, the returned status would be one of "failed", "timeout" or "cancelled". - -Command expiration & timeouts -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To prevent abuse of resources, agents will kill long-running modules after a -given period of time. That timeout is hardcoded in the agent configuration -at compile time and defaults to 5 minutes. - -.. code:: go - - // timeout after which a module run is killed - var MODULETIMEOUT time.Duration = 300 * time.Second - -That timeout represents the **maximum** execution time of a single operation. If -an action contains 3 operations, each operation gets its own timeout. But because -operations run in parallel in the agent, the maximum runtime of an action should -be very close to the value of MODULETIMEOUT. - -In a typical deployment, it is safe to increase MODULETIMEOUT to allow for -longer operations. A value of 20 minutes is usual. Make sure to fine tune this -to your environment, and get the approval of your ops team because mig-agent -may end up consuming resources (but never more than 50% of the cpu available on -a system). - -Oftentimes, an investigator will want a timeout that is much shorter than the value -of MODULETIMEOUT. In the MIG command line, the flag `-e` controls the -expiration. It defaults to 5 minutes but can be set to 30 seconds for simple -investigations. When that happens, the agent will calculate an appropriate expiration -for the operations being run. If the expiration set on the action is set to 30 seconds, -the agent will kill operations that run for more than 30 seconds. - -If the expiration is larger than the value of MODULETIMEOUT (for example, 2 -hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to -allow agents that only check in periodically to pick up actions long after they -are launched. - Access Control Lists -------------------- @@ -410,172 +253,6 @@ control to authorize access to specific functions of modules. For example, an investigator could be authorized to call the `regex` function of filechecker module, but only in `/etc`. This functionality is not implemented yet. -Agent initialization process ----------------------------- -The agent tries to be as autonomous as possible. One of the goal is to ship -agents without requiring external provisioning tools, such as Chef or Puppet. -Therefore, the agent attempts to install itself as a service, and also supports -a builtin upgrade protocol (described in the next section). - -As a portable binary, the agent needs to detect the type of operating system -and init method that is used by an endpoint. Depending on the endpoint, -different initialization methods are used. The diagram below explains the -decision process followed by the agent. - -.. image:: .files/mig-agent-initialization-process.png - -Go does not provide support for running programs in the backgroud. On endpoints -that run upstart, systemd (linux) or launchd (darwin), this is not an issue -because the init daemon takes care of running the agent in the background, -rerouting its file descriptors and restarting on crash. On Windows and System-V, -however, the agent daemonizes by forking itself into `foreground` mode, and -re-forking itself on error (such as loss of connectivity to the relay). -On Windows and System-V, if the agent is killed, it will not be restarted -automatically. - -Registration process -~~~~~~~~~~~~~~~~~~~~ - -The initialization process goes through several environment detection steps -which are used to select the proper init method. Once started, the agent will -send a heartbeat to the public relay, and also store that heartbeat in its -`run` directory. The location of the `run` directory is platform specific. - -* windows: C:\Windows\ -* darwin: /Library/Preferences/mig/ -* linux: /var/run/mig/ - -Below is a sample heartbeat message from a linux agent stored in -`/var/run/mig/mig-agent.ok`. - -.. code:: json - - { - "destructiontime": "0001-01-01T00:00:00Z", - "environment": { - "arch": "amd64", - "ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)", - "init": "upstart" - }, - "heartbeatts": "2014-07-31T14:00:20.00442837-07:00", - "name": "someserver.example.net", - "os": "linux", - "pid": 26256, - "queueloc": "linux.someserver.example.net.5hsa811oda", - "starttime": "2014-07-30T21:34:48.525449401-07:00", - "version": "201407310027+bcbdd94.prod" - } - -Check-In mode -~~~~~~~~~~~~~ -In infrastructure where running the agent as a permanent process is not -acceptable, it is possible to run the agent as a cron job. By starting the -agent with the flag **-m agent-checkin**, the agent will connect to the -configured relay, retrieve and run outstanding commands, and exit after 10 -seconds of inactivity. - -Agent upgrade process ---------------------- -MIG supports upgrading agents in the wild. The upgrade protocol is designed with -security in mind. The flow diagram below presents a high-level view: - - :: - - Investigator Scheduler Agent NewAgent FileServer - +-----------+ +-------+ +---+ +------+ +--------+ - | | | | | - | 1.initiate | | | | - |------------------>| | | | - | | 2.send command | | | - | |------------------>| 3.verify | | - | | |--------+ | | - | | | | | | - | | | | | | - | | |<-------+ | | - | | | | | - | | | 4.download | | - | | |-------------------------------------->| - | | | | | - | | | 5.checksum | | - | | |--------+ | | - | | | | | | - | | | | | | - | | |<-------+ | | - | | | | | - | | | 6.exec | | - | | |------------------>| | - | | 7.return own PID | | | - | |<------------------| | | - | | | | | - | |------+ 8.mark | | | - | | | agent as | | | - | | | upgraded | | | - | |<-----+ | | | - | | | | | - | | 9.register | | | - | |<--------------------------------------| | - | | | | | - | |------+10.find dup | | | - | | |agents in | | | - | | |registrations | | - | |<-----+ | | | - | | | | | - | | 11.send command to kill PID old agt| | - | |-------------------------------------->| | - | | | | | - | | 12.acknowledge | | | - | |<--------------------------------------| | - -All upgrade operations are initiated by an investigator (1). The upgrade is -triggered by an action to the upgrade module with the following parameters: - -.. code:: json - - "Operations": [ - { - "Module": "upgrade", - "Parameters": { - "linux/amd64": { - "to_version": "16eb58b-201404021544", - "location": "http://localhost/mig/bin/linux/amd64/mig-agent", - "checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03" - } - } - } - ], - -* Each OS family and architecture have their own parameters (ex: "linux/amd64", - "darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have: -* to_version is the version an agent should upgrade to -* location points to a HTTPS address that contains the agent binary -* checksum is a SHA256 hash of the agent binary to be verified after download - -The parameters above are signed using a standard PGP action signature. - -The upgrade action is forwarded to agents (2) like any other action. The action -signature is verified by the agent (3), and the upgrade module is called. The -module downloads the new binary (4), verifies the version and checksum (5) and -installs itself on the system. - -Assuming everything checks in, the old agent executes the binary of the new -agent (6). At that point, two agents are running on the same machine, and the -rest of the protocol is designed to shut down the old agent, and clean up. - -After executing the new agent, the old agent returns a successful result to the -scheduler, and includes its own PID in the results. -The new agent starts by registering with the scheduler (7). This tells the -scheduler that two agents are running on the same node, and one of them must -terminate. The scheduler sends a kill action to both agents with the PID of the -old agent (8). The kill action may be executed twice, but that doesn't matter. -When the scheduler receives the kill results (9), it sends a new action to check -for `mig-agent` processes (10). Only one should be found in the results (11), -and if that is the case, the scheduler tells the agent to remove the binary of -the old agent (12). When the agent returns (13), the upgrade protocol is done. - -If the PID of the old agent lingers on the system, an error is logged for the -investigator to decide what to do next. The scheduler does not attempt to clean -up the situation. - Threat Model ------------ diff --git a/doc/concepts.rst.html b/doc/concepts.rst.html index a1a264bb..8b9f28a3 100644 --- a/doc/concepts.rst.html +++ b/doc/concepts.rst.html @@ -1,6 +1,6 @@

    Mozilla InvestiGator Concepts & Internal Components

    AuthorJulien Vehent <jvehent@mozilla.com>

    MIG is a platform to perform investigative surgery on remote endpoints. +

    Mozilla InvestiGator Concepts & Internal Components

    MIG is a platform to perform investigative surgery on remote endpoints. It enables investigators to obtain information from large numbers of systems in parallel, thus accelerating investigation of incidents.

    Besides scalability, MIG is designed to provide strong security primitives:

    • Access control is ensured by requiring GPG signatures on all actions. Sensitive actions can also request signatures from multiple investigators. An attacker @@ -261,7 +263,7 @@ The agents also use the relays to send heartbeat at regular intervals, such that the scheduler always knows how many agents are alive at a given time.

      The end-to-end workflow is:

      {investigator} -https-> {API}        {Scheduler} -amqps-> {Relays} -amqps-> {Agents}
                             \           /
                           sql\         /sql
-                             {DATABASE}

      Below is a high-level view of the architecture:

      .files/MIG-Arch-Diagram.png

      1   Actions and Commands

      1.1   Actions

      Actions are JSON files created by investigator to perform tasks on agents.

      For example, an investigator who wants to verify than root passwords are hashed + {DATABASE}

      Below is a high-level view of the architecture:

      .files/MIG-Arch-Diagram.png

      1   Actions

      Actions are JSON files created by investigator to perform tasks on agents.

      For example, an investigator who wants to verify than root passwords are hashed and salted on linux systems, would use the following action:

      {
         "name": "verify root password storage method",
         "target": "agents.queueloc like 'linux.%'",
@@ -313,121 +315,10 @@ signed with the investigator's private GPG key. The signature is part of the
 action, and used by agents to verify that an action comes from a trusted
 investigator. PGPSignatures is an array that contains one or more signature
 from authorized investigators.

    • validfrom and expireafter: two dates that constrains the validity of the -action to a UTC time window.

    2   Action/Commands workflow

    The diagram below represents the full workflow from the launch of an action by +action to a UTC time window.

    2   Investigation workflow

    The diagram below represents the full workflow from the launch of an action by an investigation, to the retrieval of results from the database. The steps are explained in the legend of the diagram, and map to various components of MIG.

    Actions are submitted to the API by trusted investigators. PGPSignatures are -verified by the API and each agent prior to running any command.

    View full size diagram.

    .files/action_command_flow.svg

    2.1   Command execution flow in Agent and Modules

    Upon processing of an action, the scheduler will retrieve a list of agents to -send the action to. One action is then derived into multiple commands and sent -to agents.

    An agent receives a command from the scheduler on its personal AMQP queue (1). -It parses the command (2) and extracts all of the operations to perform. -Operations are passed to modules and executed in parallel (3). Rather than -maintaining a state of the running command, the agent create a goroutine and a -channel tasked with receiving the results from the modules. Each modules -published its results inside that channel (4). The result parsing goroutine -receives them, and when it has received all of them, populates the results (5) -array of the command with the results from each module, and send the command -back to the scheduler(6).

    When the agent is done running the command, both the channel and the goroutine -are destroyed.

                 +-------+   [ - - - - - - A G E N T - - - - - - - - - - - - ]
-             |command|+---->(listener)
-             +-------+          |(2)
-               ^                V
-               |(1)         (parser)
-               |               +       [ m o d u l e s ]
-+---------+    |            (3)|----------> op1 +----------------+
-|SCHEDULER|+---+               |------------> op2 +--------------|
-|         |<---+               |--------------> op3 +------------|
-+---------+    |               +----------------> op4 +----------+
-               |                                                 V(4)
-               |(6)                                         (receiver)
-               |                                                 |
-               |                                                 V(5)
-               +                                             (publisher)
-             +-------+                                           /
-             |results|<-----------------------------------------'
-             +-------+

    The command received by the agent is composed of a copy of the action described -previously, but signed with the private key of a trusted investigator. It also -contains additional parameters that are specific to the targetted agent, such as -command processing timestamps, name of the agent queue on the message broker, -action and command unique IDs and status and results of the command. Below is an -command derived from the root password checking action, and ran on the host named -'host1.example.net'.

    {
-  "id": 1.427392971126604e+18,
-  "action": { ... SIGNED COPY OF THE ACTION ... },
-  "agent": {
-        "id": 1.4271760437936648e+18,
-        "name": "host1.example.net",
-        "queueloc": "linux.host1.example.net.981alsd19aos1984",
-        "mode": "daemon",
-        "version": "20150324+0d0f88c.prod"
-  },
-  "status": "success",
-  "results": [
-        {
-          "foundanything": true,
-          "success": true,
-          "elements": {
-                "root_passwd_hashed_or_disabled": [
-                  {
-                        "file": "/etc/shadow",
-                        "fileinfo": {
-                          "lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC",
-                          "mode": "----------",
-                          "size": 1684
-                        },
-                        "search": {
-                          "contents": [
-                                "root:(\\*|!|\\$(1|2a|5|6)\\$).+"
-                          ],
-                          "options": {
-                                "matchall": false,
-                                "matchlimit": 0,
-                                "maxdepth": 0
-                          },
-                          "paths": [
-                                "/etc"
-                          ]
-                        }
-                  }
-                ]
-          },
-          "statistics": {
-                "exectime": "2.017849ms",
-                "filescount": 1,
-                "openfailed": 0,
-                "totalhits": 1
-          },
-          "errors": null
-        }
-  ],
-  "starttime": "2015-03-26T18:02:51.126605Z",
-  "finishtime": "2015-03-26T18:03:00.671232Z"
-}

    The results of the command show that the file '/etc/shadow' has matched, and -thus "FoundAnything" returned "True".

    The invocation of the file module has completed successfully, which is -represented by results->0->success=true. In our example, there is only one -operation in the action->operations array, so only one result is present. -When multiple operations are performed, each has its results listed in a -corresponding entry of the results array (operations[0] is in results[0], -operations[1] in results[1], etc...).

    Finally, the agent has performed all operations in the operations array -successfully, and returned status=success. Had a failure happened on the -agent, the returned status would be one of "failed", "timeout" or "cancelled".

    2.2   Command expiration & timeouts

    To prevent abuse of resources, agents will kill long-running modules after a -given period of time. That timeout is hardcoded in the agent configuration -at compile time and defaults to 5 minutes.

    // timeout after which a module run is killed
-var MODULETIMEOUT time.Duration = 300 * time.Second

    That timeout represents the maximum execution time of a single operation. If -an action contains 3 operations, each operation gets its own timeout. But because -operations run in parallel in the agent, the maximum runtime of an action should -be very close to the value of MODULETIMEOUT.

    In a typical deployment, it is safe to increase MODULETIMEOUT to allow for -longer operations. A value of 20 minutes is usual. Make sure to fine tune this -to your environment, and get the approval of your ops team because mig-agent -may end up consuming resources (but never more than 50% of the cpu available on -a system).

    Oftentimes, an investigator will want a timeout that is much shorter than the value -of MODULETIMEOUT. In the MIG command line, the flag -e controls the -expiration. It defaults to 5 minutes but can be set to 30 seconds for simple -investigations. When that happens, the agent will calculate an appropriate expiration -for the operations being run. If the expiration set on the action is set to 30 seconds, -the agent will kill operations that run for more than 30 seconds.

    If the expiration is larger than the value of MODULETIMEOUT (for example, 2 -hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to -allow agents that only check in periodically to pick up actions long after they -are launched.

    3   Access Control Lists

    Not all keys can perform all actions. The scheduler, for example, sometimes need +verified by the API and each agent prior to running any command.

    View full size diagram.

    .files/action_command_flow.svg

    3   Access Control Lists

    Not all keys can perform all actions. The scheduler, for example, sometimes need to issue specific actions to agents (such as during the upgrade protocol) but shouldn't be able to perform more dangerous actions. This is enforced by an Access Control List, or ACL, stored on the agents. An ACL describes who can @@ -471,127 +362,19 @@ has the following syntax:

    }

    The default permission is overridden by module specific permissions.

    The ACL is currently applied to modules. In the future, ACL will have finer control to authorize access to specific functions of modules. For example, an investigator could be authorized to call the regex function of filechecker -module, but only in /etc. This functionality is not implemented yet.

    4   Agent initialization process

    The agent tries to be as autonomous as possible. One of the goal is to ship -agents without requiring external provisioning tools, such as Chef or Puppet. -Therefore, the agent attempts to install itself as a service, and also supports -a builtin upgrade protocol (described in the next section).

    As a portable binary, the agent needs to detect the type of operating system -and init method that is used by an endpoint. Depending on the endpoint, -different initialization methods are used. The diagram below explains the -decision process followed by the agent.

    .files/mig-agent-initialization-process.png

    Go does not provide support for running programs in the backgroud. On endpoints -that run upstart, systemd (linux) or launchd (darwin), this is not an issue -because the init daemon takes care of running the agent in the background, -rerouting its file descriptors and restarting on crash. On Windows and System-V, -however, the agent daemonizes by forking itself into foreground mode, and -re-forking itself on error (such as loss of connectivity to the relay). -On Windows and System-V, if the agent is killed, it will not be restarted -automatically.

    4.1   Registration process

    The initialization process goes through several environment detection steps -which are used to select the proper init method. Once started, the agent will -send a heartbeat to the public relay, and also store that heartbeat in its -run directory. The location of the run directory is platform specific.

    • windows: C:Windows

    • darwin: /Library/Preferences/mig/

    • linux: /var/run/mig/

    Below is a sample heartbeat message from a linux agent stored in -/var/run/mig/mig-agent.ok.

    {
-        "destructiontime": "0001-01-01T00:00:00Z",
-        "environment": {
-                "arch": "amd64",
-                "ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)",
-                "init": "upstart"
-        },
-        "heartbeatts": "2014-07-31T14:00:20.00442837-07:00",
-        "name": "someserver.example.net",
-        "os": "linux",
-        "pid": 26256,
-        "queueloc": "linux.someserver.example.net.5hsa811oda",
-        "starttime": "2014-07-30T21:34:48.525449401-07:00",
-        "version": "201407310027+bcbdd94.prod"
-}

    4.2   Check-In mode

    In infrastructure where running the agent as a permanent process is not -acceptable, it is possible to run the agent as a cron job. By starting the -agent with the flag -m agent-checkin, the agent will connect to the -configured relay, retrieve and run outstanding commands, and exit after 10 -seconds of inactivity.

    5   Agent upgrade process

    MIG supports upgrading agents in the wild. The upgrade protocol is designed with -security in mind. The flow diagram below presents a high-level view:

    Investigator          Scheduler             Agent             NewAgent           FileServer
-+-----------+         +-------+             +---+             +------+           +--------+
-          |                   |                   |                   |                   |
-          |    1.initiate     |                   |                   |                   |
-          |------------------>|                   |                   |                   |
-          |                   |  2.send command   |                   |                   |
-          |                   |------------------>| 3.verify          |                   |
-          |                   |                   |--------+          |                   |
-          |                   |                   |        |          |                   |
-          |                   |                   |        |          |                   |
-          |                   |                   |<-------+          |                   |
-          |                   |                   |                   |                   |
-          |                   |                   |    4.download     |                   |
-          |                   |                   |-------------------------------------->|
-          |                   |                   |                   |                   |
-          |                   |                   | 5.checksum        |                   |
-          |                   |                   |--------+          |                   |
-          |                   |                   |        |          |                   |
-          |                   |                   |        |          |                   |
-          |                   |                   |<-------+          |                   |
-          |                   |                   |                   |                   |
-          |                   |                   |      6.exec       |                   |
-          |                   |                   |------------------>|                   |
-          |                   |  7.return own PID |                   |                   |
-          |                   |<------------------|                   |                   |
-          |                   |                   |                   |                   |
-          |                   |------+ 8.mark     |                   |                   |
-          |                   |      | agent as   |                   |                   |
-          |                   |      | upgraded   |                   |                   |
-          |                   |<-----+            |                   |                   |
-          |                   |                   |                   |                   |
-          |                   |    9.register     |                   |                   |
-          |                   |<--------------------------------------|                   |
-          |                   |                   |                   |                   |
-          |                   |------+10.find dup |                   |                   |
-          |                   |      |agents in   |                   |                   |
-          |                   |      |registrations                   |                   |
-          |                   |<-----+            |                   |                   |
-          |                   |                   |                   |                   |
-          |                   |    11.send command to kill PID old agt|                   |
-          |                   |-------------------------------------->|                   |
-          |                   |                   |                   |                   |
-          |                   |  12.acknowledge   |                   |                   |
-          |                   |<--------------------------------------|                   |

    All upgrade operations are initiated by an investigator (1). The upgrade is -triggered by an action to the upgrade module with the following parameters:

    "Operations": [
-    {
-        "Module": "upgrade",
-        "Parameters": {
-            "linux/amd64": {
-                "to_version": "16eb58b-201404021544",
-                "location": "http://localhost/mig/bin/linux/amd64/mig-agent",
-                "checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"
-            }
-        }
-    }
-],

    • Each OS family and architecture have their own parameters (ex: "linux/amd64", -"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:

    • to_version is the version an agent should upgrade to

    • location points to a HTTPS address that contains the agent binary

    • checksum is a SHA256 hash of the agent binary to be verified after download

    The parameters above are signed using a standard PGP action signature.

    The upgrade action is forwarded to agents (2) like any other action. The action -signature is verified by the agent (3), and the upgrade module is called. The -module downloads the new binary (4), verifies the version and checksum (5) and -installs itself on the system.

    Assuming everything checks in, the old agent executes the binary of the new -agent (6). At that point, two agents are running on the same machine, and the -rest of the protocol is designed to shut down the old agent, and clean up.

    After executing the new agent, the old agent returns a successful result to the -scheduler, and includes its own PID in the results. -The new agent starts by registering with the scheduler (7). This tells the -scheduler that two agents are running on the same node, and one of them must -terminate. The scheduler sends a kill action to both agents with the PID of the -old agent (8). The kill action may be executed twice, but that doesn't matter. -When the scheduler receives the kill results (9), it sends a new action to check -for mig-agent processes (10). Only one should be found in the results (11), -and if that is the case, the scheduler tells the agent to remove the binary of -the old agent (12). When the agent returns (13), the upgrade protocol is done.

    If the PID of the old agent lingers on the system, an error is logged for the -investigator to decide what to do next. The scheduler does not attempt to clean -up the situation.

    6   Threat Model

    Running an agent as root on a large number of endpoints means that Mozilla +module, but only in /etc. This functionality is not implemented yet.

    4   Threat Model

    Running an agent as root on a large number of endpoints means that Mozilla InvestiGator is a target of choice to compromise an infrastructure. Without proper protections, a vulnerability in the agent or in the platform could lead to a compromission of the endpoints.

    The architectural choices made in MIG diminish the exposure of the endpoints to a compromise. And while the risk cannot be reduced to zero entirely, it would take an attacker direct control on the investigators key material, or be root -on the infrastructure in order to take control of MIG.

    MIG's security controls include:

    • Strong GPG security model

    • Infrastructure resiliency

    • No port listening

    • Protection of connections to the relays

    • Randomization of the queue names

    • Whitelisting of agents

    • Limit data extraction to a minimum

    6.1   Strong GPG security model

    All actions that are passed to the MIG platform and to the agents require +on the infrastructure in order to take control of MIG.

    MIG's security controls include:

    • Strong GPG security model

    • Infrastructure resiliency

    • No port listening

    • Protection of connections to the relays

    • Randomization of the queue names

    • Whitelisting of agents

    • Limit data extraction to a minimum

    4.1   Strong GPG security model

    All actions that are passed to the MIG platform and to the agents require valid GPG signatures from one or more trusted investigators. The public keys of trusted investigators are hardcoded in the agents, making it almost impossible to override without root access to the endpoints, or access to an investigator's private key. The GPG private keys are never seen by the MIG platform (API, Scheduler, Database or Relays). A compromise of the platform would not lead to -an attacker taking control of the agents and compromising the endpoints.

    6.2   Infrastructure resiliency

    One of the design goal of MIG is to make each components as stateless as +an attacker taking control of the agents and compromising the endpoints.

    4.2   Infrastructure resiliency

    One of the design goal of MIG is to make each components as stateless as possible. The database is used as a primary data store, and the schedulers and relays keep data in transit in their respective cache. But any of these components can go down and be rebuilt without compromising the resiliency of @@ -605,12 +388,12 @@ relay whenever they chose to, and pick up outstanding tasks.

    If the relays reconnect at regular intervals continuously. It is trivial to rebuild a fresh rabbitmq cluster, even on a new IP space, as long as the FQDN of the cluster, and the TLS cert/key and credentials of the AMQPS access point -remain the same.

    6.3   No port listening

    The agents do not accept incoming connections. There is no listening port that +remain the same.

    4.3   No port listening

    The agents do not accept incoming connections. There is no listening port that an attacker could use to exploit a vulnerability in the agent. Instead, the agent connects to the platform by establishing an outbound connection to the relays. The connection uses TLS, making it theorically impossible for an attacker to MITM without access to the PKI and DNS, both of which are not -part of the MIG platform.

    6.4   Protection of connections to the relays

    The rabbitmq relay of a MIG infrastructure may very well be listening on the +part of the MIG platform.

    4.4   Protection of connections to the relays

    The rabbitmq relay of a MIG infrastructure may very well be listening on the public internet. This is used when MIG agents are distributed into various environments, as opposed to concentrated on a single network location. RabbitMQ and Erlang provide a stable network stack, but are not shielded from a network @@ -623,7 +406,7 @@ agent will present a username and password to open the AMQP connection. Again, these credentials are shared across all agents, and are not used to authenticate individual agents. Their role is to assign an ACL to the agent. The ACL limits the AMQP action an agent can perform on the cluster. -See rabbitmq configuration for more information.

    6.5   Randomization of the queue names

    The protections above limit the exposure of the AMQP endpoint, but since the +See rabbitmq configuration for more information.

    4.5   Randomization of the queue names

    The protections above limit the exposure of the AMQP endpoint, but since the secrets are shared across all agents, the possibility still exists that an attacker gains access to the secrets, and establish a connection to the relays.

    Such access would have very limited capabilities. It cannot be used to publish commands to the agents, because publication is ACL-limited to the scheduler. @@ -631,7 +414,7 @@ It can be used to publish fake results to the scheduler, or listen on the agent queue for incoming commands.

    Both are made difficult by prepending a random number to the name of an agent queue. An agent queue is named using the following scheme:

    mig.agt.<OS family>.<Hostname>.<uid>

    The OS and hostname of a given agent are easy to guess, but the uid isn't. The UID is a 64 bits integer composed of nanosecond timestamps and a random 32 -bits integer, chosen by the agent on first start. It is specific to an endpoint.

    6.6   Whitelisting of agents

    At the moment, MIG does not provide a strong mechanism to authenticate agents. +bits integer, chosen by the agent on first start. It is specific to an endpoint.

    4.6   Whitelisting of agents

    At the moment, MIG does not provide a strong mechanism to authenticate agents. It is a work in progress, but for now agents are whitelisted in the scheduler using the queuelocs that are advertised in the heartbeat messages. Spoofing the queueloc string is difficult, because it contains a random value that is @@ -640,7 +423,7 @@ order to spoof an agent's identity. This method provides a basic access control mechanism. The long term goal is to allow the scheduler to call an external database to authorize agents. In AWS, the scheduler could call the AWS API to verify that a given agent does indeed exist in the infrastructure. In a traditional datacenter, -this could be an inventory database.

    6.7   Limit data extraction to a minimum

    Agents are not meant to retrieve raw data from their endpoints. This is more +this could be an inventory database.

    4.7   Limit data extraction to a minimum

    Agents are not meant to retrieve raw data from their endpoints. This is more of a good practice rather than a technical limitation. The modules shipped with the agent are meant to return boolean answers of the type "match" or "no match".

    It could be argued that answering "match" on sensitive requests is similar to extracting data from the agents. MIG does not solve this issue.. It is the diff --git a/doc/configuration.rst b/doc/configuration.rst index 2a1db53a..45549d33 100644 --- a/doc/configuration.rst +++ b/doc/configuration.rst @@ -1,6 +1,5 @@ Mozilla InvestiGator Deployment and Configuration Documentation =============================================================== -:Author: Julien Vehent .. sectnum:: .. contents:: Table of Contents diff --git a/doc/configuration.rst.html b/doc/configuration.rst.html index e345dbad..786217c8 100644 --- a/doc/configuration.rst.html +++ b/doc/configuration.rst.html @@ -1,6 +1,6 @@

    Mozilla InvestiGator Deployment and Configuration Documentation

    AuthorJulien Vehent <jvehent@mozilla.com>

    This document describes the steps to build and configure a MIG platform. +

    Mozilla InvestiGator Deployment and Configuration Documentation

    This document describes the steps to build and configure a MIG platform. MIG has 6 major components. The Postgresql database and RabbitMQ relay are external dependencies, and while this document shows one way of deploying them, you are free to use your own method. All other components (scheduler, api, diff --git a/doc/console.rst b/doc/console.rst new file mode 100644 index 00000000..78eaaca9 --- /dev/null +++ b/doc/console.rst @@ -0,0 +1,495 @@ +=========== +MIG CONSOLE +=========== + +.. sectnum:: +.. contents:: Table of contents + +`mig-console` is a terminal-based client to the MIG API. It provides a complete +access to data stored in MIG and exposed by the API, in an interface that is +easy to use. Similar to the command line client, `mig-console` reads its +configuration from $HOME/.migrc and will prompt the user to create one if it +doesn't exists. + +The base screen of the API presents a dashboard of the current state of the +infrastructure. It includes the number of agents and endpoints, broken down +by version of the agent ,and a list of the latest actions that have run. + +Underneath the dasboard, the user is invited to interact in a prompt that +starts with **mig>**. + +.. code:: bash + + $ mig-console -c ~/.migrc-lnw + + ## ## _.---._ .---. + # # # /-\ ---|| | /\ __...---' .---. '---'-. '. + # #| | / || | /--\ .-''__.--' _.'( | )'. '. '._ : + # # \_/ ---| \_ \_/ \ .'__-'_ .--'' ._'---'_.-. '. '-'. + ### ~ -._ -._''---. -. '-._ '. + # |\ |\ /---------| ~ -.._ _ _ _ ..-_ '. '-._''--.._ + # | \| \ / |- |__ | | -~ -._ '-. -. '-._''--.._.--''. + ###| \ \/ ---__| | | ~ ~-.__ -._ '-.__ '. '. : + ##### ~~ ~---...__ _ ._ .' '. + # /\ --- /-\ |--|---- ~ ~--.....--~ + # ### /--\ | | ||-\ // + #####/ \ | \_/ | \//__ + +------ + | Agents & Endpoints summary: + | * 2 online agents on 2 endpoints + | * 0 idle agents on 0 endpoints + | * 0 endpoints are running 2 or more agents + | * 0 endpoints appeared over the last 7 days + | * 0 endpoints disappeared over the last 7 days + | * 0 endpoints have been flapping + | Online agents by version: + | * version 20150909+f2456f5.dev: 2 agents + | Idle agents by version: + | + | Latest Actions: + | ---- ID ---- + ---- Name ---- + -Sent- + ---- Date ---- + ---- Investigators ---- + | 4999271350274 file -c /home/ulfr/.migrc-l... 2 2015-09-29T15:40:35Z Julien Vehent + | 4964811669519 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:37:16Z Julien Vehent + | 4964811669506 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:37:03Z Julien Vehent + | 4964764024853 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:25:26Z Julien Vehent + | 4964764024834 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:24:57Z Julien Vehent + | 4949328330767 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:59:39Z Julien Vehent + | 4949328330754 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:59:25Z Julien Vehent + | 4948324450316 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:45:51Z Julien Vehent + | 4948324450307 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:33:17Z Julien Vehent + | 4947944865794 file -c /home/ulfr/.migrc-l... 2 2015-09-20T14:07:36Z Julien Vehent + +------ + + Connected to https://jaffa.linuxwall.info/api/v1/. Exit with ctrl+d. Type help for help. + mig> + +Entering **help** here, or on any other mode of the console, provides the user +with a list of available functionalities:: + + mig> help + The following orders are available: + action enter interactive action reader mode for action + agent enter interactive agent reader mode for agent + create action create a new action + create investigator create a new investigator, will prompt for name and public key + command enter command reader mode for command + exit leave + help show this help + history print last entries in history. count=10 by default. + investigator enter interactive investigator management mode for investigator + query send a raw query string, without the base url, to the api + search perform a search. see "search help" for more information. + showcfg display running configuration + status display platform status: connected agents, latest actions, ... + +For example, let's review action id 4999271350274:: + + mig> action 4999271350274 + Entering action reader mode. Type exit or press ctrl+d to leave. help may help. + Action: 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '. + Launched by 'Julien Vehent' on '2015-09-29 11:40:31.792627 -0400 EDT'. + Status 'completed'. + 2 sent, 2 done, 2 succeeded + action 274> + +When entering the action reader mode, a short summary of the action is displayed +to the user. We can see that the action was launched by "Julien Vehent" on 2 agents +and that it completed. The prompt is changed to **action 274** to indicate that +we are now in action reader mode. Entering **help** here provides a list of +functionalities specific to that mode:: + + The following orders are available: + command jump to command reader mode for command + + copy enter action launcher mode using current action as template + + counters display the counters of the action + + details display the details of the action, including status & times + + exit exit this mode (also works with ctrl+d) + + help show this help + + investigators print the list of investigators that signed the action + + json show the json of the action + + list returns the list of commands with their status + : * set to "all" to get all results (default) + * set to "found" to only display positive results + * set to "notfound" for negative results + list can be followed by a 'filter' pipe: + ex: ls | grep server1.(dom1|dom2) | grep -v example.net + + r refresh the action (get latest version from upstream) + + results display results of all commands + : * set to "all" to get all results (default) + * set to "found" to only display positive results + * set to "notfound" for negative results + : * set to "text" to print results in console (default) + * set to "map" to generate an open a google map + + times show the various timestamps of the action + +It is possible to review results by entering **results**. If we only want results +from agents that have found *something*, we can use **results found**. If we +want those results inside a Google Map, enter **results found map**. + +The raw json of the action is available via **json**. + +To get a list of agents on which the action ran, use **list all**. You can filter +the list using **list found** and **list notfound** to only get agents that have, +or have not, found something. This command also supports a very basic `grep`. + +To continue, let's list the agents that have found *something*:: + + action 274> list found + .. + ---- Command ID ---- ---- Agent Name & ID---- + 4999268991155 server1.example.net [4942082344151] + 4999268991154 server2.example.net [4942082360682] + 2 agents have found things + +We can inspect the command that ran on server1 by entering its ID:: + + action 274> command 4999268991155 + Entering command reader mode. Type exit or press ctrl+d to leave. help may help. + Command 4999268991155 ran on agent 'server1.example.net' based on action 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien ' + command 155> + +As you can see, the console mode has changed from `action 274` to +`command 155`. This mode has its own set of functionalities that you +can explore via **help**. + +Creating actions +---------------- + +The console provides a fine grained action generation interface. There +are two ways to create a new action: + +1. From the **mig>** mode, create a new empty action using **create action**. + +2. From the **action>** reader mode, when reviewing a previous action, + copy it to a new action using **copy**. + +Both methods enter the action launcher mode. Method 2 only enters this +mode with an preset action, rather than an empty one:: + + mig> create action + Entering action launcher with empty template + Type exit or press ctrl+d to leave. help may help. + launcher> help + The following orders are available: + addoperation append a new operation of type to the action operations + listagents list agents targetted by an action + deloperation remove operation numbered from operations array, count starts at zero + details display the action details + exit exit this mode + help show this help + json show the json of the action + launch launch the action. to return before completion, add "nofollow" + load load an action from a file at + setname set the name of the action + settarget set the target + settimes set the validity and expiration dates + sign PGP sign the action + times show the various timestamps of the action + +Action parameters can be edited prior to launching it: + +* **setname** sets the name field of the action to a new string:: + + launcher> setname Test action that pings google.com + +* **settarget** sets the target of the action. The target is evaluated + right away, and a list of targeted agents can be obtained via **listagents**:: + + launcher> settarget environment->>'os'='linux' and mode='daemon' + 2 agents will be targetted. To get the list, use 'listagents' + + launcher> listagents + ---- ID ---- + ---- Name ------- + 4942082360682 server1.example.net + 4942082344151 server2.example.net + +* **settimes** defines the validity period of the action using + and parameters. can be set to "now", and + can be a duration relative to "now":: + + launcher> settimes now +10m + launcher> times + Valid from '2015-10-06 13:09:36.189134664 +0000 UTC' until '2015-10-06 13:20:36.189134664 +0000 UTC' + Started on '0001-01-01 00:00:00 +0000 UTC' + Last updated '0001-01-01 00:00:00 +0000 UTC' + Finished on '0001-01-01 00:00:00 +0000 UTC' + +* **addoperation ** is used to add a new operation to the action. + must be a know MIG module, such as `file`, `netstat`, `memory`, + `scribe`, `ping`, `timedrift`, etc... When adding an operation, the console + enters a new module-specific mode that takes the operation parameters. For + example, this is how to add a ping operation that sends 5 TCP requests to + google.com:: + + launcher> addoperation ping + Ping module checks connectivity between an endpoint and a remote host. It supports + icmp, tcp and udp ping. See doc at http://mig.mozilla.org/doc/module_ping.html + + d Destination Address can be ipv4, ipv6 or FQDN + example: d www.mozilla.org + d 63.245.217.105 + + dp For TCP and UDP, specifies the port to test connectivity to + example: dp 53 + + p Protocol to use for the ping. This can be "icmp", "tcp" or "udp" + example: p udp + + c Number of ping/connection attempts. Defaults to 3. + example: c 5 + + t Connection timeout in seconds. Defaults to 5. + example: t 10 + ping> p tcp + ping> d google.com + ping> c 5 + ping> done + Inserting ping operation with parameters: + { + "module": "ping", + "parameters": { + "destination": google.com", + "protocol": "tcp", + "count": 5 + } + } + +* **deloperation** removes an operation from the action operations list. + Use **json** to visualize the list, and remove an operation using its + position in the list, starting with zero:: + + launcher> json + { + "id": 0, + "name": "Test action that pings google.com", + "target": "environment-\u003e\u003e'os'='linux' and mode='daemon'", + "description": {}, + "threat": {}, + "validfrom": "2015-10-06T13:09:36.189134664Z", + "expireafter": "2015-10-06T13:20:36.189134664Z", + "operations": [ + { + "module": "ping", + "parameters": { + "destination": "google.com", + "protocol": "tcp", + "count": 5 + } + } + ], + "pgpsignatures": null, + "starttime": "0001-01-01T00:00:00Z", + "finishtime": "0001-01-01T00:00:00Z", + "lastupdatetime": "0001-01-01T00:00:00Z", + "counters": {} + } + + launcher> deloperation 0 + + launcher> json + { + "id": 0, + "name": "Test action that pings google.com", + "target": "environment-\u003e\u003e'os'='linux' and mode='daemon'", + "description": {}, + "threat": {}, + "validfrom": "2015-10-06T13:09:36.189134664Z", + "expireafter": "2015-10-06T13:20:36.189134664Z", + "operations": [], + "pgpsignatures": null, + "starttime": "0001-01-01T00:00:00Z", + "finishtime": "0001-01-01T00:00:00Z", + "lastupdatetime": "0001-01-01T00:00:00Z", + "counters": {} + } + +When ready to launch the action, type **launch**. The console will enter +follower mode and print the progress of the action. When completed, the +console will directly enter action reader mode, where you can type **results** +to view the results:: + + launcher> launch + Action 'Test action that pings google.com' successfully launched with ID '5033038708749' on target 'environment->>'os'='linux' and mode='daemon'' + Following action ID 5033038708749.status=inflight...50%.status=completed + - 100.0% done in 10.004244071s + 2 sent, 2 done, 2 succeeded + + Entering action reader mode. Type exit or press ctrl+d to leave. help may help. + Action: 'Test action that pings google.com'. + Launched by 'Julien Vehent' on '2015-10-06 09:12:31.608546 -0400 EDT'. + Status 'completed'. + 2 sent, 2 done, 2 succeeded + action 749> results + server2.example.net ping of google.com failed. Target is no reachable. + server2.example.net command success + server1.example.net ping of google.com failed. Target is no reachable. + server1.example.net command success + 2 agents have all results + +Searching +--------- + +Return to the base mode using ctrl+d or **exit**:: + + command 155> exit + exit + action 274> exit + exit + mig> + +When in **mig>** mode, you can use the **search** functionality to +explore passed actions and commands, or find investigators. Details +of how to run searches can be obtains via **mig> search help**. + +Searches can be slow if your dataset is very large, so make sure to +use time windows and limits to help speed up a search:: + + mig> search action where investigatorname=%vehent% and agentname=server1% and after=2015-09-01T00:00:00Z and before=2015-10-01T00:00:00Z + Searching action after 2015-09-01T00:00:00Z and before 2015-10-01T00:00:00Z, limited to 100 results + ----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Investigators ---- + - Sent - + - Status - + --- Last Updated --- + 4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-29T15:40:35Z + 4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:16Z + 4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:03Z + 4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:25:26Z + 4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:24:57Z + 4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:39Z + 4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:25Z + 4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:45:51Z + 4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:33:17Z + 4947944865794 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T14:07:36Z + 4947909869570 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:58:41Z + 4947901022223 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:42Z + 4947901022210 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:26Z + 4947890798596 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:55:02Z + 4885615083769 timedrift -c /home/ulfr/.mi... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:02:56Z + 4885615083755 pkg -c /home/ulfr/.migrc-ln... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:33Z + 4885615083739 memory -c /home/ulfr/.migrc... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:04Z + 4885615083724 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T16:58:00Z + +Managing investigators +---------------------- + +The console can be used to create new investigators, active and +disable them, and well as review their activity. + +To review the activity of an investigator, find its ID using the +search command and enter investigator mode:: + + mig> search investigator where name=%vehent% + Searching investigator after 2011-03-31T13:16:55Z and before 2020-04-12T13:16:55Z, limited to 100 results + - ID - + ---- Name ---- + --- Status --- + 2 Julien Vehent active + + mig> investigator 2 + Entering investigator mode. Type exit or press ctrl+d to leave. help may help. + Investigator 2 named 'Julien Vehent' + + inv 2> help + The following orders are available: + details print the details of the investigator + exit exit this mode + help show this help + lastactions print the last actions ran by the investigator. limit=10 by default. + pubkey show the armored public key of the investigator + r refresh the investigator (get latest version from upstream) + setstatus changes the status of the investigator to (can be 'active' or 'disabled') + +The command **lastactions** will print the latest activity of this investigator:: + + inv 2> lastactions + ----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Date ---- + -- Status -- + 5033038708749 Test action that pings goog... environment->>'os'='linux' ... 2015-10-06T09:12:31-04:00 completed + 4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-29T11:40:31-04:00 completed + 4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:37:12-04:00 completed + 4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:36:59-04:00 completed + 4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:25:22-04:00 inflight + 4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:24:52-04:00 inflight + 4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:35-04:00 completed + 4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:21-04:00 completed + 4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:45:47-04:00 completed + 4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:33:13-04:00 completed + +To disable him, use **setstatus disabled**. Disabled investigators are no longer +allowed to send investigations via the API:: + + inv 2> setstatus disabled + Investigator status set to disabled + + inv 2> details + Investigator ID 2 + name Julien Vehent + status disabled + key id E60892BB9BD89A69F759A1A0A3D652173B763E8F + created 2015-09-09 09:53:28.989481 -0400 EDT + modified 2015-09-09 09:53:28.989481 -0400 EDT + + inv 2> setstatus active + Investigator status set to active + +Creating investigators +~~~~~~~~~~~~~~~~~~~~~~ + +To create a new investigator, go back to **mig>** mode and type +**create investigator**. The console will prompt the name of the new +investigator as well as the location of her public PGP key. You can either +provide a local path to the public key file on disk, on provide a fingerprint +in the format "0x<40 char sha1 hash>". When a fingerprint is provided, the +console will attempt to retrieve the key from the keyserver `gpg.mozilla.org`:: + + mig> create investigator + Entering investigator creation mode. Please provide the full name + and the public key of the new investigator. + name> Bob Kelso + Name: 'Bob Kelso' + Please provide a public key. You can either provide a local path to the + armored public key file, or a full length PGP fingerprint. + example: + pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B + pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B + retrieving public key from http://gpg.mozilla.org + -----BEGIN PGP PUBLIC KEY BLOCK----- + Version: SKS 1.1.5 + Comment: Hostname: keyserver.mozilla.org + + mQENBEbv+5sBCADNHPvUIajRoxb/qylLrzwm9e+9sB8R/jhY4gxOzGZRDHECPvNeTUd9eogV + n24rQDTWowkE+t9sW7vlD3TUWdBEAhXEpDZBfzlTBWIzEb1m3hwPOQM10ZNX6jPS1WlGsfoE + LsUC0HmFTtOx4b5os9mIYbsjsDWd/JZjn0yUIv4eb28+fle6BkbgqIotLW4d1gTrxVlFc3be + m+4OqimQ/v2LZDV+uObEkbh4UvmTtOCCx8zAOyZohPmICUbmJBc8KWWhzLOo8b9ns/GqP41q + /9IuTQDXP2GUAKXzBKSdQiNzJP8Skfu4tPyGsGJSErprPC9t43HPPUgfeW9/sfuaw+vnABEB + AAG0JUp1bGllbiBWRUhFTlQgPGp1bGllbkBsaW51eHdhbGwuaW5mbz6JATYEEwECACAFAkbv + +5sCGy8GCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRBkUJDmQ2dze5U9CADK4Z6X02TP9afJ + AyWF32zM3UdMksJ/F2wuo2HBHT0iOomw4ecNzHyO1P5BTglm5LC5ZZrV+Dx6Jve75JiSDTSD + V3AhpR+M83rw8YKkeUrbTvfsy3+qhB7HYNIbCKT0lgAAL05SmDnYwYMQIV+p3T0F8BgGhkGT + vHdKLhzEhKNOMaMCwCd1SsiBepA976oBUp9h5Vt6TEFyG6hCcFP90DFjNlK17yMNjbdrgyd6 + FkGEePKK3RhaLxcPShAgCnYzYYMABLYu1ow5AxxtEJTBHkJFkIzE9XM/lmyekYhWfX/Q4jpn + 1aiiVlf2klZlFSFy/TXuuQH1JO3YlKo9uHjlvSQb + =+Ru1 + -----END PGP PUBLIC KEY BLOCK----- + + create investigator? (y/n)> y + Investigator 'Bob Kelso' successfully created with ID 4 + + mig> investigator 4 + Entering investigator mode. Type exit or press ctrl+d to leave. help may help. + Investigator 4 named 'Bob Kelso' + + inv 4> details + Investigator ID 4 + name Bob Kelso + status active + key id 716CFA6BA8EBB21E860AE231645090E64367737B + created 2015-10-06 09:23:14.473307 -0400 EDT + modified 2015-10-06 09:23:14.473307 -0400 EDT + +The new investigator now has access to the API. diff --git a/doc/console.rst.html b/doc/console.rst.html new file mode 100644 index 00000000..438ac1de --- /dev/null +++ b/doc/console.rst.html @@ -0,0 +1,605 @@ +

    MIG CONSOLE

    mig-console is a terminal-based client to the MIG API. It provides a complete +access to data stored in MIG and exposed by the API, in an interface that is +easy to use. Similar to the command line client, mig-console reads its +configuration from $HOME/.migrc and will prompt the user to create one if it +doesn't exists.

    The base screen of the API presents a dashboard of the current state of the +infrastructure. It includes the number of agents and endpoints, broken down +by version of the agent ,and a list of the latest actions that have run.

    Underneath the dasboard, the user is invited to interact in a prompt that +starts with mig>.

    $ mig-console -c ~/.migrc-lnw
+
+## ##                                     _.---._     .---.
+# # # /-\ ---||  |    /\         __...---' .---. '---'-.   '.
+#   #|   | / ||  |   /--\    .-''__.--' _.'( | )'.  '.  '._ :
+#   # \_/ ---| \_ \_/    \ .'__-'_ .--'' ._'---'_.-.  '.   '-'.
+         ###                         ~ -._ -._''---. -.    '-._   '.
+          # |\ |\    /---------|          ~ -.._ _ _ _ ..-_ '.  '-._''--.._
+          # | \| \  / |- |__ | |                       -~ -._  '-.  -. '-._''--.._.--''.
+         ###|  \  \/  ---__| | |                            ~ ~-.__     -._  '-.__   '. '.                                                                                                                                                                   :
+                  #####                                               ~~ ~---...__ _    ._ .' '.
+                  #      /\  --- /-\ |--|----                                    ~  ~--.....--~
+                  # ### /--\  | |   ||-\  //
+                  #####/    \ |  \_/ |  \//__
++------
+| Agents & Endpoints summary:
+| * 2 online agents on 2 endpoints
+| * 0 idle agents on 0 endpoints
+| * 0 endpoints are running 2 or more agents
+| * 0 endpoints appeared over the last 7 days
+| * 0 endpoints disappeared over the last 7 days
+| * 0 endpoints have been flapping
+| Online agents by version:
+| * version 20150909+f2456f5.dev: 2 agents
+| Idle agents by version:
+|
+| Latest Actions:
+| ----  ID  ---- + ----         Name         ---- + -Sent- + ----    Date    ---- + ---- Investigators ----
+| 4999271350274    file -c /home/ulfr/.migrc-l...        2   2015-09-29T15:40:35Z   Julien Vehent
+| 4964811669519    file -c /home/ulfr/.migrc-l...        2   2015-09-23T13:37:16Z   Julien Vehent
+| 4964811669506    file -c /home/ulfr/.migrc-l...        2   2015-09-23T13:37:03Z   Julien Vehent
+| 4964764024853    file -c /home/ulfr/.migrc-l...        2   2015-09-23T13:25:26Z   Julien Vehent
+| 4964764024834    file -c /home/ulfr/.migrc-l...        2   2015-09-23T13:24:57Z   Julien Vehent
+| 4949328330767    file -c /home/ulfr/.migrc-l...        2   2015-09-20T19:59:39Z   Julien Vehent
+| 4949328330754    file -c /home/ulfr/.migrc-l...        2   2015-09-20T19:59:25Z   Julien Vehent
+| 4948324450316    file -c /home/ulfr/.migrc-l...        2   2015-09-20T19:45:51Z   Julien Vehent
+| 4948324450307    file -c /home/ulfr/.migrc-l...        2   2015-09-20T19:33:17Z   Julien Vehent
+| 4947944865794    file -c /home/ulfr/.migrc-l...        2   2015-09-20T14:07:36Z   Julien Vehent
++------
+
+Connected to https://jaffa.linuxwall.info/api/v1/. Exit with ctrl+d. Type help for help.
+mig>

    Entering help here, or on any other mode of the console, provides the user +with a list of available functionalities:

    mig> help
+The following orders are available:
+action <id>             enter interactive action reader mode for action <id>
+agent <id>              enter interactive agent reader mode for agent <id>
+create action           create a new action
+create investigator     create a new investigator, will prompt for name and public key
+command <id>            enter command reader mode for command <id>
+exit                    leave
+help                    show this help
+history <count>         print last <count> entries in history. count=10 by default.
+investigator <id>       enter interactive investigator management mode for investigator <id>
+query <uri>             send a raw query string, without the base url, to the api
+search <search>         perform a search. see "search help" for more information.
+showcfg                 display running configuration
+status                  display platform status: connected agents, latest actions, ...

    For example, let's review action id 4999271350274:

    mig> action 4999271350274
+Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
+Action: 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '.
+Launched by 'Julien Vehent' on '2015-09-29 11:40:31.792627 -0400 EDT'.
+Status 'completed'.
+2 sent, 2 done, 2 succeeded
+action 274>

    When entering the action reader mode, a short summary of the action is displayed +to the user. We can see that the action was launched by "Julien Vehent" on 2 agents +and that it completed. The prompt is changed to action 274 to indicate that +we are now in action reader mode. Entering help here provides a list of +functionalities specific to that mode:

    The following orders are available:
+command <id>    jump to command reader mode for command <id>
+
+copy            enter action launcher mode using current action as template
+
+counters        display the counters of the action
+
+details         display the details of the action, including status & times
+
+exit            exit this mode (also works with ctrl+d)
+
+help            show this help
+
+investigators   print the list of investigators that signed the action
+
+json            show the json of the action
+
+list <show>     returns the list of commands with their status
+                                <show>: * set to "all" to get all results (default)
+                                                * set to "found" to only display positive results
+                                                * set to "notfound" for negative results
+                                list can be followed by a 'filter' pipe:
+                                ex: ls | grep server1.(dom1|dom2) | grep -v example.net
+
+r               refresh the action (get latest version from upstream)
+
+results <show> <render> display results of all commands
+                                                <show>: * set to "all" to get all results (default)
+                                                                * set to "found" to only display positive results
+                                                                * set to "notfound" for negative results
+                                                <render>: * set to "text" to print results in console (default)
+                                                                  * set to "map" to generate an open a google map
+
+times           show the various timestamps of the action

    It is possible to review results by entering results. If we only want results +from agents that have found something, we can use results found. If we +want those results inside a Google Map, enter results found map.

    The raw json of the action is available via json.

    To get a list of agents on which the action ran, use list all. You can filter +the list using list found and list notfound to only get agents that have, +or have not, found something. This command also supports a very basic grep.

    To continue, let's list the agents that have found something:

    action 274> list found
+..
+---- Command ID ----    ---- Agent Name & ID----
+           4999268991155    server1.example.net [4942082344151]
+           4999268991154    server2.example.net [4942082360682]
+2 agents have found things

    We can inspect the command that ran on server1 by entering its ID:

    action 274> command 4999268991155
+Entering command reader mode. Type exit or press ctrl+d to leave. help may help.
+Command 4999268991155 ran on agent 'server1.example.net' based on action 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '
+command 155>

    As you can see, the console mode has changed from action 274 to +command 155. This mode has its own set of functionalities that you +can explore via help.

    1   Creating actions

    The console provides a fine grained action generation interface. There +are two ways to create a new action:

    1. From the mig> mode, create a new empty action using create action.

    2. From the action> reader mode, when reviewing a previous action, +copy it to a new action using copy.

    Both methods enter the action launcher mode. Method 2 only enters this +mode with an preset action, rather than an empty one:

    mig> create action
+Entering action launcher with empty template
+Type exit or press ctrl+d to leave. help may help.
+launcher> help
+The following orders are available:
+addoperation <module>   append a new operation of type <module> to the action operations
+listagents              list agents targetted by an action
+deloperation <opnum>    remove operation numbered <opnum> from operations array, count starts at zero
+details                 display the action details
+exit                    exit this mode
+help                    show this help
+json <pack>             show the json of the action
+launch <nofollow>       launch the action. to return before completion, add "nofollow"
+load <path>             load an action from a file at <path>
+setname <name>          set the name of the action
+settarget <target>      set the target
+settimes <start> <stop> set the validity and expiration dates
+sign                    PGP sign the action
+times                   show the various timestamps of the action

    Action parameters can be edited prior to launching it:

    • setname sets the name field of the action to a new string:

      launcher> setname Test action that pings google.com

    • settarget sets the target of the action. The target is evaluated +right away, and a list of targeted agents can be obtained via listagents:

      launcher> settarget environment->>'os'='linux' and mode='daemon'
+2 agents will be targetted. To get the list, use 'listagents'
+
+launcher> listagents
+----    ID      ---- + ----         Name         -------
+           4942082360682   server1.example.net
+           4942082344151   server2.example.net

    • settimes defines the validity period of the action using +<start> and <stop> parameters. <start> can be set to "now", and <stop> +can be a duration relative to "now":

      launcher> settimes now +10m
+launcher> times
+Valid from   '2015-10-06 13:09:36.189134664 +0000 UTC' until '2015-10-06 13:20:36.189134664 +0000 UTC'
+Started on   '0001-01-01 00:00:00 +0000 UTC'
+Last updated '0001-01-01 00:00:00 +0000 UTC'
+Finished on  '0001-01-01 00:00:00 +0000 UTC'

    • addoperation <module> is used to add a new operation to the action. +<module> must be a know MIG module, such as file, netstat, memory, +scribe, ping, timedrift, etc... When adding an operation, the console +enters a new module-specific mode that takes the operation parameters. For +example, this is how to add a ping operation that sends 5 TCP requests to +google.com:

      launcher> addoperation ping
+Ping module checks connectivity between an endpoint and a remote host. It supports
+icmp, tcp and udp ping. See doc at http://mig.mozilla.org/doc/module_ping.html
+
+d <ip/fqdn>     Destination Address can be ipv4, ipv6 or FQDN
+                                example: d www.mozilla.org
+                                                 d 63.245.217.105
+
+dp <port>       For TCP and UDP, specifies the port to test connectivity to
+                                example: dp 53
+
+p <protocol>    Protocol to use for the ping. This can be "icmp", "tcp" or "udp"
+                                example: p udp
+
+c <count>       Number of ping/connection attempts. Defaults to 3.
+                                example: c 5
+
+t <timeout>     Connection timeout in seconds. Defaults to 5.
+                                example: t 10
+ping> p tcp
+ping> d google.com
+ping> c 5
+ping> done
+Inserting ping operation with parameters:
+{
+  "module": "ping",
+  "parameters": {
+        "destination": google.com",
+        "protocol": "tcp",
+        "count": 5
+  }
+}

    • deloperation removes an operation from the action operations list. +Use json to visualize the list, and remove an operation using its +position in the list, starting with zero:

      launcher> json
+{
+  "id": 0,
+  "name": "Test action that pings google.com",
+  "target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
+  "description": {},
+  "threat": {},
+  "validfrom": "2015-10-06T13:09:36.189134664Z",
+  "expireafter": "2015-10-06T13:20:36.189134664Z",
+  "operations": [
+        {
+          "module": "ping",
+          "parameters": {
+                "destination": "google.com",
+                "protocol": "tcp",
+                "count": 5
+          }
+        }
+  ],
+  "pgpsignatures": null,
+  "starttime": "0001-01-01T00:00:00Z",
+  "finishtime": "0001-01-01T00:00:00Z",
+  "lastupdatetime": "0001-01-01T00:00:00Z",
+  "counters": {}
+}
+
+launcher> deloperation 0
+
+launcher> json
+{
+  "id": 0,
+  "name": "Test action that pings google.com",
+  "target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
+  "description": {},
+  "threat": {},
+  "validfrom": "2015-10-06T13:09:36.189134664Z",
+  "expireafter": "2015-10-06T13:20:36.189134664Z",
+  "operations": [],
+  "pgpsignatures": null,
+  "starttime": "0001-01-01T00:00:00Z",
+  "finishtime": "0001-01-01T00:00:00Z",
+  "lastupdatetime": "0001-01-01T00:00:00Z",
+  "counters": {}
+}

    When ready to launch the action, type launch. The console will enter +follower mode and print the progress of the action. When completed, the +console will directly enter action reader mode, where you can type results +to view the results:

    launcher> launch
+Action 'Test action that pings google.com' successfully launched with ID '5033038708749' on target 'environment->>'os'='linux' and mode='daemon''
+Following action ID 5033038708749.status=inflight...50%.status=completed
+- 100.0% done in 10.004244071s
+2 sent, 2 done, 2 succeeded
+
+Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
+Action: 'Test action that pings google.com'.
+Launched by 'Julien Vehent' on '2015-10-06 09:12:31.608546 -0400 EDT'.
+Status 'completed'.
+2 sent, 2 done, 2 succeeded
+action 749> results
+server2.example.net  ping of google.com failed. Target is no reachable.
+server2.example.net command success
+server1.example.net  ping of google.com failed. Target is no reachable.
+server1.example.net command success
+2 agents have all results

    2   Searching

    Return to the base mode using ctrl+d or exit:

    command 155> exit
+exit
+action 274> exit
+exit
+mig>

    When in mig> mode, you can use the search functionality to +explore passed actions and commands, or find investigators. Details +of how to run searches can be obtains via mig> search help.

    Searches can be slow if your dataset is very large, so make sure to +use time windows and limits to help speed up a search:

    mig> search action where investigatorname=%vehent% and agentname=server1% and after=2015-09-01T00:00:00Z and before=2015-10-01T00:00:00Z
+Searching action after 2015-09-01T00:00:00Z and before 2015-10-01T00:00:00Z, limited to 100 results
+----- ID ----- + --------   Action Name ------- + ----------- Target  ---------- + ---- Investigators ---- + - Sent - + - Status - + --- Last Updated ---
+4999271350274    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-29T15:40:35Z
+4964811669519    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-23T13:37:16Z
+4964811669506    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-23T13:37:03Z
+4964764024853    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   inflight     2015-09-23T13:25:26Z
+4964764024834    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   inflight     2015-09-23T13:24:57Z
+4949328330767    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T19:59:39Z
+4949328330754    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T19:59:25Z
+4948324450316    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T19:45:51Z
+4948324450307    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T19:33:17Z
+4947944865794    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T14:07:36Z
+4947909869570    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T13:58:41Z
+4947901022223    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T13:56:42Z
+4947901022210    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T13:56:26Z
+4947890798596    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    2   completed    2015-09-20T13:55:02Z
+4885615083769    timedrift -c /home/ulfr/.mi...   status='online' AND mode='d...   Julien Vehent                    3   completed    2015-09-09T17:02:56Z
+4885615083755    pkg -c /home/ulfr/.migrc-ln...   status='online' AND mode='d...   Julien Vehent                    3   completed    2015-09-09T17:01:33Z
+4885615083739    memory -c /home/ulfr/.migrc...   status='online' AND mode='d...   Julien Vehent                    3   completed    2015-09-09T17:01:04Z
+4885615083724    file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   Julien Vehent                    3   completed    2015-09-09T16:58:00Z

    3   Managing investigators

    The console can be used to create new investigators, active and +disable them, and well as review their activity.

    To review the activity of an investigator, find its ID using the +search command and enter investigator mode:

    mig> search investigator where name=%vehent%
+Searching investigator after 2011-03-31T13:16:55Z and before 2020-04-12T13:16:55Z, limited to 100 results
+- ID - + ----         Name         ---- + --- Status ---
+         2   Julien Vehent                    active
+
+mig> investigator 2
+Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
+Investigator 2 named 'Julien Vehent'
+
+inv 2> help
+The following orders are available:
+details                 print the details of the investigator
+exit                    exit this mode
+help                    show this help
+lastactions <limit>     print the last actions ran by the investigator. limit=10 by default.
+pubkey                  show the armored public key of the investigator
+r                       refresh the investigator (get latest version from upstream)
+setstatus <status>      changes the status of the investigator to <status> (can be 'active' or 'disabled')

    The command lastactions will print the latest activity of this investigator:

    inv 2> lastactions
+----- ID ----- + --------    Action Name ------- + ----------- Target   ---------- + ----    Date    ---- +  -- Status --
+5033038708749     Test action that pings goog...   environment->>'os'='linux' ...   2015-10-06T09:12:31-04:00    completed
+4999271350274     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-29T11:40:31-04:00    completed
+4964811669519     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-23T09:37:12-04:00    completed
+4964811669506     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-23T09:36:59-04:00    completed
+4964764024853     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-23T09:25:22-04:00    inflight
+4964764024834     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-23T09:24:52-04:00    inflight
+4949328330767     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-20T15:59:35-04:00    completed
+4949328330754     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-20T15:59:21-04:00    completed
+4948324450316     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-20T15:45:47-04:00    completed
+4948324450307     file -c /home/ulfr/.migrc-l...   status='online' AND mode='d...   2015-09-20T15:33:13-04:00    completed

    To disable him, use setstatus disabled. Disabled investigators are no longer +allowed to send investigations via the API:

    inv 2> setstatus disabled
+Investigator status set to disabled
+
+inv 2> details
+Investigator ID 2
+name     Julien Vehent
+status   disabled
+key id   E60892BB9BD89A69F759A1A0A3D652173B763E8F
+created  2015-09-09 09:53:28.989481 -0400 EDT
+modified 2015-09-09 09:53:28.989481 -0400 EDT
+
+inv 2> setstatus active
+Investigator status set to active

    3.1   Creating investigators

    To create a new investigator, go back to mig> mode and type +create investigator. The console will prompt the name of the new +investigator as well as the location of her public PGP key. You can either +provide a local path to the public key file on disk, on provide a fingerprint +in the format "0x<40 char sha1 hash>". When a fingerprint is provided, the +console will attempt to retrieve the key from the keyserver gpg.mozilla.org:

    mig> create investigator
+Entering investigator creation mode. Please provide the full name
+and the public key of the new investigator.
+name> Bob Kelso
+Name: 'Bob Kelso'
+Please provide a public key. You can either provide a local path to the
+armored public key file, or a full length PGP fingerprint.
+example:
+pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
+pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
+retrieving public key from http://gpg.mozilla.org
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: SKS 1.1.5
+Comment: Hostname: keyserver.mozilla.org
+
+mQENBEbv+5sBCADNHPvUIajRoxb/qylLrzwm9e+9sB8R/jhY4gxOzGZRDHECPvNeTUd9eogV
+n24rQDTWowkE+t9sW7vlD3TUWdBEAhXEpDZBfzlTBWIzEb1m3hwPOQM10ZNX6jPS1WlGsfoE
+LsUC0HmFTtOx4b5os9mIYbsjsDWd/JZjn0yUIv4eb28+fle6BkbgqIotLW4d1gTrxVlFc3be
+m+4OqimQ/v2LZDV+uObEkbh4UvmTtOCCx8zAOyZohPmICUbmJBc8KWWhzLOo8b9ns/GqP41q
+/9IuTQDXP2GUAKXzBKSdQiNzJP8Skfu4tPyGsGJSErprPC9t43HPPUgfeW9/sfuaw+vnABEB
+AAG0JUp1bGllbiBWRUhFTlQgPGp1bGllbkBsaW51eHdhbGwuaW5mbz6JATYEEwECACAFAkbv
++5sCGy8GCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRBkUJDmQ2dze5U9CADK4Z6X02TP9afJ
+AyWF32zM3UdMksJ/F2wuo2HBHT0iOomw4ecNzHyO1P5BTglm5LC5ZZrV+Dx6Jve75JiSDTSD
+V3AhpR+M83rw8YKkeUrbTvfsy3+qhB7HYNIbCKT0lgAAL05SmDnYwYMQIV+p3T0F8BgGhkGT
+vHdKLhzEhKNOMaMCwCd1SsiBepA976oBUp9h5Vt6TEFyG6hCcFP90DFjNlK17yMNjbdrgyd6
+FkGEePKK3RhaLxcPShAgCnYzYYMABLYu1ow5AxxtEJTBHkJFkIzE9XM/lmyekYhWfX/Q4jpn
+1aiiVlf2klZlFSFy/TXuuQH1JO3YlKo9uHjlvSQb
+=+Ru1
+-----END PGP PUBLIC KEY BLOCK-----
+
+create investigator? (y/n)> y
+Investigator 'Bob Kelso' successfully created with ID 4
+
+mig> investigator 4
+Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
+Investigator 4 named 'Bob Kelso'
+
+inv 4> details
+Investigator ID 4
+name     Bob Kelso
+status   active
+key id   716CFA6BA8EBB21E860AE231645090E64367737B
+created  2015-10-06 09:23:14.473307 -0400 EDT
+modified 2015-10-06 09:23:14.473307 -0400 EDT

    The new investigator now has access to the API.

    \ No newline at end of file diff --git a/doc/data.rst b/doc/data.rst index 1e94ed4e..1036a450 100644 --- a/doc/data.rst +++ b/doc/data.rst @@ -1,7 +1,6 @@ ======== MIG Data ======== -:Author: Julien Vehent .. sectnum:: .. contents:: Table of Contents diff --git a/doc/data.rst.html b/doc/data.rst.html index 2be4c669..49187d73 100644 --- a/doc/data.rst.html +++ b/doc/data.rst.html @@ -1,6 +1,6 @@

    MIG Data

    AuthorJulien Vehent <jvehent@mozilla.com>

    1   Postgresql

    1.1   Database creation script

    Two scripts can be used to create a database for MIG.

    • createlocaldb.sh will create a database on an instance of postgresql +

      MIG Data

      1   Postgresql

      1.1   Database creation script

      Two scripts can be used to create a database for MIG.

      • createlocaldb.sh will create a database on an instance of postgresql running locally. This is used by the standalone installation script.

      • createremotedb.sh will connect to an existing MIG database on a remote postgresql server. This is a standard production setup. It assumes that you have created a database beforehand. You can pass the DB credentials by diff --git a/doc/docstyle.css b/doc/docstyle.css index d0db1503..b9ce0053 100644 --- a/doc/docstyle.css +++ b/doc/docstyle.css @@ -1,6 +1,6 @@ body { width: 95%; - max-width: 900px; + max-width: 70%; margin: 20px; padding: 0; background: #151515 url("../images/bkg.png") 0 0; @@ -29,7 +29,7 @@ section { h1, h2, h3, h4, h5, h6 { /*margin: 0 0 20px;*/ - margin: 0; + /*margin: 0;*/ } /* Header,

        @@ -78,6 +78,8 @@ header h2 { body { width: 100%; + margin-left: auto; + margin-right: auto; -webkit-font-smoothing: antialiased; } section img { diff --git a/doc/mig-worker-agent-intel.html b/doc/mig-worker-agent-intel.html index 71d4028f..11d7fbd4 100644 --- a/doc/mig-worker-agent-intel.html +++ b/doc/mig-worker-agent-intel.html @@ -1,6 +1,6 @@

        MIG Modules

        AuthorJulien Vehent <jvehent@mozilla.com>

        In this document, we explain how modules are written and integrated into MIG.

        The reception of a command by an agent triggers the execution of modules. A +

        MIG Modules

        In this document, we explain how modules are written and integrated into MIG.

        The reception of a command by an agent triggers the execution of modules. A module is a Go package that is imported into the agent at compilation, and that performs a very specific set of tasks. For example, the file module provides a way to scan a file system for files that contain regexes, match a diff --git a/doc/runner.rst b/doc/runner.rst index 0c826ac3..f4841dd0 100644 --- a/doc/runner.rst +++ b/doc/runner.rst @@ -1,7 +1,6 @@ ========== MIG RUNNER ========== -:Author: Aaron Meihm .. sectnum:: .. contents:: Table of contents diff --git a/doc/runner.rst.html b/doc/runner.rst.html new file mode 100644 index 00000000..28d5a4c6 --- /dev/null +++ b/doc/runner.rst.html @@ -0,0 +1,293 @@ +

        MIG RUNNER

        MIG runner is a service that can be deployed to automatically schedule actions +to run within the MIG environment and retrieve/process the results.

        The runner interacts directly with the MIG API in the same manner as a client +would. When an action is scheduled to run, the runner will deploy the action +and schedule a time to gather results (shortly after the action has expired). +Once the action has expired, the runner will retrieve results from the API +and store these results in the runner directory. The runner can also send the +results from MIG to an external program for automatic parsing or formatting, +for example to create events for MozDef and send them.

        1   Runner configuration file

        An example configuration file for use by mig-runner is shown below.

        ; Sample MIG runner configuration file
+
+[runner]
+directory = "/home/mig-runner/runner" ; The path to the root runner directory
+checkdirectory = 30 ; How often to check runners/ for job changes
+
+[logging]
+mode = "stdout" ; stdout | file | syslog
+level = "debug"
+
+[client]
+clientconfpath = "default" ; Path to client conf, default for $HOME/.migrc
+delayresults = "30s"; Duration after action expiry to fetch results

        If the GPG key used by mig-runner is protected by a passphrase, the +passphrase option can be included under the client section. If this is +specified this passphrase will be used to access the private key.

        The delayresults value is optional. If not set, the runner will attempt +to fetch action results when the action has expired. If this is set to a +duration string value, the runner will wait the specified duration after +action expiry before fetching results (for example to ensure all results +are written to the database by the scheduler).

        The checkdirectory option specifies the number of seconds that elapse +between scans of the runner directory for job changes. The runner will +automatically add or remove jobs as new jobs are added in the spool +directory. When the runner identifies changes to a job, this will be +indicated in the log file as the old job configuration will be removed +and the new configuration installed.

        The directory option specifies the root directory that stores all the +mig-runner related control information. A typical runner directory may look +something like this.

        runner/
+|
++ runners/
+| |
+| + job1/
+| + job2/
+|
++ plugins/

        2   Job configuration

        Under each job directory, a file entity.cfg defines the parameters used to +run this job.

        [configuration]
+schedule = "<cronexpr>"
+plugin = "<plugin name>"

        The schedule option should be set to a cron style expression to note when +the job should be run.

        The plugin is optional. If set, the value will be interpreted as an +executable in the plugins directory. The results of the job will be piped +into stdin of this executable in JSON format (mig.RunnerResult). The +plugin can then parse and forward the data as needed.

        Optionally the expiry setting can be set to a go Duration string to use +for action expiry (for example 10m for 10 minutes). If this is not set +in the job configuration, a default of 5 minutes will be used.

        Optionally the sendonly setting can be set to true, which will result in +the runner only dispatching the action, but not attempting to retrieve +any results. As an example, this can be used to deploy actions that would +be processed by MIG workers, rather than retrieved and processed on the +runner side.

        The results are also written into a results/ subdirectory under the +runner directory, using the action ID as a file name. This happens +regardless of any plugin configuration for the job.

        In the job directory, the MIG action should should be launched should be +called action.json. The time validity and expiration fields will be +filled in by the runner process before dispatching the action to the +API.

        3   Output plugins

        The runner writes JSON output to stdin of any configured output plugin. This +is intended to provide flexibility, allowing plugins to be developed in +any language. If a plugin is being developed in go, the mig.RunnerResult +type can be used to parse incoming data. In other languages the JSON can +be parsed as desired.

        \ No newline at end of file