зеркало из https://github.com/mozilla/mig.git
[doc] tons of new docs, with some CSS changes
This commit is contained in:
Julien Vehent
Родитель
75161adc20
Коммит
7c4a70049e
336
doc/agent.rst
336
doc/agent.rst
|
|
@ -1,10 +1,234 @@
|
|||
Agent
|
||||
=====
|
||||
======================
|
||||
MIG Agent Architecture
|
||||
======================
|
||||
|
||||
The agent accepts different classes of inputs on stdin, as one-line JSON objects. The most common one is the
|
||||
``parameters`` class, but it could also receive a ``stop`` input that
|
||||
indicates that the module should stop its execution immediately. The format of
|
||||
module input messages is defined by ``modules.Message``.
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
||||
Initialization process
|
||||
----------------------------
|
||||
The agent tries to be as autonomous as possible. One of the goal is to ship
|
||||
agents without requiring external provisioning tools, such as Chef or Puppet.
|
||||
Therefore, the agent attempts to install itself as a service, and also supports
|
||||
a builtin upgrade protocol (described in the next section).
|
||||
|
||||
As a portable binary, the agent needs to detect the type of operating system
|
||||
and init method that is used by an endpoint. Depending on the endpoint,
|
||||
different initialization methods are used. The diagram below explains the
|
||||
decision process followed by the agent.
|
||||
|
||||
.. image:: .files/mig-agent-initialization-process.png
|
||||
|
||||
Go does not provide support for running programs in the backgroud. On endpoints
|
||||
that run upstart, systemd (linux) or launchd (darwin), this is not an issue
|
||||
because the init daemon takes care of running the agent in the background,
|
||||
rerouting its file descriptors and restarting on crash. On Windows and System-V,
|
||||
however, the agent daemonizes by forking itself into `foreground` mode, and
|
||||
re-forking itself on error (such as loss of connectivity to the relay).
|
||||
On Windows and System-V, if the agent is killed, it will not be restarted
|
||||
automatically.
|
||||
|
||||
Registration process
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The initialization process goes through several environment detection steps
|
||||
which are used to select the proper init method. Once started, the agent will
|
||||
send a heartbeat to the public relay, and also store that heartbeat in its
|
||||
`run` directory. The location of the `run` directory is platform specific.
|
||||
|
||||
* windows: C:\Windows\
|
||||
* darwin: /Library/Preferences/mig/
|
||||
* linux: /var/run/mig/
|
||||
|
||||
Below is a sample heartbeat message from a linux agent stored in
|
||||
`/var/run/mig/mig-agent.ok`.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"destructiontime": "0001-01-01T00:00:00Z",
|
||||
"environment": {
|
||||
"arch": "amd64",
|
||||
"ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)",
|
||||
"init": "upstart"
|
||||
},
|
||||
"heartbeatts": "2014-07-31T14:00:20.00442837-07:00",
|
||||
"name": "someserver.example.net",
|
||||
"os": "linux",
|
||||
"pid": 26256,
|
||||
"queueloc": "linux.someserver.example.net.5hsa811oda",
|
||||
"starttime": "2014-07-30T21:34:48.525449401-07:00",
|
||||
"version": "201407310027+bcbdd94.prod"
|
||||
}
|
||||
|
||||
Check-In mode
|
||||
~~~~~~~~~~~~~
|
||||
In infrastructure where running the agent as a permanent process is not
|
||||
acceptable, it is possible to run the agent as a cron job. By starting the
|
||||
agent with the flag **-m agent-checkin**, the agent will connect to the
|
||||
configured relay, retrieve and run outstanding commands, and exit after 10
|
||||
seconds of inactivity.
|
||||
|
||||
Communication with modules
|
||||
--------------------------
|
||||
|
||||
Upon processing of an action, the scheduler will retrieve a list of agents to
|
||||
send the action to. One action is then derived into multiple commands and sent
|
||||
to agents.
|
||||
|
||||
An agent receives a command from the scheduler on its personal AMQP queue (1).
|
||||
It parses the command (2) and extracts all of the operations to perform.
|
||||
Operations are passed to modules and executed in parallel (3). Rather than
|
||||
maintaining a state of the running command, the agent create a goroutine and a
|
||||
channel tasked with receiving the results from the modules. Each modules
|
||||
published its results inside that channel (4). The result parsing goroutine
|
||||
receives them, and when it has received all of them, populates the `results` (5)
|
||||
array of the command with the results from each module, and send the command
|
||||
back to the scheduler(6).
|
||||
|
||||
When the agent is done running the command, both the channel and the goroutine
|
||||
are destroyed.
|
||||
|
||||
::
|
||||
|
||||
+-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ]
|
||||
|command|+---->(listener)
|
||||
+-------+ |(2)
|
||||
^ V
|
||||
|(1) (parser)
|
||||
| + [ m o d u l e s ]
|
||||
+---------+ | (3)|----------> op1 +----------------+
|
||||
|SCHEDULER|+---+ |------------> op2 +--------------|
|
||||
| |<---+ |--------------> op3 +------------|
|
||||
+---------+ | +----------------> op4 +----------+
|
||||
| V(4)
|
||||
|(6) (receiver)
|
||||
| |
|
||||
| V(5)
|
||||
+ (publisher)
|
||||
+-------+ /
|
||||
|results|<-----------------------------------------'
|
||||
+-------+
|
||||
|
||||
The command received by the agent is composed of a copy of the action described
|
||||
previously, but signed with the private key of a trusted investigator. It also
|
||||
contains additional parameters that are specific to the targetted agent, such as
|
||||
command processing timestamps, name of the agent queue on the message broker,
|
||||
action and command unique IDs and status and results of the command. Below is an
|
||||
command derived from the root password checking action, and ran on the host named
|
||||
'host1.example.net'.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"id": 1.427392971126604e+18,
|
||||
"action": { ... SIGNED COPY OF THE ACTION ... },
|
||||
"agent": {
|
||||
"id": 1.4271760437936648e+18,
|
||||
"name": "host1.example.net",
|
||||
"queueloc": "linux.host1.example.net.981alsd19aos1984",
|
||||
"mode": "daemon",
|
||||
"version": "20150324+0d0f88c.prod"
|
||||
},
|
||||
"status": "success",
|
||||
"results": [
|
||||
{
|
||||
"foundanything": true,
|
||||
"success": true,
|
||||
"elements": {
|
||||
"root_passwd_hashed_or_disabled": [
|
||||
{
|
||||
"file": "/etc/shadow",
|
||||
"fileinfo": {
|
||||
"lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC",
|
||||
"mode": "----------",
|
||||
"size": 1684
|
||||
},
|
||||
"search": {
|
||||
"contents": [
|
||||
"root:(\\*|!|\\$(1|2a|5|6)\\$).+"
|
||||
],
|
||||
"options": {
|
||||
"matchall": false,
|
||||
"matchlimit": 0,
|
||||
"maxdepth": 0
|
||||
},
|
||||
"paths": [
|
||||
"/etc"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"statistics": {
|
||||
"exectime": "2.017849ms",
|
||||
"filescount": 1,
|
||||
"openfailed": 0,
|
||||
"totalhits": 1
|
||||
},
|
||||
"errors": null
|
||||
}
|
||||
],
|
||||
"starttime": "2015-03-26T18:02:51.126605Z",
|
||||
"finishtime": "2015-03-26T18:03:00.671232Z"
|
||||
}
|
||||
|
||||
The results of the command show that the file '/etc/shadow' has matched, and
|
||||
thus "FoundAnything" returned "True".
|
||||
|
||||
The invocation of the file module has completed successfully, which is
|
||||
represented by **results->0->success=true**. In our example, there is only one
|
||||
operation in the **action->operations** array, so only one result is present.
|
||||
When multiple operations are performed, each has its results listed in a
|
||||
corresponding entry of the results array (operations[0] is in results[0],
|
||||
operations[1] in results[1], etc...).
|
||||
|
||||
Finally, the agent has performed all operations in the operations array
|
||||
successfully, and returned **status=success**. Had a failure happened on the
|
||||
agent, the returned status would be one of "failed", "timeout" or "cancelled".
|
||||
|
||||
Command expiration & timeouts
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To prevent abuse of resources, agents will kill long-running modules after a
|
||||
given period of time. That timeout is hardcoded in the agent configuration
|
||||
at compile time and defaults to 5 minutes.
|
||||
|
||||
.. code:: go
|
||||
|
||||
// timeout after which a module run is killed
|
||||
var MODULETIMEOUT time.Duration = 300 * time.Second
|
||||
|
||||
That timeout represents the **maximum** execution time of a single operation. If
|
||||
an action contains 3 operations, each operation gets its own timeout. But because
|
||||
operations run in parallel in the agent, the maximum runtime of an action should
|
||||
be very close to the value of MODULETIMEOUT.
|
||||
|
||||
In a typical deployment, it is safe to increase MODULETIMEOUT to allow for
|
||||
longer operations. A value of 20 minutes is usual. Make sure to fine tune this
|
||||
to your environment, and get the approval of your ops team because mig-agent
|
||||
may end up consuming resources (but never more than 50% of the cpu available on
|
||||
a system).
|
||||
|
||||
Oftentimes, an investigator will want a timeout that is much shorter than the value
|
||||
of MODULETIMEOUT. In the MIG command line, the flag `-e` controls the
|
||||
expiration. It defaults to 5 minutes but can be set to 30 seconds for simple
|
||||
investigations. When that happens, the agent will calculate an appropriate expiration
|
||||
for the operations being run. If the expiration set on the action is set to 30 seconds,
|
||||
the agent will kill operations that run for more than 30 seconds.
|
||||
|
||||
If the expiration is larger than the value of MODULETIMEOUT (for example, 2
|
||||
hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to
|
||||
allow agents that only check in periodically to pick up actions long after they
|
||||
are launched.
|
||||
|
||||
Agent/Modules message format
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The agent accepts different classes of inputs on stdin, as one-line JSON objects.
|
||||
The most common one is the ``parameters`` class, but it could also receive a
|
||||
``stop`` input that indicates that the module should stop its execution immediately.
|
||||
The format of module input messages is defined by ``modules.Message``.
|
||||
|
||||
.. code:: go
|
||||
|
||||
|
|
@ -25,4 +249,104 @@ and copies them into ``Message.Parameters``. It then sets ``Message.Class`` to
|
|||
``modules.MsgClassParameters``, marshals the struct into JSON, and pass the
|
||||
resulting ``[]byte`` to the module as an IO stream.
|
||||
|
||||
Agent upgrade process
|
||||
---------------------
|
||||
MIG supports upgrading agents in the wild. The upgrade protocol is designed with
|
||||
security in mind. The flow diagram below presents a high-level view:
|
||||
|
||||
::
|
||||
|
||||
Investigator Scheduler Agent NewAgent FileServer
|
||||
+-----------+ +-------+ +---+ +------+ +--------+
|
||||
| | | | |
|
||||
| 1.initiate | | | |
|
||||
|------------------>| | | |
|
||||
| | 2.send command | | |
|
||||
| |------------------>| 3.verify | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 4.download | |
|
||||
| | |-------------------------------------->|
|
||||
| | | | |
|
||||
| | | 5.checksum | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 6.exec | |
|
||||
| | |------------------>| |
|
||||
| | 7.return own PID | | |
|
||||
| |<------------------| | |
|
||||
| | | | |
|
||||
| |------+ 8.mark | | |
|
||||
| | | agent as | | |
|
||||
| | | upgraded | | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 9.register | | |
|
||||
| |<--------------------------------------| |
|
||||
| | | | |
|
||||
| |------+10.find dup | | |
|
||||
| | |agents in | | |
|
||||
| | |registrations | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 11.send command to kill PID old agt| |
|
||||
| |-------------------------------------->| |
|
||||
| | | | |
|
||||
| | 12.acknowledge | | |
|
||||
| |<--------------------------------------| |
|
||||
|
||||
All upgrade operations are initiated by an investigator (1). The upgrade is
|
||||
triggered by an action to the upgrade module with the following parameters:
|
||||
|
||||
.. code:: json
|
||||
|
||||
"Operations": [
|
||||
{
|
||||
"Module": "upgrade",
|
||||
"Parameters": {
|
||||
"linux/amd64": {
|
||||
"to_version": "16eb58b-201404021544",
|
||||
"location": "http://localhost/mig/bin/linux/amd64/mig-agent",
|
||||
"checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
|
||||
* Each OS family and architecture have their own parameters (ex: "linux/amd64",
|
||||
"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:
|
||||
* to_version is the version an agent should upgrade to
|
||||
* location points to a HTTPS address that contains the agent binary
|
||||
* checksum is a SHA256 hash of the agent binary to be verified after download
|
||||
|
||||
The parameters above are signed using a standard PGP action signature.
|
||||
|
||||
The upgrade action is forwarded to agents (2) like any other action. The action
|
||||
signature is verified by the agent (3), and the upgrade module is called. The
|
||||
module downloads the new binary (4), verifies the version and checksum (5) and
|
||||
installs itself on the system.
|
||||
|
||||
Assuming everything checks in, the old agent executes the binary of the new
|
||||
agent (6). At that point, two agents are running on the same machine, and the
|
||||
rest of the protocol is designed to shut down the old agent, and clean up.
|
||||
|
||||
After executing the new agent, the old agent returns a successful result to the
|
||||
scheduler, and includes its own PID in the results.
|
||||
The new agent starts by registering with the scheduler (7). This tells the
|
||||
scheduler that two agents are running on the same node, and one of them must
|
||||
terminate. The scheduler sends a kill action to both agents with the PID of the
|
||||
old agent (8). The kill action may be executed twice, but that doesn't matter.
|
||||
When the scheduler receives the kill results (9), it sends a new action to check
|
||||
for `mig-agent` processes (10). Only one should be found in the results (11),
|
||||
and if that is the case, the scheduler tells the agent to remove the binary of
|
||||
the old agent (12). When the agent returns (13), the upgrade protocol is done.
|
||||
|
||||
If the PID of the old agent lingers on the system, an error is logged for the
|
||||
investigator to decide what to do next. The scheduler does not attempt to clean
|
||||
up the situation.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
@ -227,10 +229,156 @@ cite {
|
|||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>Agent</h1><p>The agent accepts different classes of inputs on stdin, as one-line JSON objects. The most common one is the
|
||||
<code>parameters</code> class, but it could also receive a <code>stop</code> input that
|
||||
indicates that the module should stop its execution immediately. The format of
|
||||
module input messages is defined by <code>modules.Message</code>.</p><pre><code class="code go"><span class="comment single">// Message defines the input messages received by modules.
|
||||
</style></head><body><h1>MIG Agent Architecture</h1><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#initialization-process" id="id1">1 Initialization process</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#registration-process" id="id2">1.1 Registration process</a></p></li><li><p><a class="reference internal" href="#check-in-mode" id="id3">1.2 Check-In mode</a></p></li></ul></li><li><p><a class="reference internal" href="#communication-with-modules" id="id4">2 Communication with modules</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#command-expiration-timeouts" id="id5">2.1 Command expiration & timeouts</a></p></li><li><p><a class="reference internal" href="#agent-modules-message-format" id="id6">2.2 Agent/Modules message format</a></p></li></ul></li><li><p><a class="reference internal" href="#agent-upgrade-process" id="id7">3 Agent upgrade process</a></p></li></ul></div><section id="initialization-process"><header><h2><a href="#id1">1 Initialization process</a></h2></header><p>The agent tries to be as autonomous as possible. One of the goal is to ship
|
||||
agents without requiring external provisioning tools, such as Chef or Puppet.
|
||||
Therefore, the agent attempts to install itself as a service, and also supports
|
||||
a builtin upgrade protocol (described in the next section).</p><p>As a portable binary, the agent needs to detect the type of operating system
|
||||
and init method that is used by an endpoint. Depending on the endpoint,
|
||||
different initialization methods are used. The diagram below explains the
|
||||
decision process followed by the agent.</p><img alt=".files/mig-agent-initialization-process.png" src=".files/mig-agent-initialization-process.png"><p>Go does not provide support for running programs in the backgroud. On endpoints
|
||||
that run upstart, systemd (linux) or launchd (darwin), this is not an issue
|
||||
because the init daemon takes care of running the agent in the background,
|
||||
rerouting its file descriptors and restarting on crash. On Windows and System-V,
|
||||
however, the agent daemonizes by forking itself into <cite>foreground</cite> mode, and
|
||||
re-forking itself on error (such as loss of connectivity to the relay).
|
||||
On Windows and System-V, if the agent is killed, it will not be restarted
|
||||
automatically.</p><section id="registration-process"><header><h3><a href="#id2">1.1 Registration process</a></h3></header><p>The initialization process goes through several environment detection steps
|
||||
which are used to select the proper init method. Once started, the agent will
|
||||
send a heartbeat to the public relay, and also store that heartbeat in its
|
||||
<cite>run</cite> directory. The location of the <cite>run</cite> directory is platform specific.</p><ul><li><p>windows: C:Windows</p></li><li><p>darwin: /Library/Preferences/mig/</p></li><li><p>linux: /var/run/mig/</p></li></ul><p>Below is a sample heartbeat message from a linux agent stored in
|
||||
<cite>/var/run/mig/mig-agent.ok</cite>.</p><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"destructiontime"</span><span class="punctuation">:</span> <span class="literal string double">"0001-01-01T00:00:00Z"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"environment"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"arch"</span><span class="punctuation">:</span> <span class="literal string double">"amd64"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"ident"</span><span class="punctuation">:</span> <span class="literal string double">"Red Hat Enterprise Linux Server release 6.5 (Santiago)"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"init"</span><span class="punctuation">:</span> <span class="literal string double">"upstart"</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"heartbeatts"</span><span class="punctuation">:</span> <span class="literal string double">"2014-07-31T14:00:20.00442837-07:00"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"someserver.example.net"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"os"</span><span class="punctuation">:</span> <span class="literal string double">"linux"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"pid"</span><span class="punctuation">:</span> <span class="literal number integer">26256</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"queueloc"</span><span class="punctuation">:</span> <span class="literal string double">"linux.someserver.example.net.5hsa811oda"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"starttime"</span><span class="punctuation">:</span> <span class="literal string double">"2014-07-30T21:34:48.525449401-07:00"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"201407310027+bcbdd94.prod"</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="check-in-mode"><header><h3><a href="#id3">1.2 Check-In mode</a></h3></header><p>In infrastructure where running the agent as a permanent process is not
|
||||
acceptable, it is possible to run the agent as a cron job. By starting the
|
||||
agent with the flag <strong>-m agent-checkin</strong>, the agent will connect to the
|
||||
configured relay, retrieve and run outstanding commands, and exit after 10
|
||||
seconds of inactivity.</p></section></section><section id="communication-with-modules"><header><h2><a href="#id4">2 Communication with modules</a></h2></header><p>Upon processing of an action, the scheduler will retrieve a list of agents to
|
||||
send the action to. One action is then derived into multiple commands and sent
|
||||
to agents.</p><p>An agent receives a command from the scheduler on its personal AMQP queue (1).
|
||||
It parses the command (2) and extracts all of the operations to perform.
|
||||
Operations are passed to modules and executed in parallel (3). Rather than
|
||||
maintaining a state of the running command, the agent create a goroutine and a
|
||||
channel tasked with receiving the results from the modules. Each modules
|
||||
published its results inside that channel (4). The result parsing goroutine
|
||||
receives them, and when it has received all of them, populates the <cite>results</cite> (5)
|
||||
array of the command with the results from each module, and send the command
|
||||
back to the scheduler(6).</p><p>When the agent is done running the command, both the channel and the goroutine
|
||||
are destroyed.</p><blockquote><pre> +-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ]
|
||||
|command|+---->(listener)
|
||||
+-------+ |(2)
|
||||
^ V
|
||||
|(1) (parser)
|
||||
| + [ m o d u l e s ]
|
||||
+---------+ | (3)|----------> op1 +----------------+
|
||||
|SCHEDULER|+---+ |------------> op2 +--------------|
|
||||
| |<---+ |--------------> op3 +------------|
|
||||
+---------+ | +----------------> op4 +----------+
|
||||
| V(4)
|
||||
|(6) (receiver)
|
||||
| |
|
||||
| V(5)
|
||||
+ (publisher)
|
||||
+-------+ /
|
||||
|results|<-----------------------------------------'
|
||||
+-------+</pre></blockquote><p>The command received by the agent is composed of a copy of the action described
|
||||
previously, but signed with the private key of a trusted investigator. It also
|
||||
contains additional parameters that are specific to the targetted agent, such as
|
||||
command processing timestamps, name of the agent queue on the message broker,
|
||||
action and command unique IDs and status and results of the command. Below is an
|
||||
command derived from the root password checking action, and ran on the host named
|
||||
'host1.example.net'.</p><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"id"</span><span class="punctuation">:</span> <span class="literal number float">1.427392971126604e+18</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"action"</span><span class="punctuation">:</span> <span class="punctuation">{</span> <span class="error">...</span> <span class="error">SIGNED</span> <span class="error">COPY</span> <span class="error">OF</span> <span class="error">THE</span> <span class="error">ACTION</span> <span class="error">...</span> <span class="punctuation">},</span>
|
||||
<span class="name tag">"agent"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"id"</span><span class="punctuation">:</span> <span class="literal number float">1.4271760437936648e+18</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"host1.example.net"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"queueloc"</span><span class="punctuation">:</span> <span class="literal string double">"linux.host1.example.net.981alsd19aos1984"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"mode"</span><span class="punctuation">:</span> <span class="literal string double">"daemon"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"20150324+0d0f88c.prod"</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"status"</span><span class="punctuation">:</span> <span class="literal string double">"success"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"results"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"foundanything"</span><span class="punctuation">:</span> <span class="keyword constant">true</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"success"</span><span class="punctuation">:</span> <span class="keyword constant">true</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"elements"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"root_passwd_hashed_or_disabled"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"file"</span><span class="punctuation">:</span> <span class="literal string double">"/etc/shadow"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"fileinfo"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"lastmodified"</span><span class="punctuation">:</span> <span class="literal string double">"2015-02-07 01:51:07.17850601 +0000 UTC"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"mode"</span><span class="punctuation">:</span> <span class="literal string double">"----------"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"size"</span><span class="punctuation">:</span> <span class="literal number integer">1684</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"search"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"contents"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="literal string double">"root:(\\*|!|\\$(1|2a|5|6)\\$).+"</span>
|
||||
<span class="punctuation">],</span>
|
||||
<span class="name tag">"options"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"matchall"</span><span class="punctuation">:</span> <span class="keyword constant">false</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"matchlimit"</span><span class="punctuation">:</span> <span class="literal number integer">0</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"maxdepth"</span><span class="punctuation">:</span> <span class="literal number integer">0</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"paths"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="literal string double">"/etc"</span>
|
||||
<span class="punctuation">]</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">]</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"statistics"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"exectime"</span><span class="punctuation">:</span> <span class="literal string double">"2.017849ms"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"filescount"</span><span class="punctuation">:</span> <span class="literal number integer">1</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"openfailed"</span><span class="punctuation">:</span> <span class="literal number integer">0</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"totalhits"</span><span class="punctuation">:</span> <span class="literal number integer">1</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"errors"</span><span class="punctuation">:</span> <span class="keyword constant">null</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">],</span>
|
||||
<span class="name tag">"starttime"</span><span class="punctuation">:</span> <span class="literal string double">"2015-03-26T18:02:51.126605Z"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"finishtime"</span><span class="punctuation">:</span> <span class="literal string double">"2015-03-26T18:03:00.671232Z"</span>
|
||||
<span class="punctuation">}</span></code></pre><p>The results of the command show that the file '/etc/shadow' has matched, and
|
||||
thus "FoundAnything" returned "True".</p><p>The invocation of the file module has completed successfully, which is
|
||||
represented by <strong>results->0->success=true</strong>. In our example, there is only one
|
||||
operation in the <strong>action->operations</strong> array, so only one result is present.
|
||||
When multiple operations are performed, each has its results listed in a
|
||||
corresponding entry of the results array (operations[0] is in results[0],
|
||||
operations[1] in results[1], etc...).</p><p>Finally, the agent has performed all operations in the operations array
|
||||
successfully, and returned <strong>status=success</strong>. Had a failure happened on the
|
||||
agent, the returned status would be one of "failed", "timeout" or "cancelled".</p><section id="command-expiration-timeouts"><header><h3><a href="#id5">2.1 Command expiration & timeouts</a></h3></header><p>To prevent abuse of resources, agents will kill long-running modules after a
|
||||
given period of time. That timeout is hardcoded in the agent configuration
|
||||
at compile time and defaults to 5 minutes.</p><pre><code class="code go"><span class="comment single">// timeout after which a module run is killed
|
||||
</span><span class="keyword declaration">var</span> <span class="name other">MODULETIMEOUT</span> <span class="name other">time</span><span class="punctuation">.</span><span class="name other">Duration</span> <span class="punctuation">=</span> <span class="literal number integer">300</span> <span class="operator">*</span> <span class="name other">time</span><span class="punctuation">.</span><span class="name other">Second</span></code></pre><p>That timeout represents the <strong>maximum</strong> execution time of a single operation. If
|
||||
an action contains 3 operations, each operation gets its own timeout. But because
|
||||
operations run in parallel in the agent, the maximum runtime of an action should
|
||||
be very close to the value of MODULETIMEOUT.</p><p>In a typical deployment, it is safe to increase MODULETIMEOUT to allow for
|
||||
longer operations. A value of 20 minutes is usual. Make sure to fine tune this
|
||||
to your environment, and get the approval of your ops team because mig-agent
|
||||
may end up consuming resources (but never more than 50% of the cpu available on
|
||||
a system).</p><p>Oftentimes, an investigator will want a timeout that is much shorter than the value
|
||||
of MODULETIMEOUT. In the MIG command line, the flag <cite>-e</cite> controls the
|
||||
expiration. It defaults to 5 minutes but can be set to 30 seconds for simple
|
||||
investigations. When that happens, the agent will calculate an appropriate expiration
|
||||
for the operations being run. If the expiration set on the action is set to 30 seconds,
|
||||
the agent will kill operations that run for more than 30 seconds.</p><p>If the expiration is larger than the value of MODULETIMEOUT (for example, 2
|
||||
hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to
|
||||
allow agents that only check in periodically to pick up actions long after they
|
||||
are launched.</p></section><section id="agent-modules-message-format"><header><h3><a href="#id6">2.2 Agent/Modules message format</a></h3></header><p>The agent accepts different classes of inputs on stdin, as one-line JSON objects.
|
||||
The most common one is the <code>parameters</code> class, but it could also receive a
|
||||
<code>stop</code> input that indicates that the module should stop its execution immediately.
|
||||
The format of module input messages is defined by <code>modules.Message</code>.</p><pre><code class="code go"><span class="comment single">// Message defines the input messages received by modules.
|
||||
</span><span class="keyword declaration">type</span> <span class="name other">Message</span> <span class="keyword declaration">struct</span> <span class="punctuation">{</span>
|
||||
<span class="name other">Class</span> <span class="keyword type">string</span> <span class="comment single">// represent the type of message being passed to the module
|
||||
</span> <span class="name other">Parameters</span> <span class="keyword declaration">interface</span><span class="punctuation">{}</span> <span class="comment single">// for `parameters` class, this interface contains the module parameters
|
||||
|
|
@ -243,4 +391,77 @@ module input messages is defined by <code>modules.Message</code>.</p><pre><code
|
|||
extracts the operation parameters from <code>Command.Action.Operations[N].Parameters</code>
|
||||
and copies them into <code>Message.Parameters</code>. It then sets <code>Message.Class</code> to
|
||||
<code>modules.MsgClassParameters</code>, marshals the struct into JSON, and pass the
|
||||
resulting <code>[]byte</code> to the module as an IO stream.</p></body></html>
|
||||
resulting <code>[]byte</code> to the module as an IO stream.</p></section></section><section id="agent-upgrade-process"><header><h2><a href="#id7">3 Agent upgrade process</a></h2></header><p>MIG supports upgrading agents in the wild. The upgrade protocol is designed with
|
||||
security in mind. The flow diagram below presents a high-level view:</p><blockquote><pre>Investigator Scheduler Agent NewAgent FileServer
|
||||
+-----------+ +-------+ +---+ +------+ +--------+
|
||||
| | | | |
|
||||
| 1.initiate | | | |
|
||||
|------------------>| | | |
|
||||
| | 2.send command | | |
|
||||
| |------------------>| 3.verify | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 4.download | |
|
||||
| | |-------------------------------------->|
|
||||
| | | | |
|
||||
| | | 5.checksum | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 6.exec | |
|
||||
| | |------------------>| |
|
||||
| | 7.return own PID | | |
|
||||
| |<------------------| | |
|
||||
| | | | |
|
||||
| |------+ 8.mark | | |
|
||||
| | | agent as | | |
|
||||
| | | upgraded | | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 9.register | | |
|
||||
| |<--------------------------------------| |
|
||||
| | | | |
|
||||
| |------+10.find dup | | |
|
||||
| | |agents in | | |
|
||||
| | |registrations | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 11.send command to kill PID old agt| |
|
||||
| |-------------------------------------->| |
|
||||
| | | | |
|
||||
| | 12.acknowledge | | |
|
||||
| |<--------------------------------------| |</pre></blockquote><p>All upgrade operations are initiated by an investigator (1). The upgrade is
|
||||
triggered by an action to the upgrade module with the following parameters:</p><pre><code class="code json"><span class="literal string double">"Operations"</span><span class="error">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"Module"</span><span class="punctuation">:</span> <span class="literal string double">"upgrade"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"Parameters"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"linux/amd64"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"to_version"</span><span class="punctuation">:</span> <span class="literal string double">"16eb58b-201404021544"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"location"</span><span class="punctuation">:</span> <span class="literal string double">"http://localhost/mig/bin/linux/amd64/mig-agent"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"checksum"</span><span class="punctuation">:</span> <span class="literal string double">"31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">]</span><span class="error">,</span></code></pre><ul><li><p>Each OS family and architecture have their own parameters (ex: "linux/amd64",
|
||||
"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:</p></li><li><p>to_version is the version an agent should upgrade to</p></li><li><p>location points to a HTTPS address that contains the agent binary</p></li><li><p>checksum is a SHA256 hash of the agent binary to be verified after download</p></li></ul><p>The parameters above are signed using a standard PGP action signature.</p><p>The upgrade action is forwarded to agents (2) like any other action. The action
|
||||
signature is verified by the agent (3), and the upgrade module is called. The
|
||||
module downloads the new binary (4), verifies the version and checksum (5) and
|
||||
installs itself on the system.</p><p>Assuming everything checks in, the old agent executes the binary of the new
|
||||
agent (6). At that point, two agents are running on the same machine, and the
|
||||
rest of the protocol is designed to shut down the old agent, and clean up.</p><p>After executing the new agent, the old agent returns a successful result to the
|
||||
scheduler, and includes its own PID in the results.
|
||||
The new agent starts by registering with the scheduler (7). This tells the
|
||||
scheduler that two agents are running on the same node, and one of them must
|
||||
terminate. The scheduler sends a kill action to both agents with the PID of the
|
||||
old agent (8). The kill action may be executed twice, but that doesn't matter.
|
||||
When the scheduler receives the kill results (9), it sends a new action to check
|
||||
for <cite>mig-agent</cite> processes (10). Only one should be found in the results (11),
|
||||
and if that is the case, the scheduler tells the agent to remove the binary of
|
||||
the old agent (12). When the agent returns (13), the upgrade protocol is done.</p><p>If the PID of the old agent lingers on the system, an error is logged for the
|
||||
investigator to decide what to do next. The scheduler does not attempt to clean
|
||||
up the situation.</p></section></body></html>
|
||||
363
doc/api.rst
363
doc/api.rst
|
|
@ -1,7 +1,6 @@
|
|||
=======
|
||||
MIG API
|
||||
=======
|
||||
:Author: Julien Vehent <jvehent@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
|
@ -14,186 +13,8 @@ The API follows the core principles of REST, and provides discoverable
|
|||
endpoints. API responses follows the **cljs** format defined in
|
||||
`Collection+JSON - Hypermedia Type <http://amundsen.com/media-types/collection/>`_.
|
||||
|
||||
Authentication with X-PGPAUTHORIZATION version 1
|
||||
------------------------------------------------
|
||||
|
||||
Authenticating against the MIG API requires sending a PGP signed token in the
|
||||
request header named `X-PGPAUTHORIZATION`. The key that signs the token must
|
||||
belong to an active investigator. Construction of the token works as follows:
|
||||
|
||||
1. make a string named **str** composed of a version, a UTC timestamp in RFC3339 format
|
||||
and a random nonce, each separated by semicolons. The current version is **1**
|
||||
and may be upgraded in the future. The nonce value must be a positive integer.
|
||||
|
||||
**str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE>**
|
||||
|
||||
UTC is a hard requirement. The timestamp must end with the suffix **Z**
|
||||
which indicates the UTC timezone. In bash, a correct timestamp can be
|
||||
generated with the command `$ date -u +%Y-%m-%dT%H:%M:%SZ`.
|
||||
|
||||
An example string would look like: `1;2006-01-02T15:04:05Z;1825922807490630059`
|
||||
|
||||
The string must be terminated by a newline character, hexadecimal code `0x0a`.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
00000000 31 3b 32 30 30 36 2d 30 31 2d 30 32 54 31 35 3a |1;2006-01-02T15:|
|
||||
00000010 30 34 3a 30 35 5a 3b 31 38 32 35 39 32 32 38 30 |04:05Z;182592280|
|
||||
00000020 37 34 39 30 36 33 30 30 35 39 0a |7490630059.|
|
||||
0000002b
|
||||
|
||||
2. PGP sign **str** with the private key of the investigator. Armor and detach
|
||||
the signature into **armoredSig**::
|
||||
|
||||
$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
|
||||
JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
|
||||
kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
|
||||
b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
|
||||
PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
|
||||
MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
|
||||
=4nrR
|
||||
-----END PGP SIGNATURE-----
|
||||
|
||||
3. Create **sig** by taking **armoredSig** and removing the PGP headers, footers,
|
||||
empty lines and newlines.
|
||||
|
||||
example: `iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT`
|
||||
|
||||
4. Create **token** by concatenating **str**, a semicolon, and **sig**.
|
||||
**token=<str>;<sig>**
|
||||
example: `1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated>`
|
||||
|
||||
5. Send **token** in the header named **X-PGPAUTHORIZATION** with the request::
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/
|
||||
|
||||
6. The API verifies the version and validity period of the timestamp. By default, a
|
||||
token will be rejected if its timestamp deviates from the server time by more
|
||||
than 10 minutes. Administrators can configure this value. In effect, this
|
||||
means a timestamp is valid for twice the duration of the window. By default,
|
||||
that's 10 minutes before current server time, and 10 minutes after current
|
||||
server time.
|
||||
|
||||
7. If the timestamp is valid, the API next verifies the signature against the data
|
||||
and authenticates the user. Failure to verify the signature returns an error
|
||||
with the HTTP code 401 Unauthorized.
|
||||
|
||||
8. The user is authorized, the API processes and answer the request.
|
||||
|
||||
Security implications
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. A token can be used an unlimited number of times within its validity period.
|
||||
There is no check to guarantee that a token is only used once. It is
|
||||
assumed that the token is transmitted over a secure channel such as HTTPS to
|
||||
prevent token theft by a malicious user.
|
||||
|
||||
2. API clients and servers must use proper time synchronization for the timestamp
|
||||
verification to work. A client or a server that has inaccurate time may not be
|
||||
able to establish connections. We believe this requirement to be reasonable
|
||||
considering the sensitivity of the API.
|
||||
|
||||
Example 1: invalid timestamp
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The signature is valid but the timestamp is beyond the acceptable time window.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>' http://localhost:12345/api/v1/
|
||||
|
||||
{
|
||||
"collection": {
|
||||
"error": {
|
||||
"code": "6077873045059431424",
|
||||
"message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
|
||||
},
|
||||
"href": "http://localhost:12345/api/v1/",
|
||||
"template": {},
|
||||
"version": "1.0"
|
||||
}
|
||||
}
|
||||
|
||||
Example 2: invalid signature
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The signature is not valid, or is signed by a key that the API does not
|
||||
recognize.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>' http://localhost:12345/api/v1/
|
||||
|
||||
{
|
||||
"collection": {
|
||||
"error": {
|
||||
"code": "6077875007260332032",
|
||||
"message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
|
||||
},
|
||||
"href": "http://localhost:12345/api/v1/",
|
||||
"template": {},
|
||||
"version": "1.0"
|
||||
}
|
||||
}
|
||||
|
||||
Generating a token in Bash
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code::
|
||||
|
||||
$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
|
||||
sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
|
||||
| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
|
||||
|
||||
X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....
|
||||
|
||||
Generating a token in Python
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code:: python
|
||||
|
||||
#!/usr/bin/env python
|
||||
import os
|
||||
import gnupg
|
||||
from time import gmtime, strftime
|
||||
import random
|
||||
import requests
|
||||
import json
|
||||
|
||||
def makeToken(gpghome, keyid):
|
||||
gpg = gnupg.GPG(gnupghome=gpghome)
|
||||
version = "1"
|
||||
timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
|
||||
nonce = str(random.randint(10000, 18446744073709551616))
|
||||
token = version + ";" + timestamp + ";" + nonce
|
||||
sig = gpg.sign(token + "\n",
|
||||
keyid=keyid,
|
||||
detach=True, clearsign=True)
|
||||
token += ";"
|
||||
linectr=0
|
||||
for line in iter(str(sig).splitlines()):
|
||||
linectr+=1
|
||||
if linectr < 4 or line.startswith('-') or not line:
|
||||
continue
|
||||
token += line
|
||||
return token
|
||||
|
||||
if __name__ == '__main__':
|
||||
token = makeToken("/home/ulfr/.gnupg",
|
||||
"E60892BB9BD89A69F759A1A0A3D652173B763E8F")
|
||||
r = requests.get("http://localhost:12345/api/v1/dashboard",
|
||||
headers={'X-PGPAUTHORIZATION': token})
|
||||
print token
|
||||
print r.text
|
||||
|
||||
API endpoints
|
||||
-------------
|
||||
Endpoints
|
||||
---------
|
||||
|
||||
The API root is at `/api/v1` by defualt. All the endpoints described below are
|
||||
reachable behind the root. If you change the location of the API root, update
|
||||
|
|
@ -1125,3 +946,183 @@ Geolocations are returned as CLJS items in this format:
|
|||
|
||||
When using the parameter `&report=geolocations`, the `search` endpoint of the
|
||||
API will generate a list of geolocations from the results of the search.
|
||||
|
||||
Authentication with X-PGPAUTHORIZATION version 1
|
||||
------------------------------------------------
|
||||
|
||||
Authenticating against the MIG API requires sending a PGP signed token in the
|
||||
request header named `X-PGPAUTHORIZATION`. The key that signs the token must
|
||||
belong to an active investigator. Construction of the token works as follows:
|
||||
|
||||
1. make a string named **str** composed of a version, a UTC timestamp in RFC3339 format
|
||||
and a random nonce, each separated by semicolons. The current version is **1**
|
||||
and may be upgraded in the future. The nonce value must be a positive integer.
|
||||
|
||||
**str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE>**
|
||||
|
||||
UTC is a hard requirement. The timestamp must end with the suffix **Z**
|
||||
which indicates the UTC timezone. In bash, a correct timestamp can be
|
||||
generated with the command `$ date -u +%Y-%m-%dT%H:%M:%SZ`.
|
||||
|
||||
An example string would look like: `1;2006-01-02T15:04:05Z;1825922807490630059`
|
||||
|
||||
The string must be terminated by a newline character, hexadecimal code `0x0a`.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ hexdump -C <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
00000000 31 3b 32 30 30 36 2d 30 31 2d 30 32 54 31 35 3a |1;2006-01-02T15:|
|
||||
00000010 30 34 3a 30 35 5a 3b 31 38 32 35 39 32 32 38 30 |04:05Z;182592280|
|
||||
00000020 37 34 39 30 36 33 30 30 35 39 0a |7490630059.|
|
||||
0000002b
|
||||
|
||||
2. PGP sign **str** with the private key of the investigator. Armor and detach
|
||||
the signature into **armoredSig**::
|
||||
|
||||
$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
|
||||
JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
|
||||
kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
|
||||
b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
|
||||
PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
|
||||
MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
|
||||
=4nrR
|
||||
-----END PGP SIGNATURE-----
|
||||
|
||||
3. Create **sig** by taking **armoredSig** and removing the PGP headers, footers,
|
||||
empty lines and newlines.
|
||||
|
||||
example: `iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT`
|
||||
|
||||
4. Create **token** by concatenating **str**, a semicolon, and **sig**.
|
||||
**token=<str>;<sig>**
|
||||
example: `1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated>`
|
||||
|
||||
5. Send **token** in the header named **X-PGPAUTHORIZATION** with the request::
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/
|
||||
|
||||
6. The API verifies the version and validity period of the timestamp. By default, a
|
||||
token will be rejected if its timestamp deviates from the server time by more
|
||||
than 10 minutes. Administrators can configure this value. In effect, this
|
||||
means a timestamp is valid for twice the duration of the window. By default,
|
||||
that's 10 minutes before current server time, and 10 minutes after current
|
||||
server time.
|
||||
|
||||
7. If the timestamp is valid, the API next verifies the signature against the data
|
||||
and authenticates the user. Failure to verify the signature returns an error
|
||||
with the HTTP code 401 Unauthorized.
|
||||
|
||||
8. The user is authorized, the API processes and answer the request.
|
||||
|
||||
Security implications
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. A token can be used an unlimited number of times within its validity period.
|
||||
There is no check to guarantee that a token is only used once. It is
|
||||
assumed that the token is transmitted over a secure channel such as HTTPS to
|
||||
prevent token theft by a malicious user.
|
||||
|
||||
2. API clients and servers must use proper time synchronization for the timestamp
|
||||
verification to work. A client or a server that has inaccurate time may not be
|
||||
able to establish connections. We believe this requirement to be reasonable
|
||||
considering the sensitivity of the API.
|
||||
|
||||
Example 1: invalid timestamp
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The signature is valid but the timestamp is beyond the acceptable time window.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>' http://localhost:12345/api/v1/
|
||||
|
||||
{
|
||||
"collection": {
|
||||
"error": {
|
||||
"code": "6077873045059431424",
|
||||
"message": "Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"
|
||||
},
|
||||
"href": "http://localhost:12345/api/v1/",
|
||||
"template": {},
|
||||
"version": "1.0"
|
||||
}
|
||||
}
|
||||
|
||||
Example 2: invalid signature
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The signature is not valid, or is signed by a key that the API does not
|
||||
recognize.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ curl -H 'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>' http://localhost:12345/api/v1/
|
||||
|
||||
{
|
||||
"collection": {
|
||||
"error": {
|
||||
"code": "6077875007260332032",
|
||||
"message": "Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"
|
||||
},
|
||||
"href": "http://localhost:12345/api/v1/",
|
||||
"template": {},
|
||||
"version": "1.0"
|
||||
}
|
||||
}
|
||||
|
||||
Generating a token in Bash
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code::
|
||||
|
||||
$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
|
||||
sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
|
||||
| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
|
||||
|
||||
X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....
|
||||
|
||||
Generating a token in Python
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code:: python
|
||||
|
||||
#!/usr/bin/env python
|
||||
import os
|
||||
import gnupg
|
||||
from time import gmtime, strftime
|
||||
import random
|
||||
import requests
|
||||
import json
|
||||
|
||||
def makeToken(gpghome, keyid):
|
||||
gpg = gnupg.GPG(gnupghome=gpghome)
|
||||
version = "1"
|
||||
timestamp = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
|
||||
nonce = str(random.randint(10000, 18446744073709551616))
|
||||
token = version + ";" + timestamp + ";" + nonce
|
||||
sig = gpg.sign(token + "\n",
|
||||
keyid=keyid,
|
||||
detach=True, clearsign=True)
|
||||
token += ";"
|
||||
linectr=0
|
||||
for line in iter(str(sig).splitlines()):
|
||||
linectr+=1
|
||||
if linectr < 4 or line.startswith('-') or not line:
|
||||
continue
|
||||
token += line
|
||||
return token
|
||||
|
||||
if __name__ == '__main__':
|
||||
token = makeToken("/home/ulfr/.gnupg",
|
||||
"E60892BB9BD89A69F759A1A0A3D652173B763E8F")
|
||||
r = requests.get("http://localhost:12345/api/v1/dashboard",
|
||||
headers={'X-PGPAUTHORIZATION': token})
|
||||
print token
|
||||
print r.text
|
||||
|
||||
|
||||
|
|
|
|||
236
doc/api.rst.html
236
doc/api.rst.html
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
@ -227,113 +229,13 @@ cite {
|
|||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG API</h1><table><tr><td class="field-label">Author</td><td>Julien Vehent <<a class="reference external" href="mailto:jvehent@mozilla.com">jvehent@mozilla.com</a>></td></tr></table><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#authentication-with-x-pgpauthorization-version-1" id="id1">1 Authentication with X-PGPAUTHORIZATION version 1</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#security-implications" id="id2">1.1 Security implications</a></p></li><li><p><a class="reference internal" href="#example-1-invalid-timestamp" id="id3">1.2 Example 1: invalid timestamp</a></p></li><li><p><a class="reference internal" href="#example-2-invalid-signature" id="id4">1.3 Example 2: invalid signature</a></p></li><li><p><a class="reference internal" href="#generating-a-token-in-bash" id="id5">1.4 Generating a token in Bash</a></p></li><li><p><a class="reference internal" href="#generating-a-token-in-python" id="id6">1.5 Generating a token in Python</a></p></li></ul></li><li><p><a class="reference internal" href="#api-endpoints" id="id7">2 API endpoints</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#get-api-v1-heartbeat" id="id8">2.1 GET /api/v1/heartbeat</a></p></li><li><p><a class="reference internal" href="#get-api-v1-ip" id="id9">2.2 GET /api/v1/ip</a></p></li><li><p><a class="reference internal" href="#get-api-v1-dashboard" id="id10">2.3 GET /api/v1/dashboard</a></p></li><li><p><a class="reference internal" href="#get-api-v1-action" id="id11">2.4 GET /api/v1/action</a></p></li><li><p><a class="reference internal" href="#post-api-v1-action-create" id="id12">2.5 POST /api/v1/action/create/</a></p></li><li><p><a class="reference internal" href="#get-api-v1-agent" id="id13">2.6 GET /api/v1/agent</a></p></li><li><p><a class="reference internal" href="#get-api-v1-command" id="id14">2.7 GET /api/v1/command</a></p></li><li><p><a class="reference internal" href="#get-api-v1-investigator" id="id15">2.8 GET /api/v1/investigator</a></p></li><li><p><a class="reference internal" href="#post-api-v1-investigator-create" id="id16">2.9 POST /api/v1/investigator/create/</a></p></li><li><p><a class="reference internal" href="#post-api-v1-investigator-update" id="id17">2.10 POST /api/v1/investigator/update/</a></p></li><li><p><a class="reference internal" href="#get-api-v1-search" id="id18">2.11 GET /api/v1/search</a></p></li></ul></li><li><p><a class="reference internal" href="#data-transformation" id="id19">3 Data transformation</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#compliance-items" id="id20">3.1 Compliance Items</a></p></li><li><p><a class="reference internal" href="#geolocations" id="id21">3.2 Geolocations</a></p></li></ul></li></ul></div><p>Interactions between an investigator (a human being) and the MIG platform are
|
||||
</style></head><body><h1>MIG API</h1><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#endpoints" id="id1">1 Endpoints</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#get-api-v1-heartbeat" id="id2">1.1 GET /api/v1/heartbeat</a></p></li><li><p><a class="reference internal" href="#get-api-v1-ip" id="id3">1.2 GET /api/v1/ip</a></p></li><li><p><a class="reference internal" href="#get-api-v1-dashboard" id="id4">1.3 GET /api/v1/dashboard</a></p></li><li><p><a class="reference internal" href="#get-api-v1-action" id="id5">1.4 GET /api/v1/action</a></p></li><li><p><a class="reference internal" href="#post-api-v1-action-create" id="id6">1.5 POST /api/v1/action/create/</a></p></li><li><p><a class="reference internal" href="#get-api-v1-agent" id="id7">1.6 GET /api/v1/agent</a></p></li><li><p><a class="reference internal" href="#get-api-v1-command" id="id8">1.7 GET /api/v1/command</a></p></li><li><p><a class="reference internal" href="#get-api-v1-investigator" id="id9">1.8 GET /api/v1/investigator</a></p></li><li><p><a class="reference internal" href="#post-api-v1-investigator-create" id="id10">1.9 POST /api/v1/investigator/create/</a></p></li><li><p><a class="reference internal" href="#post-api-v1-investigator-update" id="id11">1.10 POST /api/v1/investigator/update/</a></p></li><li><p><a class="reference internal" href="#get-api-v1-search" id="id12">1.11 GET /api/v1/search</a></p></li></ul></li><li><p><a class="reference internal" href="#data-transformation" id="id13">2 Data transformation</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#compliance-items" id="id14">2.1 Compliance Items</a></p></li><li><p><a class="reference internal" href="#geolocations" id="id15">2.2 Geolocations</a></p></li></ul></li><li><p><a class="reference internal" href="#authentication-with-x-pgpauthorization-version-1" id="id16">3 Authentication with X-PGPAUTHORIZATION version 1</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#security-implications" id="id17">3.1 Security implications</a></p></li><li><p><a class="reference internal" href="#example-1-invalid-timestamp" id="id18">3.2 Example 1: invalid timestamp</a></p></li><li><p><a class="reference internal" href="#example-2-invalid-signature" id="id19">3.3 Example 2: invalid signature</a></p></li><li><p><a class="reference internal" href="#generating-a-token-in-bash" id="id20">3.4 Generating a token in Bash</a></p></li><li><p><a class="reference internal" href="#generating-a-token-in-python" id="id21">3.5 Generating a token in Python</a></p></li></ul></li></ul></div><p>Interactions between an investigator (a human being) and the MIG platform are
|
||||
performed through a REST API. The API exposes functions to create actions,
|
||||
retrieve results, and generally monitor the activity of the agents.</p><p>The API follows the core principles of REST, and provides discoverable
|
||||
endpoints. API responses follows the <strong>cljs</strong> format defined in
|
||||
<a class="reference external" href="http://amundsen.com/media-types/collection/">Collection+JSON - Hypermedia Type</a>.</p><section id="authentication-with-x-pgpauthorization-version-1"><header><h2><a href="#id1">1 Authentication with X-PGPAUTHORIZATION version 1</a></h2></header><p>Authenticating against the MIG API requires sending a PGP signed token in the
|
||||
request header named <cite>X-PGPAUTHORIZATION</cite>. The key that signs the token must
|
||||
belong to an active investigator. Construction of the token works as follows:</p><ol class="arabic"><li><p>make a string named <strong>str</strong> composed of a version, a UTC timestamp in RFC3339 format
|
||||
and a random nonce, each separated by semicolons. The current version is <strong>1</strong>
|
||||
and may be upgraded in the future. The nonce value must be a positive integer.</p><p><strong>str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE></strong></p><p>UTC is a hard requirement. The timestamp must end with the suffix <strong>Z</strong>
|
||||
which indicates the UTC timezone. In bash, a correct timestamp can be
|
||||
generated with the command <cite>$ date -u +%Y-%m-%dT%H:%M:%SZ</cite>.</p><p>An example string would look like: <cite>1;2006-01-02T15:04:05Z;1825922807490630059</cite></p><p>The string must be terminated by a newline character, hexadecimal code <cite>0x0a</cite>.</p></li></ol><pre><code class="code bash"><span class="name variable">$ </span>hexdump -C <span class="operator"><<<</span> <span class="literal string single">'1;2006-01-02T15:04:05Z;1825922807490630059'</span>
|
||||
<span class="literal number">00000000</span> <span class="literal number">31</span> 3b <span class="literal number">32</span> <span class="literal number">30</span> <span class="literal number">30</span> <span class="literal number">36</span> 2d <span class="literal number">30</span> <span class="literal number">31</span> 2d <span class="literal number">30</span> <span class="literal number">32</span> <span class="literal number">54</span> <span class="literal number">31</span> <span class="literal number">35</span> 3a <span class="punctuation">|</span>1<span class="punctuation">;</span>2006-01-02T15:<span class="punctuation">|</span>
|
||||
<span class="literal number">00000010</span> <span class="literal number">30</span> <span class="literal number">34</span> 3a <span class="literal number">30</span> <span class="literal number">35</span> 5a 3b <span class="literal number">31</span> <span class="literal number">38</span> <span class="literal number">32</span> <span class="literal number">35</span> <span class="literal number">39</span> <span class="literal number">32</span> <span class="literal number">32</span> <span class="literal number">38</span> <span class="literal number">30</span> <span class="punctuation">|</span>04:05Z<span class="punctuation">;</span>182592280<span class="punctuation">|</span>
|
||||
<span class="literal number">00000020</span> <span class="literal number">37</span> <span class="literal number">34</span> <span class="literal number">39</span> <span class="literal number">30</span> <span class="literal number">36</span> <span class="literal number">33</span> <span class="literal number">30</span> <span class="literal number">30</span> <span class="literal number">35</span> <span class="literal number">39</span> 0a <span class="punctuation">|</span>7490630059.<span class="punctuation">|</span>
|
||||
0000002b</code></pre><ol class="arabic" start="2"><li><p>PGP sign <strong>str</strong> with the private key of the investigator. Armor and detach
|
||||
the signature into <strong>armoredSig</strong>:</p><pre>$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
|
||||
JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
|
||||
kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
|
||||
b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
|
||||
PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
|
||||
MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
|
||||
=4nrR
|
||||
-----END PGP SIGNATURE-----</pre></li><li><p>Create <strong>sig</strong> by taking <strong>armoredSig</strong> and removing the PGP headers, footers,
|
||||
empty lines and newlines.</p><blockquote><p>example: <cite>iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT</cite></p></blockquote></li><li><p>Create <strong>token</strong> by concatenating <strong>str</strong>, a semicolon, and <strong>sig</strong>.
|
||||
<strong>token=<str>;<sig></strong>
|
||||
example: <cite>1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated></cite></p></li><li><p>Send <strong>token</strong> in the header named <strong>X-PGPAUTHORIZATION</strong> with the request:</p><pre>$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/</pre></li><li><p>The API verifies the version and validity period of the timestamp. By default, a
|
||||
token will be rejected if its timestamp deviates from the server time by more
|
||||
than 10 minutes. Administrators can configure this value. In effect, this
|
||||
means a timestamp is valid for twice the duration of the window. By default,
|
||||
that's 10 minutes before current server time, and 10 minutes after current
|
||||
server time.</p></li><li><p>If the timestamp is valid, the API next verifies the signature against the data
|
||||
and authenticates the user. Failure to verify the signature returns an error
|
||||
with the HTTP code 401 Unauthorized.</p></li><li><p>The user is authorized, the API processes and answer the request.</p></li></ol><section id="security-implications"><header><h3><a href="#id2">1.1 Security implications</a></h3></header><ol class="arabic"><li><p>A token can be used an unlimited number of times within its validity period.
|
||||
There is no check to guarantee that a token is only used once. It is
|
||||
assumed that the token is transmitted over a secure channel such as HTTPS to
|
||||
prevent token theft by a malicious user.</p></li><li><p>API clients and servers must use proper time synchronization for the timestamp
|
||||
verification to work. A client or a server that has inaccurate time may not be
|
||||
able to establish connections. We believe this requirement to be reasonable
|
||||
considering the sensitivity of the API.</p></li></ol></section><section id="example-1-invalid-timestamp"><header><h3><a href="#id3">1.2 Example 1: invalid timestamp</a></h3></header><p>The signature is valid but the timestamp is beyond the acceptable time window.</p><pre><code class="code bash"><span class="name variable">$ </span>curl -H <span class="literal string single">'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>'</span> http://localhost:12345/api/v1/
|
||||
|
||||
<span class="operator">{</span>
|
||||
<span class="literal string double">"collection"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"error"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"code"</span>: <span class="literal string double">"6077873045059431424"</span>,
|
||||
<span class="literal string double">"message"</span>: <span class="literal string double">"Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"</span>
|
||||
<span class="operator">}</span>,
|
||||
<span class="literal string double">"href"</span>: <span class="literal string double">"http://localhost:12345/api/v1/"</span>,
|
||||
<span class="literal string double">"template"</span>: <span class="operator">{}</span>,
|
||||
<span class="literal string double">"version"</span>: <span class="literal string double">"1.0"</span>
|
||||
<span class="operator">}</span>
|
||||
<span class="operator">}</span></code></pre></section><section id="example-2-invalid-signature"><header><h3><a href="#id4">1.3 Example 2: invalid signature</a></h3></header><p>The signature is not valid, or is signed by a key that the API does not
|
||||
recognize.</p><pre><code class="code bash"><span class="name variable">$ </span>curl -H <span class="literal string single">'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>'</span> http://localhost:12345/api/v1/
|
||||
|
||||
<span class="operator">{</span>
|
||||
<span class="literal string double">"collection"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"error"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"code"</span>: <span class="literal string double">"6077875007260332032"</span>,
|
||||
<span class="literal string double">"message"</span>: <span class="literal string double">"Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"</span>
|
||||
<span class="operator">}</span>,
|
||||
<span class="literal string double">"href"</span>: <span class="literal string double">"http://localhost:12345/api/v1/"</span>,
|
||||
<span class="literal string double">"template"</span>: <span class="operator">{}</span>,
|
||||
<span class="literal string double">"version"</span>: <span class="literal string double">"1.0"</span>
|
||||
<span class="operator">}</span>
|
||||
<span class="operator">}</span></code></pre></section><section id="generating-a-token-in-bash"><header><h3><a href="#id5">1.4 Generating a token in Bash</a></h3></header><pre><code class="code">$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
|
||||
sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
|
||||
| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
|
||||
|
||||
X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....</code></pre></section><section id="generating-a-token-in-python"><header><h3><a href="#id6">1.5 Generating a token in Python</a></h3></header><pre><code class="code python"><span class="comment">#!/usr/bin/env python</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">os</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">gnupg</span>
|
||||
<span class="keyword namespace">from</span> <span class="name namespace">time</span> <span class="keyword namespace">import</span> <span class="name">gmtime</span><span class="punctuation">,</span> <span class="name">strftime</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">random</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">requests</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">json</span>
|
||||
|
||||
<span class="keyword">def</span> <span class="name function">makeToken</span><span class="punctuation">(</span><span class="name">gpghome</span><span class="punctuation">,</span> <span class="name">keyid</span><span class="punctuation">):</span>
|
||||
<span class="name">gpg</span> <span class="operator">=</span> <span class="name">gnupg</span><span class="operator">.</span><span class="name">GPG</span><span class="punctuation">(</span><span class="name">gnupghome</span><span class="operator">=</span><span class="name">gpghome</span><span class="punctuation">)</span>
|
||||
<span class="name">version</span> <span class="operator">=</span> <span class="literal string">"1"</span>
|
||||
<span class="name">timestamp</span> <span class="operator">=</span> <span class="name">strftime</span><span class="punctuation">(</span><span class="literal string">"%Y-%m-</span><span class="literal string interpol">%d</span><span class="literal string">T%H:%M:%SZ"</span><span class="punctuation">,</span> <span class="name">gmtime</span><span class="punctuation">())</span>
|
||||
<span class="name">nonce</span> <span class="operator">=</span> <span class="name builtin">str</span><span class="punctuation">(</span><span class="name">random</span><span class="operator">.</span><span class="name">randint</span><span class="punctuation">(</span><span class="literal number integer">10000</span><span class="punctuation">,</span> <span class="literal number integer">18446744073709551616</span><span class="punctuation">))</span>
|
||||
<span class="name">token</span> <span class="operator">=</span> <span class="name">version</span> <span class="operator">+</span> <span class="literal string">";"</span> <span class="operator">+</span> <span class="name">timestamp</span> <span class="operator">+</span> <span class="literal string">";"</span> <span class="operator">+</span> <span class="name">nonce</span>
|
||||
<span class="name">sig</span> <span class="operator">=</span> <span class="name">gpg</span><span class="operator">.</span><span class="name">sign</span><span class="punctuation">(</span><span class="name">token</span> <span class="operator">+</span> <span class="literal string">"</span><span class="literal string escape">\n</span><span class="literal string">"</span><span class="punctuation">,</span>
|
||||
<span class="name">keyid</span><span class="operator">=</span><span class="name">keyid</span><span class="punctuation">,</span>
|
||||
<span class="name">detach</span><span class="operator">=</span><span class="name builtin pseudo">True</span><span class="punctuation">,</span> <span class="name">clearsign</span><span class="operator">=</span><span class="name builtin pseudo">True</span><span class="punctuation">)</span>
|
||||
<span class="name">token</span> <span class="operator">+=</span> <span class="literal string">";"</span>
|
||||
<span class="name">linectr</span><span class="operator">=</span><span class="literal number integer">0</span>
|
||||
<span class="keyword">for</span> <span class="name">line</span> <span class="operator word">in</span> <span class="name builtin">iter</span><span class="punctuation">(</span><span class="name builtin">str</span><span class="punctuation">(</span><span class="name">sig</span><span class="punctuation">)</span><span class="operator">.</span><span class="name">splitlines</span><span class="punctuation">()):</span>
|
||||
<span class="name">linectr</span><span class="operator">+=</span><span class="literal number integer">1</span>
|
||||
<span class="keyword">if</span> <span class="name">linectr</span> <span class="operator"><</span> <span class="literal number integer">4</span> <span class="operator word">or</span> <span class="name">line</span><span class="operator">.</span><span class="name">startswith</span><span class="punctuation">(</span><span class="literal string">'-'</span><span class="punctuation">)</span> <span class="operator word">or</span> <span class="operator word">not</span> <span class="name">line</span><span class="punctuation">:</span>
|
||||
<span class="keyword">continue</span>
|
||||
<span class="name">token</span> <span class="operator">+=</span> <span class="name">line</span>
|
||||
<span class="keyword">return</span> <span class="name">token</span>
|
||||
|
||||
<span class="keyword">if</span> <span class="name">__name__</span> <span class="operator">==</span> <span class="literal string">'__main__'</span><span class="punctuation">:</span>
|
||||
<span class="name">token</span> <span class="operator">=</span> <span class="name">makeToken</span><span class="punctuation">(</span><span class="literal string">"/home/ulfr/.gnupg"</span><span class="punctuation">,</span>
|
||||
<span class="literal string">"E60892BB9BD89A69F759A1A0A3D652173B763E8F"</span><span class="punctuation">)</span>
|
||||
<span class="name">r</span> <span class="operator">=</span> <span class="name">requests</span><span class="operator">.</span><span class="name">get</span><span class="punctuation">(</span><span class="literal string">"http://localhost:12345/api/v1/dashboard"</span><span class="punctuation">,</span>
|
||||
<span class="name">headers</span><span class="operator">=</span><span class="punctuation">{</span><span class="literal string">'X-PGPAUTHORIZATION'</span><span class="punctuation">:</span> <span class="name">token</span><span class="punctuation">})</span>
|
||||
<span class="keyword">print</span> <span class="name">token</span>
|
||||
<span class="keyword">print</span> <span class="name">r</span><span class="operator">.</span><span class="name">text</span></code></pre></section></section><section id="api-endpoints"><header><h2><a href="#id7">2 API endpoints</a></h2></header><p>The API root is at <cite>/api/v1</cite> by defualt. All the endpoints described below are
|
||||
<a class="reference external" href="http://amundsen.com/media-types/collection/">Collection+JSON - Hypermedia Type</a>.</p><section id="endpoints"><header><h2><a href="#id1">1 Endpoints</a></h2></header><p>The API root is at <cite>/api/v1</cite> by defualt. All the endpoints described below are
|
||||
reachable behind the root. If you change the location of the API root, update
|
||||
the query paths accordingly.</p><section id="get-api-v1-heartbeat"><header><h3><a href="#id8">2.1 GET /api/v1/heartbeat</a></h3></header><ul><li><p>Description: basic endpoint that returns a HTTP 200</p></li><li><p>Parameters: none</p></li><li><p>Authentication: none</p></li><li><p>Response Code: 200 OK</p></li><li><p>Reponse: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
the query paths accordingly.</p><section id="get-api-v1-heartbeat"><header><h3><a href="#id2">1.1 GET /api/v1/heartbeat</a></h3></header><ul><li><p>Description: basic endpoint that returns a HTTP 200</p></li><li><p>Parameters: none</p></li><li><p>Authentication: none</p></li><li><p>Response Code: 200 OK</p></li><li><p>Reponse: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"collection"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"error"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"href"</span><span class="punctuation">:</span> <span class="literal string double">"https://api.mig.mozilla.org/api/v1/heartbeat"</span><span class="punctuation">,</span>
|
||||
|
|
@ -351,9 +253,9 @@ the query paths accordingly.</p><section id="get-api-v1-heartbeat"><header><h3><
|
|||
<span class="name tag">"template"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"1.0"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-ip"><header><h3><a href="#id9">2.2 GET /api/v1/ip</a></h3></header><ul><li><p>Description: basic endpoint that returns the public IP of the caller. If the
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-ip"><header><h3><a href="#id3">1.2 GET /api/v1/ip</a></h3></header><ul><li><p>Description: basic endpoint that returns the public IP of the caller. If the
|
||||
API is behind a load balancer, it returns the value of X-Forwarded-For.</p></li><li><p>Parameters: none</p></li><li><p>Authentication: none</p></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Text</p></li></ul><pre><code class="code bash"><span class="name variable">$ </span>curl https://api.mig.mozilla.org/api/v1/ip
|
||||
108.36.248.44</code></pre></section><section id="get-api-v1-dashboard"><header><h3><a href="#id10">2.3 GET /api/v1/dashboard</a></h3></header><ul><li><p>Description: returns a status dashboard with counters of active and idle
|
||||
108.36.248.44</code></pre></section><section id="get-api-v1-dashboard"><header><h3><a href="#id4">1.3 GET /api/v1/dashboard</a></h3></header><ul><li><p>Description: returns a status dashboard with counters of active and idle
|
||||
agents, and a list of the last 10 actions ran.</p></li><li><p>Parameters: none</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"collection"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"error"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
|
|
@ -526,7 +428,7 @@ agents, and a list of the last 10 actions ran.</p></li><li><p>Parameters: none</
|
|||
<span class="name tag">"template"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"1.0"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-action"><header><h3><a href="#id11">2.4 GET /api/v1/action</a></h3></header><ul><li><p>Description: retrieve an action by its ID. Include links to related commands.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>actionid</cite>: a uint64 that identifies an action by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-action"><header><h3><a href="#id5">1.4 GET /api/v1/action</a></h3></header><ul><li><p>Description: retrieve an action by its ID. Include links to related commands.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>actionid</cite>: a uint64 that identifies an action by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"collection"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"error"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"href"</span><span class="punctuation">:</span> <span class="literal string double">"https://api.mig.example.net/api/v1/action?actionid=6115472790658567168"</span><span class="punctuation">,</span>
|
||||
|
|
@ -639,7 +541,7 @@ agents, and a list of the last 10 actions ran.</p></li><li><p>Parameters: none</
|
|||
<span class="name tag">"template"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"1.0"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="post-api-v1-action-create"><header><h3><a href="#id12">2.5 POST /api/v1/action/create/</a></h3></header><ul><li><p>Description: send a signed action to the API for submission to the scheduler.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>action</cite>: a signed action in JSON format</p></li></ul></dd></dl></li><li><p>Response Code: 202 Accepted</p></li><li><p>Response: Collection+JSON</p></li></ul></section><section id="get-api-v1-agent"><header><h3><a href="#id13">2.6 GET /api/v1/agent</a></h3></header><ul><li><p>Description: retrieve an agent by its ID</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>agentid</cite>: a uint64 that identifies an agent by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="post-api-v1-action-create"><header><h3><a href="#id6">1.5 POST /api/v1/action/create/</a></h3></header><ul><li><p>Description: send a signed action to the API for submission to the scheduler.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>action</cite>: a signed action in JSON format</p></li></ul></dd></dl></li><li><p>Response Code: 202 Accepted</p></li><li><p>Response: Collection+JSON</p></li></ul></section><section id="get-api-v1-agent"><header><h3><a href="#id7">1.6 GET /api/v1/agent</a></h3></header><ul><li><p>Description: retrieve an agent by its ID</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>agentid</cite>: a uint64 that identifies an agent by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"collection"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"error"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"href"</span><span class="punctuation">:</span> <span class="literal string double">"https://api.mig.example.net/api/v1/agent?agentid=1423779015943326976"</span><span class="punctuation">,</span>
|
||||
|
|
@ -681,7 +583,7 @@ agents, and a list of the last 10 actions ran.</p></li><li><p>Parameters: none</
|
|||
<span class="name tag">"template"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"1.0"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-command"><header><h3><a href="#id14">2.7 GET /api/v1/command</a></h3></header><ul><li><p>Description: retrieve a command by its ID. Include link to related action.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>commandid</cite>: a uint64 that identifies a command by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code bash"><span class="operator">{</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="get-api-v1-command"><header><h3><a href="#id8">1.7 GET /api/v1/command</a></h3></header><ul><li><p>Description: retrieve a command by its ID. Include link to related action.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>commandid</cite>: a uint64 that identifies a command by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code bash"><span class="operator">{</span>
|
||||
<span class="literal string double">"collection"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"error"</span>: <span class="operator">{}</span>,
|
||||
<span class="literal string double">"href"</span>: <span class="literal string double">"https://api.mig.example.net/api/v1/command?commandid=1424700180901330688"</span>,
|
||||
|
|
@ -896,7 +798,7 @@ agents, and a list of the last 10 actions ran.</p></li><li><p>Parameters: none</
|
|||
"</span>template<span class="literal string double">": {},
|
||||
"</span>version<span class="literal string double">": "</span>1.0<span class="literal string double">"
|
||||
}
|
||||
}</span></code></pre></section><section id="get-api-v1-investigator"><header><h3><a href="#id15">2.8 GET /api/v1/investigator</a></h3></header><ul><li><p>Description: retrieve an investigator by its ID. Include link to the
|
||||
}</span></code></pre></section><section id="get-api-v1-investigator"><header><h3><a href="#id9">1.8 GET /api/v1/investigator</a></h3></header><ul><li><p>Description: retrieve an investigator by its ID. Include link to the
|
||||
investigator's action history.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>investigatorid</cite>: a uint64 that identifies a command by its ID</p></li></ul></dd></dl></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li></ul><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"collection"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"error"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
|
|
@ -929,8 +831,8 @@ investigator's action history.</p></li><li><p>Authentication: X-PGPAUTHORIZATION
|
|||
<span class="name tag">"template"</span><span class="punctuation">:</span> <span class="punctuation">{},</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"1.0"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="post-api-v1-investigator-create"><header><h3><a href="#id16">2.9 POST /api/v1/investigator/create/</a></h3></header><ul><li><p>Description: create a new investigator in the database</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>name</cite>: string that represents the full name</p></li><li><p><cite>publickey</cite>: armored GPG public key</p></li></ul></dd></dl></li><li><p>Response Code: 201 Created</p></li><li><p>Response: Collection+JSON</p></li><li><p>Example: (without authentication)</p></li></ul><pre><code class="code bash"><span class="name variable">$ </span>gpg --export -a --export-options <span class="name builtin">export</span>-minimal bob_kelso@example.net > /tmp/bobpubkey
|
||||
<span class="name variable">$ </span>curl -iv -F <span class="literal string double">"name=Bob Kelso"</span> -F <span class="name variable">publickey</span><span class="operator">=</span>@/tmp/pubkey https://api.mig.example.net/api/v1/investigator/create/</code></pre></section><section id="post-api-v1-investigator-update"><header><h3><a href="#id17">2.10 POST /api/v1/investigator/update/</a></h3></header><ul><li><p>Description: update an existing investigator in the database</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>id</cite>: investigator id, to identify the target investigator</p></li><li><p><cite>status</cite>: new status of the investigator, to be updated</p></li></ul></dd></dl></li><li><p>Response Code: 201 Created</p></li><li><p>Response: Collection+JSON</p></li><li><p>Example: (without authentication)</p></li></ul><pre><code class="code bash"><span class="name variable">$ </span>curl -iv -X POST -d <span class="name variable">id</span><span class="operator">=</span><span class="literal number">1234</span> -d <span class="name variable">status</span><span class="operator">=</span>disabled https://api.mig.example.net/api/v1/investigator/update/</code></pre></section><section id="get-api-v1-search"><header><h3><a href="#id18">2.11 GET /api/v1/search</a></h3></header><ul><li><p>Description: search for actions, commands, agents or investigators.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>type</cite>: define the type of item returned by the search.
|
||||
<span class="punctuation">}</span></code></pre></section><section id="post-api-v1-investigator-create"><header><h3><a href="#id10">1.9 POST /api/v1/investigator/create/</a></h3></header><ul><li><p>Description: create a new investigator in the database</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>name</cite>: string that represents the full name</p></li><li><p><cite>publickey</cite>: armored GPG public key</p></li></ul></dd></dl></li><li><p>Response Code: 201 Created</p></li><li><p>Response: Collection+JSON</p></li><li><p>Example: (without authentication)</p></li></ul><pre><code class="code bash"><span class="name variable">$ </span>gpg --export -a --export-options <span class="name builtin">export</span>-minimal bob_kelso@example.net > /tmp/bobpubkey
|
||||
<span class="name variable">$ </span>curl -iv -F <span class="literal string double">"name=Bob Kelso"</span> -F <span class="name variable">publickey</span><span class="operator">=</span>@/tmp/pubkey https://api.mig.example.net/api/v1/investigator/create/</code></pre></section><section id="post-api-v1-investigator-update"><header><h3><a href="#id11">1.10 POST /api/v1/investigator/update/</a></h3></header><ul><li><p>Description: update an existing investigator in the database</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><dl><dt>Parameters: (POST body)</dt><dd><ul><li><p><cite>id</cite>: investigator id, to identify the target investigator</p></li><li><p><cite>status</cite>: new status of the investigator, to be updated</p></li></ul></dd></dl></li><li><p>Response Code: 201 Created</p></li><li><p>Response: Collection+JSON</p></li><li><p>Example: (without authentication)</p></li></ul><pre><code class="code bash"><span class="name variable">$ </span>curl -iv -X POST -d <span class="name variable">id</span><span class="operator">=</span><span class="literal number">1234</span> -d <span class="name variable">status</span><span class="operator">=</span>disabled https://api.mig.example.net/api/v1/investigator/update/</code></pre></section><section id="get-api-v1-search"><header><h3><a href="#id12">1.11 GET /api/v1/search</a></h3></header><ul><li><p>Description: search for actions, commands, agents or investigators.</p></li><li><p>Authentication: X-PGPAUTHORIZATION</p></li><li><p>Response Code: 200 OK</p></li><li><p>Response: Collection+JSON</p></li><li><dl><dt>Parameters:</dt><dd><ul><li><p><cite>type</cite>: define the type of item returned by the search.
|
||||
Valid types are: <cite>action</cite>, <cite>command</cite>, <cite>agent</cite> or <cite>investigator</cite>.</p><blockquote><ul><li><p><cite>action</cite>: (default) return a list of actions</p></li><li><p><cite>command</cite>: return a list of commands</p></li><li><p><cite>agent</cite>: return a list of agents that have shown activity</p></li><li><p><cite>investigator</cite>: return a list of investigators that have show activity</p></li></ul></blockquote></li><li><p><cite>actionid</cite>: filter results on numeric action ID</p></li><li><p><cite>actionname</cite>: filter results on string action name, accept <cite>ILIKE</cite> pattern</p></li><li><p><cite>after</cite>: return results recorded after this RFC3339 date. If not set,
|
||||
return results for last 10 years. Impact on search depends on the type:</p><blockquote><ul><li><p><cite>action</cite>: select actions with a <cite>validfrom</cite> date greater than <cite>after</cite>.</p></li><li><p><cite>agent</cite>: select agents that have sent a heartbeat since <cite>after</cite>.</p></li><li><p><cite>command</cite>: select commands with a <cite>starttime</cite> date greated than <cite>after</cite>.</p></li><li><p><cite>investigator</cite>: select investigators with a <cite>createdat</cite> date greater
|
||||
than <cite>after</cite>.</p></li></ul></blockquote></li><li><p><cite>agentid</cite>: filter results on the agent ID</p></li><li><p><cite>agentname</cite>: filter results on string agent name, accept <cite>ILIKE</cite> pattern</p></li><li><p><cite>agentversion</cite>: filter results on agent version string, accept <cite>ILIKE</cite> pattern</p></li><li><p><cite>before</cite>: return results recorded before this RFC3339 date. If not set,
|
||||
|
|
@ -948,8 +850,8 @@ parameters, the value is used as a SQL <cite>ILIKE</cite> search pattern, as des
|
|||
hours. For more information on the <cite>compliance</cite> format, see section 2.</p><pre><code class="code bash">/api/v1/search?type<span class="operator">=</span><span class="name builtin">command</span><span class="punctuation">&</span><span class="name variable">threatfamily</span><span class="operator">=</span>compliance<span class="punctuation">&</span><span class="name variable">status</span><span class="operator">=</span><span class="keyword">done</span>
|
||||
<span class="punctuation">&</span><span class="name variable">report</span><span class="operator">=</span>complianceitems<span class="punctuation">&</span><span class="name variable">limit</span><span class="operator">=</span>100000
|
||||
<span class="punctuation">&</span><span class="name variable">after</span><span class="operator">=</span>2014-05-30T00:00:00-04:00<span class="punctuation">&</span><span class="name variable">before</span><span class="operator">=</span>2014-05-30T23:59:59-04:00</code></pre><p>List the agents that have sent a heartbeat in the last hour.</p><pre><code class="code bash">/api/v1/search?type<span class="operator">=</span>agent<span class="punctuation">&</span><span class="name variable">after</span><span class="operator">=</span>2014-05-30T15:00:00-04:00<span class="punctuation">&</span><span class="name variable">limit</span><span class="operator">=</span>200</code></pre><p>Find actions ran between two dates (limited to 10 results as is the default).</p><pre><code class="code bash">/api/v1/search?type<span class="operator">=</span>action<span class="punctuation">&</span><span class="name variable">status</span><span class="operator">=</span>sent
|
||||
<span class="punctuation">&</span><span class="name variable">after</span><span class="operator">=</span>2014-05-01T00:00:00-00:00<span class="punctuation">&</span><span class="name variable">before</span><span class="operator">=</span>2014-05-30T00:00:00-00:00</code></pre><p>Find the last 10 commands signed by an investigator identified by name.</p><pre><code class="code bash">/api/v1/search?investigatorname<span class="operator">=</span>%25bob%25smith%25<span class="punctuation">&</span><span class="name variable">limit</span><span class="operator">=</span>10<span class="punctuation">&</span><span class="name builtin">type</span><span class="operator">=</span><span class="name builtin">command</span></code></pre></section></section><section id="data-transformation"><header><h2><a href="#id19">3 Data transformation</a></h2></header><p>The API implements several data transformation functions between the base
|
||||
format of <cite>action</cite> and <cite>command</cite>, and reporting formats.</p><section id="compliance-items"><header><h3><a href="#id20">3.1 Compliance Items</a></h3></header><p>The compliance item format is used to measure the compliance of a target with
|
||||
<span class="punctuation">&</span><span class="name variable">after</span><span class="operator">=</span>2014-05-01T00:00:00-00:00<span class="punctuation">&</span><span class="name variable">before</span><span class="operator">=</span>2014-05-30T00:00:00-00:00</code></pre><p>Find the last 10 commands signed by an investigator identified by name.</p><pre><code class="code bash">/api/v1/search?investigatorname<span class="operator">=</span>%25bob%25smith%25<span class="punctuation">&</span><span class="name variable">limit</span><span class="operator">=</span>10<span class="punctuation">&</span><span class="name builtin">type</span><span class="operator">=</span><span class="name builtin">command</span></code></pre></section></section><section id="data-transformation"><header><h2><a href="#id13">2 Data transformation</a></h2></header><p>The API implements several data transformation functions between the base
|
||||
format of <cite>action</cite> and <cite>command</cite>, and reporting formats.</p><section id="compliance-items"><header><h3><a href="#id14">2.1 Compliance Items</a></h3></header><p>The compliance item format is used to measure the compliance of a target with
|
||||
particular requirement. A single compliance item represent the compliance of
|
||||
one target (host) with one check (test + value).</p><p>In MIG, an <cite>action</cite> can contain compliance checks. An <cite>action</cite> creates one
|
||||
<cite>command</cite> per <cite>agent</cite>. Upon completion, the agent stores the results in the
|
||||
|
|
@ -980,7 +882,7 @@ commands could, in turn, generate thousands of compliance items.</p><p>The forma
|
|||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"attemptstoaltercrontab_user_config"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span></code></pre><p>When using the parameter <cite>&report=complianceitems</cite>, the <cite>search</cite> endpoint of the API
|
||||
will generate a list of compliance items from the results of the search.</p></section><section id="geolocations"><header><h3><a href="#id21">3.2 Geolocations</a></h3></header><p>The geolocations format transforms command results into an array of geolocated
|
||||
will generate a list of compliance items from the results of the search.</p></section><section id="geolocations"><header><h3><a href="#id15">2.2 Geolocations</a></h3></header><p>The geolocations format transforms command results into an array of geolocated
|
||||
endpoints for consumption by a map, like Google Maps. The format discards
|
||||
results details, and only stores the value of FoundAnything.</p><p>This feature requires using <strong>MaxMind's GeoIP2-City</strong> database. The database
|
||||
must be configured in the API as follow:</p><pre><code class="code">[maxmind]
|
||||
|
|
@ -994,4 +896,104 @@ must be configured in the API as follow:</p><pre><code class="code">[maxmind]
|
|||
<span class="name tag">"latitude"</span><span class="punctuation">:</span> <span class="literal number float">39.4284</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"longitude"</span><span class="punctuation">:</span> <span class="literal number float">-74.4957</span>
|
||||
<span class="punctuation">}</span></code></pre><p>When using the parameter <cite>&report=geolocations</cite>, the <cite>search</cite> endpoint of the
|
||||
API will generate a list of geolocations from the results of the search.</p></section></section></body></html>
|
||||
API will generate a list of geolocations from the results of the search.</p></section></section><section id="authentication-with-x-pgpauthorization-version-1"><header><h2><a href="#id16">3 Authentication with X-PGPAUTHORIZATION version 1</a></h2></header><p>Authenticating against the MIG API requires sending a PGP signed token in the
|
||||
request header named <cite>X-PGPAUTHORIZATION</cite>. The key that signs the token must
|
||||
belong to an active investigator. Construction of the token works as follows:</p><ol class="arabic"><li><p>make a string named <strong>str</strong> composed of a version, a UTC timestamp in RFC3339 format
|
||||
and a random nonce, each separated by semicolons. The current version is <strong>1</strong>
|
||||
and may be upgraded in the future. The nonce value must be a positive integer.</p><p><strong>str=<VERSION>;<UTC TIMESTAMP RFC3339>;<NONCE></strong></p><p>UTC is a hard requirement. The timestamp must end with the suffix <strong>Z</strong>
|
||||
which indicates the UTC timezone. In bash, a correct timestamp can be
|
||||
generated with the command <cite>$ date -u +%Y-%m-%dT%H:%M:%SZ</cite>.</p><p>An example string would look like: <cite>1;2006-01-02T15:04:05Z;1825922807490630059</cite></p><p>The string must be terminated by a newline character, hexadecimal code <cite>0x0a</cite>.</p></li></ol><pre><code class="code bash"><span class="name variable">$ </span>hexdump -C <span class="operator"><<<</span> <span class="literal string single">'1;2006-01-02T15:04:05Z;1825922807490630059'</span>
|
||||
<span class="literal number">00000000</span> <span class="literal number">31</span> 3b <span class="literal number">32</span> <span class="literal number">30</span> <span class="literal number">30</span> <span class="literal number">36</span> 2d <span class="literal number">30</span> <span class="literal number">31</span> 2d <span class="literal number">30</span> <span class="literal number">32</span> <span class="literal number">54</span> <span class="literal number">31</span> <span class="literal number">35</span> 3a <span class="punctuation">|</span>1<span class="punctuation">;</span>2006-01-02T15:<span class="punctuation">|</span>
|
||||
<span class="literal number">00000010</span> <span class="literal number">30</span> <span class="literal number">34</span> 3a <span class="literal number">30</span> <span class="literal number">35</span> 5a 3b <span class="literal number">31</span> <span class="literal number">38</span> <span class="literal number">32</span> <span class="literal number">35</span> <span class="literal number">39</span> <span class="literal number">32</span> <span class="literal number">32</span> <span class="literal number">38</span> <span class="literal number">30</span> <span class="punctuation">|</span>04:05Z<span class="punctuation">;</span>182592280<span class="punctuation">|</span>
|
||||
<span class="literal number">00000020</span> <span class="literal number">37</span> <span class="literal number">34</span> <span class="literal number">39</span> <span class="literal number">30</span> <span class="literal number">36</span> <span class="literal number">33</span> <span class="literal number">30</span> <span class="literal number">30</span> <span class="literal number">35</span> <span class="literal number">39</span> 0a <span class="punctuation">|</span>7490630059.<span class="punctuation">|</span>
|
||||
0000002b</code></pre><ol class="arabic" start="2"><li><p>PGP sign <strong>str</strong> with the private key of the investigator. Armor and detach
|
||||
the signature into <strong>armoredSig</strong>:</p><pre>$ gpg -a --detach-sig <<< '1;2006-01-02T15:04:05Z;1825922807490630059'
|
||||
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQEcBAABCAAGBQJUZ5psAAoJEKPWUhc7dj6PFd8IALWQS4x9Kzssww1pxc7uq9mg
|
||||
JT/3jHLwAYPQV3ltqFcI5R2EGHo5DsXXjX6lfOc7DgbteB9UV+H++KG0oVUTTjuP
|
||||
kncmFYmoBEDqbXju6EASBLyUlt3M43N9DmQaAaeoyW2gB0p0aEYRZoN3Cf0O0qhU
|
||||
b3nnsCz6IyuBcQAZh1Jnmf7AMwRmXier8OflObQ9wJ1iYF9KCD0TgP1Z+kaCvMqC
|
||||
PWQ5XaNaXn665V19mjAMicOtO9U3A/v4ApYyUSPyq0cuLrT8z/Z1vdjyeZVTaOM8
|
||||
MhnoKfgBnegQnP+BPQZlWcjaBsquenC/joYRhq20nAEwSjZ1Nm7+qHo/DW0bYOA=
|
||||
=4nrR
|
||||
-----END PGP SIGNATURE-----</pre></li><li><p>Create <strong>sig</strong> by taking <strong>armoredSig</strong> and removing the PGP headers, footers,
|
||||
empty lines and newlines.</p><blockquote><p>example: <cite>iQEcBAABCAAGBQJUWPDpAAoJEKPWUhc7dj6PQdgH/0TRMOEAL4SL6v+JvixWtEGJzXBCqBpRBsygHAKT+m4AxwniVa9vr8vfWm14eFpZTGdlDx39Ko+tdFoHn5Z1yKEeQWEQYXqhneAnv0pYR1aIjXM8MY63TNePWBZxUerlRkjv2IH16/W5aBrbOctOxEs1BKuN2pd4Hgubr+2f43gcRcWW+Ww/5Fyg1lKzH8jP84uqiIT8wQOdBrwUkgRdSdfMQbYFjsgY57G+ZsMobNhhlFedgKuZShJCd+G1GlwsfZPsZOSLmVZahI7wjR3vckCJ66eff3e/xX7Gt0zGGa5i1dgH5Q6TSjRGRBE37FwD4C6fycUEuy9yKI7iFziw33Y==k6gT</cite></p></blockquote></li><li><p>Create <strong>token</strong> by concatenating <strong>str</strong>, a semicolon, and <strong>sig</strong>.
|
||||
<strong>token=<str>;<sig></strong>
|
||||
example: <cite>1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaPWUhc7dj6...<truncated></cite></p></li><li><p>Send <strong>token</strong> in the header named <strong>X-PGPAUTHORIZATION</strong> with the request:</p><pre>$ curl -H 'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;owEBYQGe/pANAwAIAaP...<truncated>' localhost:12345/api/v1/</pre></li><li><p>The API verifies the version and validity period of the timestamp. By default, a
|
||||
token will be rejected if its timestamp deviates from the server time by more
|
||||
than 10 minutes. Administrators can configure this value. In effect, this
|
||||
means a timestamp is valid for twice the duration of the window. By default,
|
||||
that's 10 minutes before current server time, and 10 minutes after current
|
||||
server time.</p></li><li><p>If the timestamp is valid, the API next verifies the signature against the data
|
||||
and authenticates the user. Failure to verify the signature returns an error
|
||||
with the HTTP code 401 Unauthorized.</p></li><li><p>The user is authorized, the API processes and answer the request.</p></li></ol><section id="security-implications"><header><h3><a href="#id17">3.1 Security implications</a></h3></header><ol class="arabic"><li><p>A token can be used an unlimited number of times within its validity period.
|
||||
There is no check to guarantee that a token is only used once. It is
|
||||
assumed that the token is transmitted over a secure channel such as HTTPS to
|
||||
prevent token theft by a malicious user.</p></li><li><p>API clients and servers must use proper time synchronization for the timestamp
|
||||
verification to work. A client or a server that has inaccurate time may not be
|
||||
able to establish connections. We believe this requirement to be reasonable
|
||||
considering the sensitivity of the API.</p></li></ol></section><section id="example-1-invalid-timestamp"><header><h3><a href="#id18">3.2 Example 1: invalid timestamp</a></h3></header><p>The signature is valid but the timestamp is beyond the acceptable time window.</p><pre><code class="code bash"><span class="name variable">$ </span>curl -H <span class="literal string single">'X-PGPAUTHORIZATION: 1;2006-01-02T15:04:05Z;1825922807490630059;iQEcB...<truncated>'</span> http://localhost:12345/api/v1/
|
||||
|
||||
<span class="operator">{</span>
|
||||
<span class="literal string double">"collection"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"error"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"code"</span>: <span class="literal string double">"6077873045059431424"</span>,
|
||||
<span class="literal string double">"message"</span>: <span class="literal string double">"Authorization verification failed with error 'verifySignedToken() -> token timestamp is not within acceptable time limits'"</span>
|
||||
<span class="operator">}</span>,
|
||||
<span class="literal string double">"href"</span>: <span class="literal string double">"http://localhost:12345/api/v1/"</span>,
|
||||
<span class="literal string double">"template"</span>: <span class="operator">{}</span>,
|
||||
<span class="literal string double">"version"</span>: <span class="literal string double">"1.0"</span>
|
||||
<span class="operator">}</span>
|
||||
<span class="operator">}</span></code></pre></section><section id="example-2-invalid-signature"><header><h3><a href="#id19">3.3 Example 2: invalid signature</a></h3></header><p>The signature is not valid, or is signed by a key that the API does not
|
||||
recognize.</p><pre><code class="code bash"><span class="name variable">$ </span>curl -H <span class="literal string single">'X-PGPAUTHORIZATION: 1;2014-11-04T15:36:05Z;1825922807490630059;iQEcBA...<truncated>'</span> http://localhost:12345/api/v1/
|
||||
|
||||
<span class="operator">{</span>
|
||||
<span class="literal string double">"collection"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"error"</span>: <span class="operator">{</span>
|
||||
<span class="literal string double">"code"</span>: <span class="literal string double">"6077875007260332032"</span>,
|
||||
<span class="literal string double">"message"</span>: <span class="literal string double">"Authorization verification failed with error 'verifySignedToken() -> GetFingerprintFromSignature() -> openpgp: invalid signature: hash tag doesn't match'"</span>
|
||||
<span class="operator">}</span>,
|
||||
<span class="literal string double">"href"</span>: <span class="literal string double">"http://localhost:12345/api/v1/"</span>,
|
||||
<span class="literal string double">"template"</span>: <span class="operator">{}</span>,
|
||||
<span class="literal string double">"version"</span>: <span class="literal string double">"1.0"</span>
|
||||
<span class="operator">}</span>
|
||||
<span class="operator">}</span></code></pre></section><section id="generating-a-token-in-bash"><header><h3><a href="#id20">3.4 Generating a token in Bash</a></h3></header><pre><code class="code">$ token="1;$(date -u +%Y-%m-%dT%H:%M:%SZ);$RANDOM$RANDOM$RANDOM$RANDOM"; \
|
||||
sig=$(gpg -a --detach-sig <<< $token |tail -8 |head -7 \
|
||||
| sed ':a;N;$!ba;s/\n//g'); echo "X-PGPAUTHORIZATION: $token;$sig"
|
||||
|
||||
X-PGPAUTHORIZATION: 1;2014-11-04T19:13:37Z;13094113753132512760;iQEcBAA.....</code></pre></section><section id="generating-a-token-in-python"><header><h3><a href="#id21">3.5 Generating a token in Python</a></h3></header><pre><code class="code python"><span class="comment">#!/usr/bin/env python</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">os</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">gnupg</span>
|
||||
<span class="keyword namespace">from</span> <span class="name namespace">time</span> <span class="keyword namespace">import</span> <span class="name">gmtime</span><span class="punctuation">,</span> <span class="name">strftime</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">random</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">requests</span>
|
||||
<span class="keyword namespace">import</span> <span class="name namespace">json</span>
|
||||
|
||||
<span class="keyword">def</span> <span class="name function">makeToken</span><span class="punctuation">(</span><span class="name">gpghome</span><span class="punctuation">,</span> <span class="name">keyid</span><span class="punctuation">):</span>
|
||||
<span class="name">gpg</span> <span class="operator">=</span> <span class="name">gnupg</span><span class="operator">.</span><span class="name">GPG</span><span class="punctuation">(</span><span class="name">gnupghome</span><span class="operator">=</span><span class="name">gpghome</span><span class="punctuation">)</span>
|
||||
<span class="name">version</span> <span class="operator">=</span> <span class="literal string">"1"</span>
|
||||
<span class="name">timestamp</span> <span class="operator">=</span> <span class="name">strftime</span><span class="punctuation">(</span><span class="literal string">"%Y-%m-</span><span class="literal string interpol">%d</span><span class="literal string">T%H:%M:%SZ"</span><span class="punctuation">,</span> <span class="name">gmtime</span><span class="punctuation">())</span>
|
||||
<span class="name">nonce</span> <span class="operator">=</span> <span class="name builtin">str</span><span class="punctuation">(</span><span class="name">random</span><span class="operator">.</span><span class="name">randint</span><span class="punctuation">(</span><span class="literal number integer">10000</span><span class="punctuation">,</span> <span class="literal number integer">18446744073709551616</span><span class="punctuation">))</span>
|
||||
<span class="name">token</span> <span class="operator">=</span> <span class="name">version</span> <span class="operator">+</span> <span class="literal string">";"</span> <span class="operator">+</span> <span class="name">timestamp</span> <span class="operator">+</span> <span class="literal string">";"</span> <span class="operator">+</span> <span class="name">nonce</span>
|
||||
<span class="name">sig</span> <span class="operator">=</span> <span class="name">gpg</span><span class="operator">.</span><span class="name">sign</span><span class="punctuation">(</span><span class="name">token</span> <span class="operator">+</span> <span class="literal string">"</span><span class="literal string escape">\n</span><span class="literal string">"</span><span class="punctuation">,</span>
|
||||
<span class="name">keyid</span><span class="operator">=</span><span class="name">keyid</span><span class="punctuation">,</span>
|
||||
<span class="name">detach</span><span class="operator">=</span><span class="name builtin pseudo">True</span><span class="punctuation">,</span> <span class="name">clearsign</span><span class="operator">=</span><span class="name builtin pseudo">True</span><span class="punctuation">)</span>
|
||||
<span class="name">token</span> <span class="operator">+=</span> <span class="literal string">";"</span>
|
||||
<span class="name">linectr</span><span class="operator">=</span><span class="literal number integer">0</span>
|
||||
<span class="keyword">for</span> <span class="name">line</span> <span class="operator word">in</span> <span class="name builtin">iter</span><span class="punctuation">(</span><span class="name builtin">str</span><span class="punctuation">(</span><span class="name">sig</span><span class="punctuation">)</span><span class="operator">.</span><span class="name">splitlines</span><span class="punctuation">()):</span>
|
||||
<span class="name">linectr</span><span class="operator">+=</span><span class="literal number integer">1</span>
|
||||
<span class="keyword">if</span> <span class="name">linectr</span> <span class="operator"><</span> <span class="literal number integer">4</span> <span class="operator word">or</span> <span class="name">line</span><span class="operator">.</span><span class="name">startswith</span><span class="punctuation">(</span><span class="literal string">'-'</span><span class="punctuation">)</span> <span class="operator word">or</span> <span class="operator word">not</span> <span class="name">line</span><span class="punctuation">:</span>
|
||||
<span class="keyword">continue</span>
|
||||
<span class="name">token</span> <span class="operator">+=</span> <span class="name">line</span>
|
||||
<span class="keyword">return</span> <span class="name">token</span>
|
||||
|
||||
<span class="keyword">if</span> <span class="name">__name__</span> <span class="operator">==</span> <span class="literal string">'__main__'</span><span class="punctuation">:</span>
|
||||
<span class="name">token</span> <span class="operator">=</span> <span class="name">makeToken</span><span class="punctuation">(</span><span class="literal string">"/home/ulfr/.gnupg"</span><span class="punctuation">,</span>
|
||||
<span class="literal string">"E60892BB9BD89A69F759A1A0A3D652173B763E8F"</span><span class="punctuation">)</span>
|
||||
<span class="name">r</span> <span class="operator">=</span> <span class="name">requests</span><span class="operator">.</span><span class="name">get</span><span class="punctuation">(</span><span class="literal string">"http://localhost:12345/api/v1/dashboard"</span><span class="punctuation">,</span>
|
||||
<span class="name">headers</span><span class="operator">=</span><span class="punctuation">{</span><span class="literal string">'X-PGPAUTHORIZATION'</span><span class="punctuation">:</span> <span class="name">token</span><span class="punctuation">})</span>
|
||||
<span class="keyword">print</span> <span class="name">token</span>
|
||||
<span class="keyword">print</span> <span class="name">r</span><span class="operator">.</span><span class="name">text</span></code></pre></section></section></body></html>
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,34 @@
|
|||
================
|
||||
MIG command line
|
||||
================
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of contents
|
||||
|
||||
The MIG command line is a terminal-based client interface that provides fast
|
||||
interaction with the MIG API to launch actions and receive results. Its goal is
|
||||
to be simple and fast to use for basic investigations. Unlike the mig-console,
|
||||
it does not provide access to previous actions, search or investigators
|
||||
management.
|
||||
|
||||
To install the MIG command line, use `go get mig.ninja/mig/client/mig`. This
|
||||
command will place a binary called `mig` under $GOPATH/bin/mig. On first run, it
|
||||
will invite you to create a configuration:
|
||||
|
||||
.. code::
|
||||
|
||||
$ mig file -path /etc -name "^passwd$" -content "jvehent"
|
||||
no configuration file found at /home/jvehent/.migrc
|
||||
would you like to generate a new configuration file at /home/jvehent/.migrc? Y/n> Y
|
||||
found key 'Julien Vehent (ulfr) <jvehent@mozilla.com>' with fingerprint 'E60892BB9BD89A69F759A1A0A3D652173B763E8F'.
|
||||
use this key? Y/n> Y
|
||||
using key E60892BB9BD89A69F759A1A0A3D652173B763E8F
|
||||
what is the location of the API? (ex: https://mig.example.net/api/v1/) > https://api.mig.example.net/api/v1/
|
||||
MIG client configuration generated at /home/jvehent/.migrc
|
||||
|
||||
1226 agents will be targeted. ctrl+c to cancel. launching in 5 4....
|
||||
|
||||
Subsequent runs of `mig` will read the configuration from `$HOME/.migrc` directly.
|
||||
|
||||
For more examples on how to use the mig command line, see the `cheatsheet
|
||||
<cheatsheet.rst.html>`_.
|
||||
|
|
@ -0,0 +1,247 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
color: #eaeaea;
|
||||
font: 16px;
|
||||
line-height: 1.5em;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
}
|
||||
|
||||
#table-of-contents ul {
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
/* General & 'Reset' Stuff */
|
||||
|
||||
.container {
|
||||
width: 95%;
|
||||
max-width: 1000px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
section {
|
||||
display: block;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
* header - container
|
||||
* h1 - project name
|
||||
* h2 - project description
|
||||
* */
|
||||
|
||||
header {
|
||||
background: rgba(0, 0, 0, 0.1);
|
||||
width: 100%;
|
||||
/*border-bottom: 1px dashed #b5e853;*/
|
||||
/*padding: 20px 0;
|
||||
* margin: 0 0 40px 0;*/
|
||||
padding: 5px 0;
|
||||
margin: 0 0 10px 0;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 30px;
|
||||
line-height: 1.5;
|
||||
margin: 0 0 0 -40px;
|
||||
font-weight: bold;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
/*color: #b5e853;*/
|
||||
color: #089d00;
|
||||
text-shadow: 0 1px 1px rgba(0, 0, 0, 0.1),
|
||||
0 0 5px rgba(181, 232, 83, 0.1),
|
||||
0 0 10px rgba(181, 232, 83, 0.1);
|
||||
letter-spacing: -1px;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
header h1:before {
|
||||
content: "./ ";
|
||||
font-size: 24px;
|
||||
}
|
||||
|
||||
header h2 {
|
||||
font-size: 18px;
|
||||
font-weight: 300;
|
||||
}
|
||||
|
||||
/* Main Content
|
||||
* */
|
||||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
max-width: 100%
|
||||
}
|
||||
|
||||
h2 a {
|
||||
font-weight: bold;
|
||||
color: #8AB638;
|
||||
line-height: 1.4em;
|
||||
font-size: 1.4em;
|
||||
}
|
||||
h3 a, h4 a, h5 a, h6 a {
|
||||
font-weight: bold;
|
||||
color: #934500;
|
||||
line-height: 1.4em;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 30px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 28px;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
h4 {
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
h5 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
h6 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
color: #999;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
dt {
|
||||
font-style: italic;
|
||||
font-weight: bold;
|
||||
}
|
||||
/*
|
||||
ul li {
|
||||
list-style: none;
|
||||
}
|
||||
*/
|
||||
/*
|
||||
ul li:before {
|
||||
content: ">>";
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
font-size: 13px;
|
||||
color: #b5e853;
|
||||
margin-left: -37px;
|
||||
margin-right: 21px;
|
||||
line-height: 16px;
|
||||
}
|
||||
*/
|
||||
|
||||
blockquote {
|
||||
color: #aaa;
|
||||
padding-left: 10px;
|
||||
border-left: 1px dotted #666;
|
||||
}
|
||||
|
||||
|
||||
pre {
|
||||
background: rgba(0, 0, 0, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.15);
|
||||
padding: 10px;
|
||||
font-size: 14px;
|
||||
//color: #b5e853;
|
||||
border-radius: 2px;
|
||||
-moz-border-radius: 2px;
|
||||
-webkit-border-radius: 2px;
|
||||
text-wrap: normal;
|
||||
overflow: auto;
|
||||
overflow-y: hidden;
|
||||
}
|
||||
|
||||
pre.address {
|
||||
margin-bottom: 0 ;
|
||||
margin-top: 0 ;
|
||||
font: inherit }
|
||||
|
||||
pre.literal-block, pre.doctest-block, pre.math, pre.code {
|
||||
margin-left: 2em ;
|
||||
margin-right: 2em }
|
||||
|
||||
code .ln { color: grey; } /* line numbers */
|
||||
/*code, code { background-color: #eeeeee }*/
|
||||
code .comment, code .comment, code .c1 { color: #999; }
|
||||
code .keyword, code .keyword, code .kd, code .kn, code .k, code .o { color: #FC8F3F; font-weight: bold;}
|
||||
code .nb { color: #c45918;}
|
||||
code .s {color: #0a77c4;}
|
||||
code .punctuation, code .p { color: white;}
|
||||
code .literal.string, code .literal.string { color: #40BF32; }
|
||||
code .name, code .name.builtin, code .nx { color: white; }
|
||||
code .deleted, code .deleted { background-color: #DEB0A1}
|
||||
code .inserted, code .inserted { background-color: #A3D289}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
th {
|
||||
text-align: left;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
hr {
|
||||
height: 0;
|
||||
border: 0;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
color: #b5e853;
|
||||
}
|
||||
/* Links
|
||||
* a, a:hover, a:visited
|
||||
* */
|
||||
|
||||
a {
|
||||
color: #63c0f5;
|
||||
/*text-shadow: 0 0 5px rgba(104, 182, 255, 0.5);*/
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
cite {
|
||||
color: #00FF4A;
|
||||
}
|
||||
|
||||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG command line</h1><p>The MIG command line is a terminal-based client interface that provides fast
|
||||
interaction with the MIG API to launch actions and receive results. Its goal is
|
||||
to be simple and fast to use for basic investigations. Unlike the mig-console,
|
||||
it does not provide access to previous actions, search or investigators
|
||||
management.</p><p>To install the MIG command line, use <cite>go get mig.ninja/mig/client/mig</cite>. This
|
||||
command will place a binary called <cite>mig</cite> under $GOPATH/bin/mig. On first run, it
|
||||
will invite you to create a configuration:</p><pre><code class="code">$ mig file -path /etc -name "^passwd$" -content "jvehent"
|
||||
no configuration file found at /home/jvehent/.migrc
|
||||
would you like to generate a new configuration file at /home/jvehent/.migrc? Y/n> Y
|
||||
found key 'Julien Vehent (ulfr) <jvehent@mozilla.com>' with fingerprint 'E60892BB9BD89A69F759A1A0A3D652173B763E8F'.
|
||||
use this key? Y/n> Y
|
||||
using key E60892BB9BD89A69F759A1A0A3D652173B763E8F
|
||||
what is the location of the API? (ex: https://mig.example.net/api/v1/) > https://api.mig.example.net/api/v1/
|
||||
MIG client configuration generated at /home/jvehent/.migrc
|
||||
|
||||
1226 agents will be targeted. ctrl+c to cancel. launching in 5 4....</code></pre><p>Subsequent runs of <cite>mig</cite> will read the configuration from <cite>$HOME/.migrc</cite> directly.</p><p>For more examples on how to use the mig command line, see the <a class="reference external" href="cheatsheet.rst.html">cheatsheet</a>.</p></body></html>
|
||||
329
doc/concepts.rst
329
doc/concepts.rst
|
|
@ -1,7 +1,6 @@
|
|||
===================================================
|
||||
Mozilla InvestiGator Concepts & Internal Components
|
||||
===================================================
|
||||
:Author: Julien Vehent <jvehent@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
|
@ -76,11 +75,8 @@ Below is a high-level view of the architecture:
|
|||
|
||||
.. image:: .files/MIG-Arch-Diagram.png
|
||||
|
||||
Actions and Commands
|
||||
--------------------
|
||||
|
||||
Actions
|
||||
~~~~~~~
|
||||
-------
|
||||
|
||||
Actions are JSON files created by investigator to perform tasks on agents.
|
||||
|
||||
|
|
@ -168,8 +164,8 @@ Upon generation, additional fields are appended to the action:
|
|||
* **validfrom** and **expireafter**: two dates that constrains the validity of the
|
||||
action to a UTC time window.
|
||||
|
||||
Action/Commands workflow
|
||||
------------------------
|
||||
Investigation workflow
|
||||
-----------------------
|
||||
The diagram below represents the full workflow from the launch of an action by
|
||||
an investigation, to the retrieval of results from the database. The steps are
|
||||
explained in the legend of the diagram, and map to various components of MIG.
|
||||
|
|
@ -184,159 +180,6 @@ View `full size diagram`_.
|
|||
.. image:: .files/action_command_flow.svg
|
||||
|
||||
|
||||
Command execution flow in Agent and Modules
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Upon processing of an action, the scheduler will retrieve a list of agents to
|
||||
send the action to. One action is then derived into multiple commands and sent
|
||||
to agents.
|
||||
|
||||
An agent receives a command from the scheduler on its personal AMQP queue (1).
|
||||
It parses the command (2) and extracts all of the operations to perform.
|
||||
Operations are passed to modules and executed in parallel (3). Rather than
|
||||
maintaining a state of the running command, the agent create a goroutine and a
|
||||
channel tasked with receiving the results from the modules. Each modules
|
||||
published its results inside that channel (4). The result parsing goroutine
|
||||
receives them, and when it has received all of them, populates the `results` (5)
|
||||
array of the command with the results from each module, and send the command
|
||||
back to the scheduler(6).
|
||||
|
||||
When the agent is done running the command, both the channel and the goroutine
|
||||
are destroyed.
|
||||
|
||||
::
|
||||
|
||||
+-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ]
|
||||
|command|+---->(listener)
|
||||
+-------+ |(2)
|
||||
^ V
|
||||
|(1) (parser)
|
||||
| + [ m o d u l e s ]
|
||||
+---------+ | (3)|----------> op1 +----------------+
|
||||
|SCHEDULER|+---+ |------------> op2 +--------------|
|
||||
| |<---+ |--------------> op3 +------------|
|
||||
+---------+ | +----------------> op4 +----------+
|
||||
| V(4)
|
||||
|(6) (receiver)
|
||||
| |
|
||||
| V(5)
|
||||
+ (publisher)
|
||||
+-------+ /
|
||||
|results|<-----------------------------------------'
|
||||
+-------+
|
||||
|
||||
The command received by the agent is composed of a copy of the action described
|
||||
previously, but signed with the private key of a trusted investigator. It also
|
||||
contains additional parameters that are specific to the targetted agent, such as
|
||||
command processing timestamps, name of the agent queue on the message broker,
|
||||
action and command unique IDs and status and results of the command. Below is an
|
||||
command derived from the root password checking action, and ran on the host named
|
||||
'host1.example.net'.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"id": 1.427392971126604e+18,
|
||||
"action": { ... SIGNED COPY OF THE ACTION ... },
|
||||
"agent": {
|
||||
"id": 1.4271760437936648e+18,
|
||||
"name": "host1.example.net",
|
||||
"queueloc": "linux.host1.example.net.981alsd19aos1984",
|
||||
"mode": "daemon",
|
||||
"version": "20150324+0d0f88c.prod"
|
||||
},
|
||||
"status": "success",
|
||||
"results": [
|
||||
{
|
||||
"foundanything": true,
|
||||
"success": true,
|
||||
"elements": {
|
||||
"root_passwd_hashed_or_disabled": [
|
||||
{
|
||||
"file": "/etc/shadow",
|
||||
"fileinfo": {
|
||||
"lastmodified": "2015-02-07 01:51:07.17850601 +0000 UTC",
|
||||
"mode": "----------",
|
||||
"size": 1684
|
||||
},
|
||||
"search": {
|
||||
"contents": [
|
||||
"root:(\\*|!|\\$(1|2a|5|6)\\$).+"
|
||||
],
|
||||
"options": {
|
||||
"matchall": false,
|
||||
"matchlimit": 0,
|
||||
"maxdepth": 0
|
||||
},
|
||||
"paths": [
|
||||
"/etc"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"statistics": {
|
||||
"exectime": "2.017849ms",
|
||||
"filescount": 1,
|
||||
"openfailed": 0,
|
||||
"totalhits": 1
|
||||
},
|
||||
"errors": null
|
||||
}
|
||||
],
|
||||
"starttime": "2015-03-26T18:02:51.126605Z",
|
||||
"finishtime": "2015-03-26T18:03:00.671232Z"
|
||||
}
|
||||
|
||||
The results of the command show that the file '/etc/shadow' has matched, and
|
||||
thus "FoundAnything" returned "True".
|
||||
|
||||
The invocation of the file module has completed successfully, which is
|
||||
represented by **results->0->success=true**. In our example, there is only one
|
||||
operation in the **action->operations** array, so only one result is present.
|
||||
When multiple operations are performed, each has its results listed in a
|
||||
corresponding entry of the results array (operations[0] is in results[0],
|
||||
operations[1] in results[1], etc...).
|
||||
|
||||
Finally, the agent has performed all operations in the operations array
|
||||
successfully, and returned **status=success**. Had a failure happened on the
|
||||
agent, the returned status would be one of "failed", "timeout" or "cancelled".
|
||||
|
||||
Command expiration & timeouts
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To prevent abuse of resources, agents will kill long-running modules after a
|
||||
given period of time. That timeout is hardcoded in the agent configuration
|
||||
at compile time and defaults to 5 minutes.
|
||||
|
||||
.. code:: go
|
||||
|
||||
// timeout after which a module run is killed
|
||||
var MODULETIMEOUT time.Duration = 300 * time.Second
|
||||
|
||||
That timeout represents the **maximum** execution time of a single operation. If
|
||||
an action contains 3 operations, each operation gets its own timeout. But because
|
||||
operations run in parallel in the agent, the maximum runtime of an action should
|
||||
be very close to the value of MODULETIMEOUT.
|
||||
|
||||
In a typical deployment, it is safe to increase MODULETIMEOUT to allow for
|
||||
longer operations. A value of 20 minutes is usual. Make sure to fine tune this
|
||||
to your environment, and get the approval of your ops team because mig-agent
|
||||
may end up consuming resources (but never more than 50% of the cpu available on
|
||||
a system).
|
||||
|
||||
Oftentimes, an investigator will want a timeout that is much shorter than the value
|
||||
of MODULETIMEOUT. In the MIG command line, the flag `-e` controls the
|
||||
expiration. It defaults to 5 minutes but can be set to 30 seconds for simple
|
||||
investigations. When that happens, the agent will calculate an appropriate expiration
|
||||
for the operations being run. If the expiration set on the action is set to 30 seconds,
|
||||
the agent will kill operations that run for more than 30 seconds.
|
||||
|
||||
If the expiration is larger than the value of MODULETIMEOUT (for example, 2
|
||||
hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to
|
||||
allow agents that only check in periodically to pick up actions long after they
|
||||
are launched.
|
||||
|
||||
Access Control Lists
|
||||
--------------------
|
||||
|
||||
|
|
@ -410,172 +253,6 @@ control to authorize access to specific functions of modules. For example, an
|
|||
investigator could be authorized to call the `regex` function of filechecker
|
||||
module, but only in `/etc`. This functionality is not implemented yet.
|
||||
|
||||
Agent initialization process
|
||||
----------------------------
|
||||
The agent tries to be as autonomous as possible. One of the goal is to ship
|
||||
agents without requiring external provisioning tools, such as Chef or Puppet.
|
||||
Therefore, the agent attempts to install itself as a service, and also supports
|
||||
a builtin upgrade protocol (described in the next section).
|
||||
|
||||
As a portable binary, the agent needs to detect the type of operating system
|
||||
and init method that is used by an endpoint. Depending on the endpoint,
|
||||
different initialization methods are used. The diagram below explains the
|
||||
decision process followed by the agent.
|
||||
|
||||
.. image:: .files/mig-agent-initialization-process.png
|
||||
|
||||
Go does not provide support for running programs in the backgroud. On endpoints
|
||||
that run upstart, systemd (linux) or launchd (darwin), this is not an issue
|
||||
because the init daemon takes care of running the agent in the background,
|
||||
rerouting its file descriptors and restarting on crash. On Windows and System-V,
|
||||
however, the agent daemonizes by forking itself into `foreground` mode, and
|
||||
re-forking itself on error (such as loss of connectivity to the relay).
|
||||
On Windows and System-V, if the agent is killed, it will not be restarted
|
||||
automatically.
|
||||
|
||||
Registration process
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The initialization process goes through several environment detection steps
|
||||
which are used to select the proper init method. Once started, the agent will
|
||||
send a heartbeat to the public relay, and also store that heartbeat in its
|
||||
`run` directory. The location of the `run` directory is platform specific.
|
||||
|
||||
* windows: C:\Windows\
|
||||
* darwin: /Library/Preferences/mig/
|
||||
* linux: /var/run/mig/
|
||||
|
||||
Below is a sample heartbeat message from a linux agent stored in
|
||||
`/var/run/mig/mig-agent.ok`.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"destructiontime": "0001-01-01T00:00:00Z",
|
||||
"environment": {
|
||||
"arch": "amd64",
|
||||
"ident": "Red Hat Enterprise Linux Server release 6.5 (Santiago)",
|
||||
"init": "upstart"
|
||||
},
|
||||
"heartbeatts": "2014-07-31T14:00:20.00442837-07:00",
|
||||
"name": "someserver.example.net",
|
||||
"os": "linux",
|
||||
"pid": 26256,
|
||||
"queueloc": "linux.someserver.example.net.5hsa811oda",
|
||||
"starttime": "2014-07-30T21:34:48.525449401-07:00",
|
||||
"version": "201407310027+bcbdd94.prod"
|
||||
}
|
||||
|
||||
Check-In mode
|
||||
~~~~~~~~~~~~~
|
||||
In infrastructure where running the agent as a permanent process is not
|
||||
acceptable, it is possible to run the agent as a cron job. By starting the
|
||||
agent with the flag **-m agent-checkin**, the agent will connect to the
|
||||
configured relay, retrieve and run outstanding commands, and exit after 10
|
||||
seconds of inactivity.
|
||||
|
||||
Agent upgrade process
|
||||
---------------------
|
||||
MIG supports upgrading agents in the wild. The upgrade protocol is designed with
|
||||
security in mind. The flow diagram below presents a high-level view:
|
||||
|
||||
::
|
||||
|
||||
Investigator Scheduler Agent NewAgent FileServer
|
||||
+-----------+ +-------+ +---+ +------+ +--------+
|
||||
| | | | |
|
||||
| 1.initiate | | | |
|
||||
|------------------>| | | |
|
||||
| | 2.send command | | |
|
||||
| |------------------>| 3.verify | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 4.download | |
|
||||
| | |-------------------------------------->|
|
||||
| | | | |
|
||||
| | | 5.checksum | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 6.exec | |
|
||||
| | |------------------>| |
|
||||
| | 7.return own PID | | |
|
||||
| |<------------------| | |
|
||||
| | | | |
|
||||
| |------+ 8.mark | | |
|
||||
| | | agent as | | |
|
||||
| | | upgraded | | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 9.register | | |
|
||||
| |<--------------------------------------| |
|
||||
| | | | |
|
||||
| |------+10.find dup | | |
|
||||
| | |agents in | | |
|
||||
| | |registrations | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 11.send command to kill PID old agt| |
|
||||
| |-------------------------------------->| |
|
||||
| | | | |
|
||||
| | 12.acknowledge | | |
|
||||
| |<--------------------------------------| |
|
||||
|
||||
All upgrade operations are initiated by an investigator (1). The upgrade is
|
||||
triggered by an action to the upgrade module with the following parameters:
|
||||
|
||||
.. code:: json
|
||||
|
||||
"Operations": [
|
||||
{
|
||||
"Module": "upgrade",
|
||||
"Parameters": {
|
||||
"linux/amd64": {
|
||||
"to_version": "16eb58b-201404021544",
|
||||
"location": "http://localhost/mig/bin/linux/amd64/mig-agent",
|
||||
"checksum": "31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
|
||||
* Each OS family and architecture have their own parameters (ex: "linux/amd64",
|
||||
"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:
|
||||
* to_version is the version an agent should upgrade to
|
||||
* location points to a HTTPS address that contains the agent binary
|
||||
* checksum is a SHA256 hash of the agent binary to be verified after download
|
||||
|
||||
The parameters above are signed using a standard PGP action signature.
|
||||
|
||||
The upgrade action is forwarded to agents (2) like any other action. The action
|
||||
signature is verified by the agent (3), and the upgrade module is called. The
|
||||
module downloads the new binary (4), verifies the version and checksum (5) and
|
||||
installs itself on the system.
|
||||
|
||||
Assuming everything checks in, the old agent executes the binary of the new
|
||||
agent (6). At that point, two agents are running on the same machine, and the
|
||||
rest of the protocol is designed to shut down the old agent, and clean up.
|
||||
|
||||
After executing the new agent, the old agent returns a successful result to the
|
||||
scheduler, and includes its own PID in the results.
|
||||
The new agent starts by registering with the scheduler (7). This tells the
|
||||
scheduler that two agents are running on the same node, and one of them must
|
||||
terminate. The scheduler sends a kill action to both agents with the PID of the
|
||||
old agent (8). The kill action may be executed twice, but that doesn't matter.
|
||||
When the scheduler receives the kill results (9), it sends a new action to check
|
||||
for `mig-agent` processes (10). Only one should be found in the results (11),
|
||||
and if that is the case, the scheduler tells the agent to remove the binary of
|
||||
the old agent (12). When the agent returns (13), the upgrade protocol is done.
|
||||
|
||||
If the PID of the old agent lingers on the system, an error is logged for the
|
||||
investigator to decide what to do next. The scheduler does not attempt to clean
|
||||
up the situation.
|
||||
|
||||
Threat Model
|
||||
------------
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
@ -227,7 +229,7 @@ cite {
|
|||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>Mozilla InvestiGator Concepts & Internal Components</h1><table><tr><td class="field-label">Author</td><td>Julien Vehent <<a class="reference external" href="mailto:jvehent@mozilla.com">jvehent@mozilla.com</a>></td></tr></table><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#actions-and-commands" id="id1">1 Actions and Commands</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#actions" id="id2">1.1 Actions</a></p></li></ul></li><li><p><a class="reference internal" href="#action-commands-workflow" id="id3">2 Action/Commands workflow</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#command-execution-flow-in-agent-and-modules" id="id4">2.1 Command execution flow in Agent and Modules</a></p></li><li><p><a class="reference internal" href="#command-expiration-timeouts" id="id5">2.2 Command expiration & timeouts</a></p></li></ul></li><li><p><a class="reference internal" href="#access-control-lists" id="id6">3 Access Control Lists</a></p></li><li><p><a class="reference internal" href="#agent-initialization-process" id="id7">4 Agent initialization process</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#registration-process" id="id8">4.1 Registration process</a></p></li><li><p><a class="reference internal" href="#check-in-mode" id="id9">4.2 Check-In mode</a></p></li></ul></li><li><p><a class="reference internal" href="#agent-upgrade-process" id="id10">5 Agent upgrade process</a></p></li><li><p><a class="reference internal" href="#threat-model" id="id11">6 Threat Model</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#strong-gpg-security-model" id="id12">6.1 Strong GPG security model</a></p></li><li><p><a class="reference internal" href="#infrastructure-resiliency" id="id13">6.2 Infrastructure resiliency</a></p></li><li><p><a class="reference internal" href="#no-port-listening" id="id14">6.3 No port listening</a></p></li><li><p><a class="reference internal" href="#protection-of-connections-to-the-relays" id="id15">6.4 Protection of connections to the relays</a></p></li><li><p><a class="reference internal" href="#randomization-of-the-queue-names" id="id16">6.5 Randomization of the queue names</a></p></li><li><p><a class="reference internal" href="#whitelisting-of-agents" id="id17">6.6 Whitelisting of agents</a></p></li><li><p><a class="reference internal" href="#limit-data-extraction-to-a-minimum" id="id18">6.7 Limit data extraction to a minimum</a></p></li></ul></li></ul></div><p>MIG is a platform to perform investigative surgery on remote endpoints.
|
||||
</style></head><body><h1>Mozilla InvestiGator Concepts & Internal Components</h1><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#actions" id="id1">1 Actions</a></p></li><li><p><a class="reference internal" href="#investigation-workflow" id="id2">2 Investigation workflow</a></p></li><li><p><a class="reference internal" href="#access-control-lists" id="id3">3 Access Control Lists</a></p></li><li><p><a class="reference internal" href="#threat-model" id="id4">4 Threat Model</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#strong-gpg-security-model" id="id5">4.1 Strong GPG security model</a></p></li><li><p><a class="reference internal" href="#infrastructure-resiliency" id="id6">4.2 Infrastructure resiliency</a></p></li><li><p><a class="reference internal" href="#no-port-listening" id="id7">4.3 No port listening</a></p></li><li><p><a class="reference internal" href="#protection-of-connections-to-the-relays" id="id8">4.4 Protection of connections to the relays</a></p></li><li><p><a class="reference internal" href="#randomization-of-the-queue-names" id="id9">4.5 Randomization of the queue names</a></p></li><li><p><a class="reference internal" href="#whitelisting-of-agents" id="id10">4.6 Whitelisting of agents</a></p></li><li><p><a class="reference internal" href="#limit-data-extraction-to-a-minimum" id="id11">4.7 Limit data extraction to a minimum</a></p></li></ul></li></ul></div><p>MIG is a platform to perform investigative surgery on remote endpoints.
|
||||
It enables investigators to obtain information from large numbers of systems
|
||||
in parallel, thus accelerating investigation of incidents.</p><p>Besides scalability, MIG is designed to provide strong security primitives:</p><ul><li><p><strong>Access control</strong> is ensured by requiring GPG signatures on all actions. Sensitive
|
||||
actions can also request signatures from multiple investigators. An attacker
|
||||
|
|
@ -261,7 +263,7 @@ The agents also use the relays to send heartbeat at regular intervals, such that
|
|||
the scheduler always knows how many agents are alive at a given time.</p><p>The end-to-end workflow is:</p><blockquote><pre>{investigator} -https-> {API} {Scheduler} -amqps-> {Relays} -amqps-> {Agents}
|
||||
\ /
|
||||
sql\ /sql
|
||||
{DATABASE}</pre></blockquote><p>Below is a high-level view of the architecture:</p><img alt=".files/MIG-Arch-Diagram.png" src=".files/MIG-Arch-Diagram.png"><section id="actions-and-commands"><header><h2><a href="#id1">1 Actions and Commands</a></h2></header><section id="actions"><header><h3><a href="#id2">1.1 Actions</a></h3></header><p>Actions are JSON files created by investigator to perform tasks on agents.</p><p>For example, an investigator who wants to verify than root passwords are hashed
|
||||
{DATABASE}</pre></blockquote><p>Below is a high-level view of the architecture:</p><img alt=".files/MIG-Arch-Diagram.png" src=".files/MIG-Arch-Diagram.png"><section id="actions"><header><h2><a href="#id1">1 Actions</a></h2></header><p>Actions are JSON files created by investigator to perform tasks on agents.</p><p>For example, an investigator who wants to verify than root passwords are hashed
|
||||
and salted on linux systems, would use the following action:</p><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"verify root password storage method"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"target"</span><span class="punctuation">:</span> <span class="literal string double">"agents.queueloc like 'linux.%'"</span><span class="punctuation">,</span>
|
||||
|
|
@ -313,121 +315,10 @@ signed with the investigator's private GPG key. The signature is part of the
|
|||
action, and used by agents to verify that an action comes from a trusted
|
||||
investigator. <cite>PGPSignatures</cite> is an array that contains one or more signature
|
||||
from authorized investigators.</p></li><li><p><strong>validfrom</strong> and <strong>expireafter</strong>: two dates that constrains the validity of the
|
||||
action to a UTC time window.</p></li></ul></section></section><section id="action-commands-workflow"><header><h2><a href="#id3">2 Action/Commands workflow</a></h2></header><p>The diagram below represents the full workflow from the launch of an action by
|
||||
action to a UTC time window.</p></li></ul></section><section id="investigation-workflow"><header><h2><a href="#id2">2 Investigation workflow</a></h2></header><p>The diagram below represents the full workflow from the launch of an action by
|
||||
an investigation, to the retrieval of results from the database. The steps are
|
||||
explained in the legend of the diagram, and map to various components of MIG.</p><p>Actions are submitted to the API by trusted investigators. PGPSignatures are
|
||||
verified by the API and each agent prior to running any command.</p><p>View <a class="reference external" href=".files/action_command_flow.svg">full size diagram</a>.</p><object data=".files/action_command_flow.svg" type="image/svg+xml">.files/action_command_flow.svg</object><section id="command-execution-flow-in-agent-and-modules"><header><h3><a href="#id4">2.1 Command execution flow in Agent and Modules</a></h3></header><p>Upon processing of an action, the scheduler will retrieve a list of agents to
|
||||
send the action to. One action is then derived into multiple commands and sent
|
||||
to agents.</p><p>An agent receives a command from the scheduler on its personal AMQP queue (1).
|
||||
It parses the command (2) and extracts all of the operations to perform.
|
||||
Operations are passed to modules and executed in parallel (3). Rather than
|
||||
maintaining a state of the running command, the agent create a goroutine and a
|
||||
channel tasked with receiving the results from the modules. Each modules
|
||||
published its results inside that channel (4). The result parsing goroutine
|
||||
receives them, and when it has received all of them, populates the <cite>results</cite> (5)
|
||||
array of the command with the results from each module, and send the command
|
||||
back to the scheduler(6).</p><p>When the agent is done running the command, both the channel and the goroutine
|
||||
are destroyed.</p><blockquote><pre> +-------+ [ - - - - - - A G E N T - - - - - - - - - - - - ]
|
||||
|command|+---->(listener)
|
||||
+-------+ |(2)
|
||||
^ V
|
||||
|(1) (parser)
|
||||
| + [ m o d u l e s ]
|
||||
+---------+ | (3)|----------> op1 +----------------+
|
||||
|SCHEDULER|+---+ |------------> op2 +--------------|
|
||||
| |<---+ |--------------> op3 +------------|
|
||||
+---------+ | +----------------> op4 +----------+
|
||||
| V(4)
|
||||
|(6) (receiver)
|
||||
| |
|
||||
| V(5)
|
||||
+ (publisher)
|
||||
+-------+ /
|
||||
|results|<-----------------------------------------'
|
||||
+-------+</pre></blockquote><p>The command received by the agent is composed of a copy of the action described
|
||||
previously, but signed with the private key of a trusted investigator. It also
|
||||
contains additional parameters that are specific to the targetted agent, such as
|
||||
command processing timestamps, name of the agent queue on the message broker,
|
||||
action and command unique IDs and status and results of the command. Below is an
|
||||
command derived from the root password checking action, and ran on the host named
|
||||
'host1.example.net'.</p><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"id"</span><span class="punctuation">:</span> <span class="literal number float">1.427392971126604e+18</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"action"</span><span class="punctuation">:</span> <span class="punctuation">{</span> <span class="error">...</span> <span class="error">SIGNED</span> <span class="error">COPY</span> <span class="error">OF</span> <span class="error">THE</span> <span class="error">ACTION</span> <span class="error">...</span> <span class="punctuation">},</span>
|
||||
<span class="name tag">"agent"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"id"</span><span class="punctuation">:</span> <span class="literal number float">1.4271760437936648e+18</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"host1.example.net"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"queueloc"</span><span class="punctuation">:</span> <span class="literal string double">"linux.host1.example.net.981alsd19aos1984"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"mode"</span><span class="punctuation">:</span> <span class="literal string double">"daemon"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"20150324+0d0f88c.prod"</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"status"</span><span class="punctuation">:</span> <span class="literal string double">"success"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"results"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"foundanything"</span><span class="punctuation">:</span> <span class="keyword constant">true</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"success"</span><span class="punctuation">:</span> <span class="keyword constant">true</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"elements"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"root_passwd_hashed_or_disabled"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"file"</span><span class="punctuation">:</span> <span class="literal string double">"/etc/shadow"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"fileinfo"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"lastmodified"</span><span class="punctuation">:</span> <span class="literal string double">"2015-02-07 01:51:07.17850601 +0000 UTC"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"mode"</span><span class="punctuation">:</span> <span class="literal string double">"----------"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"size"</span><span class="punctuation">:</span> <span class="literal number integer">1684</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"search"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"contents"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="literal string double">"root:(\\*|!|\\$(1|2a|5|6)\\$).+"</span>
|
||||
<span class="punctuation">],</span>
|
||||
<span class="name tag">"options"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"matchall"</span><span class="punctuation">:</span> <span class="keyword constant">false</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"matchlimit"</span><span class="punctuation">:</span> <span class="literal number integer">0</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"maxdepth"</span><span class="punctuation">:</span> <span class="literal number integer">0</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"paths"</span><span class="punctuation">:</span> <span class="punctuation">[</span>
|
||||
<span class="literal string double">"/etc"</span>
|
||||
<span class="punctuation">]</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">]</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"statistics"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"exectime"</span><span class="punctuation">:</span> <span class="literal string double">"2.017849ms"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"filescount"</span><span class="punctuation">:</span> <span class="literal number integer">1</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"openfailed"</span><span class="punctuation">:</span> <span class="literal number integer">0</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"totalhits"</span><span class="punctuation">:</span> <span class="literal number integer">1</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"errors"</span><span class="punctuation">:</span> <span class="keyword constant">null</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">],</span>
|
||||
<span class="name tag">"starttime"</span><span class="punctuation">:</span> <span class="literal string double">"2015-03-26T18:02:51.126605Z"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"finishtime"</span><span class="punctuation">:</span> <span class="literal string double">"2015-03-26T18:03:00.671232Z"</span>
|
||||
<span class="punctuation">}</span></code></pre><p>The results of the command show that the file '/etc/shadow' has matched, and
|
||||
thus "FoundAnything" returned "True".</p><p>The invocation of the file module has completed successfully, which is
|
||||
represented by <strong>results->0->success=true</strong>. In our example, there is only one
|
||||
operation in the <strong>action->operations</strong> array, so only one result is present.
|
||||
When multiple operations are performed, each has its results listed in a
|
||||
corresponding entry of the results array (operations[0] is in results[0],
|
||||
operations[1] in results[1], etc...).</p><p>Finally, the agent has performed all operations in the operations array
|
||||
successfully, and returned <strong>status=success</strong>. Had a failure happened on the
|
||||
agent, the returned status would be one of "failed", "timeout" or "cancelled".</p></section><section id="command-expiration-timeouts"><header><h3><a href="#id5">2.2 Command expiration & timeouts</a></h3></header><p>To prevent abuse of resources, agents will kill long-running modules after a
|
||||
given period of time. That timeout is hardcoded in the agent configuration
|
||||
at compile time and defaults to 5 minutes.</p><pre><code class="code go"><span class="comment single">// timeout after which a module run is killed
|
||||
</span><span class="keyword declaration">var</span> <span class="name other">MODULETIMEOUT</span> <span class="name other">time</span><span class="punctuation">.</span><span class="name other">Duration</span> <span class="punctuation">=</span> <span class="literal number integer">300</span> <span class="operator">*</span> <span class="name other">time</span><span class="punctuation">.</span><span class="name other">Second</span></code></pre><p>That timeout represents the <strong>maximum</strong> execution time of a single operation. If
|
||||
an action contains 3 operations, each operation gets its own timeout. But because
|
||||
operations run in parallel in the agent, the maximum runtime of an action should
|
||||
be very close to the value of MODULETIMEOUT.</p><p>In a typical deployment, it is safe to increase MODULETIMEOUT to allow for
|
||||
longer operations. A value of 20 minutes is usual. Make sure to fine tune this
|
||||
to your environment, and get the approval of your ops team because mig-agent
|
||||
may end up consuming resources (but never more than 50% of the cpu available on
|
||||
a system).</p><p>Oftentimes, an investigator will want a timeout that is much shorter than the value
|
||||
of MODULETIMEOUT. In the MIG command line, the flag <cite>-e</cite> controls the
|
||||
expiration. It defaults to 5 minutes but can be set to 30 seconds for simple
|
||||
investigations. When that happens, the agent will calculate an appropriate expiration
|
||||
for the operations being run. If the expiration set on the action is set to 30 seconds,
|
||||
the agent will kill operations that run for more than 30 seconds.</p><p>If the expiration is larger than the value of MODULETIMEOUT (for example, 2
|
||||
hours), then MODULETIMEOUT is used. Setting a long expiration may be useful to
|
||||
allow agents that only check in periodically to pick up actions long after they
|
||||
are launched.</p></section></section><section id="access-control-lists"><header><h2><a href="#id6">3 Access Control Lists</a></h2></header><p>Not all keys can perform all actions. The scheduler, for example, sometimes need
|
||||
verified by the API and each agent prior to running any command.</p><p>View <a class="reference external" href=".files/action_command_flow.svg">full size diagram</a>.</p><object data=".files/action_command_flow.svg" type="image/svg+xml">.files/action_command_flow.svg</object></section><section id="access-control-lists"><header><h2><a href="#id3">3 Access Control Lists</a></h2></header><p>Not all keys can perform all actions. The scheduler, for example, sometimes need
|
||||
to issue specific actions to agents (such as during the upgrade protocol) but
|
||||
shouldn't be able to perform more dangerous actions. This is enforced by
|
||||
an Access Control List, or ACL, stored on the agents. An ACL describes who can
|
||||
|
|
@ -471,127 +362,19 @@ has the following syntax:</p><pre><code class="code json"><span class="punctuati
|
|||
<span class="punctuation">}</span></code></pre><p>The <cite>default</cite> permission is overridden by module specific permissions.</p><p>The ACL is currently applied to modules. In the future, ACL will have finer
|
||||
control to authorize access to specific functions of modules. For example, an
|
||||
investigator could be authorized to call the <cite>regex</cite> function of filechecker
|
||||
module, but only in <cite>/etc</cite>. This functionality is not implemented yet.</p></section><section id="agent-initialization-process"><header><h2><a href="#id7">4 Agent initialization process</a></h2></header><p>The agent tries to be as autonomous as possible. One of the goal is to ship
|
||||
agents without requiring external provisioning tools, such as Chef or Puppet.
|
||||
Therefore, the agent attempts to install itself as a service, and also supports
|
||||
a builtin upgrade protocol (described in the next section).</p><p>As a portable binary, the agent needs to detect the type of operating system
|
||||
and init method that is used by an endpoint. Depending on the endpoint,
|
||||
different initialization methods are used. The diagram below explains the
|
||||
decision process followed by the agent.</p><img alt=".files/mig-agent-initialization-process.png" src=".files/mig-agent-initialization-process.png"><p>Go does not provide support for running programs in the backgroud. On endpoints
|
||||
that run upstart, systemd (linux) or launchd (darwin), this is not an issue
|
||||
because the init daemon takes care of running the agent in the background,
|
||||
rerouting its file descriptors and restarting on crash. On Windows and System-V,
|
||||
however, the agent daemonizes by forking itself into <cite>foreground</cite> mode, and
|
||||
re-forking itself on error (such as loss of connectivity to the relay).
|
||||
On Windows and System-V, if the agent is killed, it will not be restarted
|
||||
automatically.</p><section id="registration-process"><header><h3><a href="#id8">4.1 Registration process</a></h3></header><p>The initialization process goes through several environment detection steps
|
||||
which are used to select the proper init method. Once started, the agent will
|
||||
send a heartbeat to the public relay, and also store that heartbeat in its
|
||||
<cite>run</cite> directory. The location of the <cite>run</cite> directory is platform specific.</p><ul><li><p>windows: C:Windows</p></li><li><p>darwin: /Library/Preferences/mig/</p></li><li><p>linux: /var/run/mig/</p></li></ul><p>Below is a sample heartbeat message from a linux agent stored in
|
||||
<cite>/var/run/mig/mig-agent.ok</cite>.</p><pre><code class="code json"><span class="punctuation">{</span>
|
||||
<span class="name tag">"destructiontime"</span><span class="punctuation">:</span> <span class="literal string double">"0001-01-01T00:00:00Z"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"environment"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"arch"</span><span class="punctuation">:</span> <span class="literal string double">"amd64"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"ident"</span><span class="punctuation">:</span> <span class="literal string double">"Red Hat Enterprise Linux Server release 6.5 (Santiago)"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"init"</span><span class="punctuation">:</span> <span class="literal string double">"upstart"</span>
|
||||
<span class="punctuation">},</span>
|
||||
<span class="name tag">"heartbeatts"</span><span class="punctuation">:</span> <span class="literal string double">"2014-07-31T14:00:20.00442837-07:00"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"name"</span><span class="punctuation">:</span> <span class="literal string double">"someserver.example.net"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"os"</span><span class="punctuation">:</span> <span class="literal string double">"linux"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"pid"</span><span class="punctuation">:</span> <span class="literal number integer">26256</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"queueloc"</span><span class="punctuation">:</span> <span class="literal string double">"linux.someserver.example.net.5hsa811oda"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"starttime"</span><span class="punctuation">:</span> <span class="literal string double">"2014-07-30T21:34:48.525449401-07:00"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"version"</span><span class="punctuation">:</span> <span class="literal string double">"201407310027+bcbdd94.prod"</span>
|
||||
<span class="punctuation">}</span></code></pre></section><section id="check-in-mode"><header><h3><a href="#id9">4.2 Check-In mode</a></h3></header><p>In infrastructure where running the agent as a permanent process is not
|
||||
acceptable, it is possible to run the agent as a cron job. By starting the
|
||||
agent with the flag <strong>-m agent-checkin</strong>, the agent will connect to the
|
||||
configured relay, retrieve and run outstanding commands, and exit after 10
|
||||
seconds of inactivity.</p></section></section><section id="agent-upgrade-process"><header><h2><a href="#id10">5 Agent upgrade process</a></h2></header><p>MIG supports upgrading agents in the wild. The upgrade protocol is designed with
|
||||
security in mind. The flow diagram below presents a high-level view:</p><blockquote><pre>Investigator Scheduler Agent NewAgent FileServer
|
||||
+-----------+ +-------+ +---+ +------+ +--------+
|
||||
| | | | |
|
||||
| 1.initiate | | | |
|
||||
|------------------>| | | |
|
||||
| | 2.send command | | |
|
||||
| |------------------>| 3.verify | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 4.download | |
|
||||
| | |-------------------------------------->|
|
||||
| | | | |
|
||||
| | | 5.checksum | |
|
||||
| | |--------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| | |<-------+ | |
|
||||
| | | | |
|
||||
| | | 6.exec | |
|
||||
| | |------------------>| |
|
||||
| | 7.return own PID | | |
|
||||
| |<------------------| | |
|
||||
| | | | |
|
||||
| |------+ 8.mark | | |
|
||||
| | | agent as | | |
|
||||
| | | upgraded | | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 9.register | | |
|
||||
| |<--------------------------------------| |
|
||||
| | | | |
|
||||
| |------+10.find dup | | |
|
||||
| | |agents in | | |
|
||||
| | |registrations | |
|
||||
| |<-----+ | | |
|
||||
| | | | |
|
||||
| | 11.send command to kill PID old agt| |
|
||||
| |-------------------------------------->| |
|
||||
| | | | |
|
||||
| | 12.acknowledge | | |
|
||||
| |<--------------------------------------| |</pre></blockquote><p>All upgrade operations are initiated by an investigator (1). The upgrade is
|
||||
triggered by an action to the upgrade module with the following parameters:</p><pre><code class="code json"><span class="literal string double">"Operations"</span><span class="error">:</span> <span class="punctuation">[</span>
|
||||
<span class="punctuation">{</span>
|
||||
<span class="name tag">"Module"</span><span class="punctuation">:</span> <span class="literal string double">"upgrade"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"Parameters"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"linux/amd64"</span><span class="punctuation">:</span> <span class="punctuation">{</span>
|
||||
<span class="name tag">"to_version"</span><span class="punctuation">:</span> <span class="literal string double">"16eb58b-201404021544"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"location"</span><span class="punctuation">:</span> <span class="literal string double">"http://localhost/mig/bin/linux/amd64/mig-agent"</span><span class="punctuation">,</span>
|
||||
<span class="name tag">"checksum"</span><span class="punctuation">:</span> <span class="literal string double">"31fccc576635a29e0a27bbf7416d4f32a0ebaee892475e14708641c0a3620b03"</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">}</span>
|
||||
<span class="punctuation">]</span><span class="error">,</span></code></pre><ul><li><p>Each OS family and architecture have their own parameters (ex: "linux/amd64",
|
||||
"darwin/amd64", "windows/386", ...). Then, in each OS/Arch group, we have:</p></li><li><p>to_version is the version an agent should upgrade to</p></li><li><p>location points to a HTTPS address that contains the agent binary</p></li><li><p>checksum is a SHA256 hash of the agent binary to be verified after download</p></li></ul><p>The parameters above are signed using a standard PGP action signature.</p><p>The upgrade action is forwarded to agents (2) like any other action. The action
|
||||
signature is verified by the agent (3), and the upgrade module is called. The
|
||||
module downloads the new binary (4), verifies the version and checksum (5) and
|
||||
installs itself on the system.</p><p>Assuming everything checks in, the old agent executes the binary of the new
|
||||
agent (6). At that point, two agents are running on the same machine, and the
|
||||
rest of the protocol is designed to shut down the old agent, and clean up.</p><p>After executing the new agent, the old agent returns a successful result to the
|
||||
scheduler, and includes its own PID in the results.
|
||||
The new agent starts by registering with the scheduler (7). This tells the
|
||||
scheduler that two agents are running on the same node, and one of them must
|
||||
terminate. The scheduler sends a kill action to both agents with the PID of the
|
||||
old agent (8). The kill action may be executed twice, but that doesn't matter.
|
||||
When the scheduler receives the kill results (9), it sends a new action to check
|
||||
for <cite>mig-agent</cite> processes (10). Only one should be found in the results (11),
|
||||
and if that is the case, the scheduler tells the agent to remove the binary of
|
||||
the old agent (12). When the agent returns (13), the upgrade protocol is done.</p><p>If the PID of the old agent lingers on the system, an error is logged for the
|
||||
investigator to decide what to do next. The scheduler does not attempt to clean
|
||||
up the situation.</p></section><section id="threat-model"><header><h2><a href="#id11">6 Threat Model</a></h2></header><p>Running an agent as root on a large number of endpoints means that Mozilla
|
||||
module, but only in <cite>/etc</cite>. This functionality is not implemented yet.</p></section><section id="threat-model"><header><h2><a href="#id4">4 Threat Model</a></h2></header><p>Running an agent as root on a large number of endpoints means that Mozilla
|
||||
InvestiGator is a target of choice to compromise an infrastructure.
|
||||
Without proper protections, a vulnerability in the agent or in the platform
|
||||
could lead to a compromission of the endpoints.</p><p>The architectural choices made in MIG diminish the exposure of the endpoints to
|
||||
a compromise. And while the risk cannot be reduced to zero entirely, it would
|
||||
take an attacker direct control on the investigators key material, or be root
|
||||
on the infrastructure in order to take control of MIG.</p><p>MIG's security controls include:</p><ul><li><p>Strong GPG security model</p></li><li><p>Infrastructure resiliency</p></li><li><p>No port listening</p></li><li><p>Protection of connections to the relays</p></li><li><p>Randomization of the queue names</p></li><li><p>Whitelisting of agents</p></li><li><p>Limit data extraction to a minimum</p></li></ul><section id="strong-gpg-security-model"><header><h3><a href="#id12">6.1 Strong GPG security model</a></h3></header><p>All actions that are passed to the MIG platform and to the agents require
|
||||
on the infrastructure in order to take control of MIG.</p><p>MIG's security controls include:</p><ul><li><p>Strong GPG security model</p></li><li><p>Infrastructure resiliency</p></li><li><p>No port listening</p></li><li><p>Protection of connections to the relays</p></li><li><p>Randomization of the queue names</p></li><li><p>Whitelisting of agents</p></li><li><p>Limit data extraction to a minimum</p></li></ul><section id="strong-gpg-security-model"><header><h3><a href="#id5">4.1 Strong GPG security model</a></h3></header><p>All actions that are passed to the MIG platform and to the agents require
|
||||
valid GPG signatures from one or more trusted investigators. The public keys of
|
||||
trusted investigators are hardcoded in the agents, making it almost impossible
|
||||
to override without root access to the endpoints, or access to an investigator's
|
||||
private key. The GPG private keys are never seen by the MIG platform (API,
|
||||
Scheduler, Database or Relays). A compromise of the platform would not lead to
|
||||
an attacker taking control of the agents and compromising the endpoints.</p></section><section id="infrastructure-resiliency"><header><h3><a href="#id13">6.2 Infrastructure resiliency</a></h3></header><p>One of the design goal of MIG is to make each components as stateless as
|
||||
an attacker taking control of the agents and compromising the endpoints.</p></section><section id="infrastructure-resiliency"><header><h3><a href="#id6">4.2 Infrastructure resiliency</a></h3></header><p>One of the design goal of MIG is to make each components as stateless as
|
||||
possible. The database is used as a primary data store, and the schedulers and
|
||||
relays keep data in transit in their respective cache. But any of these
|
||||
components can go down and be rebuilt without compromising the resiliency of
|
||||
|
|
@ -605,12 +388,12 @@ relay whenever they chose to, and pick up outstanding tasks.</p><p>If the relays
|
|||
reconnect at regular intervals continuously. It is trivial to rebuild
|
||||
a fresh rabbitmq cluster, even on a new IP space, as long as the FQDN of the
|
||||
cluster, and the TLS cert/key and credentials of the AMQPS access point
|
||||
remain the same.</p></section><section id="no-port-listening"><header><h3><a href="#id14">6.3 No port listening</a></h3></header><p>The agents do not accept incoming connections. There is no listening port that
|
||||
remain the same.</p></section><section id="no-port-listening"><header><h3><a href="#id7">4.3 No port listening</a></h3></header><p>The agents do not accept incoming connections. There is no listening port that
|
||||
an attacker could use to exploit a vulnerability in the agent. Instead, the
|
||||
agent connects to the platform by establishing an outbound connection to the
|
||||
relays. The connection uses TLS, making it theorically impossible for an
|
||||
attacker to MITM without access to the PKI and DNS, both of which are not
|
||||
part of the MIG platform.</p></section><section id="protection-of-connections-to-the-relays"><header><h3><a href="#id15">6.4 Protection of connections to the relays</a></h3></header><p>The rabbitmq relay of a MIG infrastructure may very well be listening on the
|
||||
part of the MIG platform.</p></section><section id="protection-of-connections-to-the-relays"><header><h3><a href="#id8">4.4 Protection of connections to the relays</a></h3></header><p>The rabbitmq relay of a MIG infrastructure may very well be listening on the
|
||||
public internet. This is used when MIG agents are distributed into various
|
||||
environments, as opposed to concentrated on a single network location. RabbitMQ
|
||||
and Erlang provide a stable network stack, but are not shielded from a network
|
||||
|
|
@ -623,7 +406,7 @@ agent will present a username and password to open the AMQP connection. Again,
|
|||
these credentials are shared across all agents, and are not used to authenticate
|
||||
individual agents. Their role is to assign an ACL to the agent.
|
||||
The ACL limits the AMQP action an agent can perform on the cluster.
|
||||
See <a class="reference external" href="configuration.rst">rabbitmq configuration</a> for more information.</p></section><section id="randomization-of-the-queue-names"><header><h3><a href="#id16">6.5 Randomization of the queue names</a></h3></header><p>The protections above limit the exposure of the AMQP endpoint, but since the
|
||||
See <a class="reference external" href="configuration.rst">rabbitmq configuration</a> for more information.</p></section><section id="randomization-of-the-queue-names"><header><h3><a href="#id9">4.5 Randomization of the queue names</a></h3></header><p>The protections above limit the exposure of the AMQP endpoint, but since the
|
||||
secrets are shared across all agents, the possibility still exists that an
|
||||
attacker gains access to the secrets, and establish a connection to the relays.</p><p>Such access would have very limited capabilities. It cannot be used to publish
|
||||
commands to the agents, because publication is ACL-limited to the scheduler.
|
||||
|
|
@ -631,7 +414,7 @@ It can be used to publish fake results to the scheduler, or listen on the
|
|||
agent queue for incoming commands.</p><p>Both are made difficult by prepending a random number to the name of an agent
|
||||
queue. An agent queue is named using the following scheme:</p><blockquote><p><cite>mig.agt.<OS family>.<Hostname>.<uid></cite></p></blockquote><p>The OS and hostname of a given agent are easy to guess, but the uid isn't.
|
||||
The UID is a 64 bits integer composed of nanosecond timestamps and a random 32
|
||||
bits integer, chosen by the agent on first start. It is specific to an endpoint.</p></section><section id="whitelisting-of-agents"><header><h3><a href="#id17">6.6 Whitelisting of agents</a></h3></header><p>At the moment, MIG does not provide a strong mechanism to authenticate agents.
|
||||
bits integer, chosen by the agent on first start. It is specific to an endpoint.</p></section><section id="whitelisting-of-agents"><header><h3><a href="#id10">4.6 Whitelisting of agents</a></h3></header><p>At the moment, MIG does not provide a strong mechanism to authenticate agents.
|
||||
It is a work in progress, but for now agents are whitelisted in the scheduler
|
||||
using the queuelocs that are advertised in the heartbeat messages. Spoofing the
|
||||
queueloc string is difficult, because it contains a random value that is
|
||||
|
|
@ -640,7 +423,7 @@ order to spoof an agent's identity. This method provides a basic access control
|
|||
mechanism. The long term goal is to allow the scheduler to call an external database
|
||||
to authorize agents. In AWS, the scheduler could call the AWS API to verify that
|
||||
a given agent does indeed exist in the infrastructure. In a traditional datacenter,
|
||||
this could be an inventory database.</p></section><section id="limit-data-extraction-to-a-minimum"><header><h3><a href="#id18">6.7 Limit data extraction to a minimum</a></h3></header><p>Agents are not <cite>meant</cite> to retrieve raw data from their endpoints. This is more
|
||||
this could be an inventory database.</p></section><section id="limit-data-extraction-to-a-minimum"><header><h3><a href="#id11">4.7 Limit data extraction to a minimum</a></h3></header><p>Agents are not <cite>meant</cite> to retrieve raw data from their endpoints. This is more
|
||||
of a good practice rather than a technical limitation. The modules shipped with
|
||||
the agent are meant to return boolean answers of the type "match" or "no match".</p><p>It could be argued that answering "match" on sensitive requests is similar to
|
||||
extracting data from the agents. MIG does not solve this issue.. It is the
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
Mozilla InvestiGator Deployment and Configuration Documentation
|
||||
===============================================================
|
||||
:Author: Julien Vehent <jvehent@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
|
|
|||
Различия файлов скрыты, потому что одна или несколько строк слишком длинны
|
|
@ -0,0 +1,495 @@
|
|||
===========
|
||||
MIG CONSOLE
|
||||
===========
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of contents
|
||||
|
||||
`mig-console` is a terminal-based client to the MIG API. It provides a complete
|
||||
access to data stored in MIG and exposed by the API, in an interface that is
|
||||
easy to use. Similar to the command line client, `mig-console` reads its
|
||||
configuration from $HOME/.migrc and will prompt the user to create one if it
|
||||
doesn't exists.
|
||||
|
||||
The base screen of the API presents a dashboard of the current state of the
|
||||
infrastructure. It includes the number of agents and endpoints, broken down
|
||||
by version of the agent ,and a list of the latest actions that have run.
|
||||
|
||||
Underneath the dasboard, the user is invited to interact in a prompt that
|
||||
starts with **mig>**.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ mig-console -c ~/.migrc-lnw
|
||||
|
||||
## ## _.---._ .---.
|
||||
# # # /-\ ---|| | /\ __...---' .---. '---'-. '.
|
||||
# #| | / || | /--\ .-''__.--' _.'( | )'. '. '._ :
|
||||
# # \_/ ---| \_ \_/ \ .'__-'_ .--'' ._'---'_.-. '. '-'.
|
||||
### ~ -._ -._''---. -. '-._ '.
|
||||
# |\ |\ /---------| ~ -.._ _ _ _ ..-_ '. '-._''--.._
|
||||
# | \| \ / |- |__ | | -~ -._ '-. -. '-._''--.._.--''.
|
||||
###| \ \/ ---__| | | ~ ~-.__ -._ '-.__ '. '. :
|
||||
##### ~~ ~---...__ _ ._ .' '.
|
||||
# /\ --- /-\ |--|---- ~ ~--.....--~
|
||||
# ### /--\ | | ||-\ //
|
||||
#####/ \ | \_/ | \//__
|
||||
+------
|
||||
| Agents & Endpoints summary:
|
||||
| * 2 online agents on 2 endpoints
|
||||
| * 0 idle agents on 0 endpoints
|
||||
| * 0 endpoints are running 2 or more agents
|
||||
| * 0 endpoints appeared over the last 7 days
|
||||
| * 0 endpoints disappeared over the last 7 days
|
||||
| * 0 endpoints have been flapping
|
||||
| Online agents by version:
|
||||
| * version 20150909+f2456f5.dev: 2 agents
|
||||
| Idle agents by version:
|
||||
|
|
||||
| Latest Actions:
|
||||
| ---- ID ---- + ---- Name ---- + -Sent- + ---- Date ---- + ---- Investigators ----
|
||||
| 4999271350274 file -c /home/ulfr/.migrc-l... 2 2015-09-29T15:40:35Z Julien Vehent
|
||||
| 4964811669519 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:37:16Z Julien Vehent
|
||||
| 4964811669506 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:37:03Z Julien Vehent
|
||||
| 4964764024853 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:25:26Z Julien Vehent
|
||||
| 4964764024834 file -c /home/ulfr/.migrc-l... 2 2015-09-23T13:24:57Z Julien Vehent
|
||||
| 4949328330767 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:59:39Z Julien Vehent
|
||||
| 4949328330754 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:59:25Z Julien Vehent
|
||||
| 4948324450316 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:45:51Z Julien Vehent
|
||||
| 4948324450307 file -c /home/ulfr/.migrc-l... 2 2015-09-20T19:33:17Z Julien Vehent
|
||||
| 4947944865794 file -c /home/ulfr/.migrc-l... 2 2015-09-20T14:07:36Z Julien Vehent
|
||||
+------
|
||||
|
||||
Connected to https://jaffa.linuxwall.info/api/v1/. Exit with ctrl+d. Type help for help.
|
||||
mig>
|
||||
|
||||
Entering **help** here, or on any other mode of the console, provides the user
|
||||
with a list of available functionalities::
|
||||
|
||||
mig> help
|
||||
The following orders are available:
|
||||
action <id> enter interactive action reader mode for action <id>
|
||||
agent <id> enter interactive agent reader mode for agent <id>
|
||||
create action create a new action
|
||||
create investigator create a new investigator, will prompt for name and public key
|
||||
command <id> enter command reader mode for command <id>
|
||||
exit leave
|
||||
help show this help
|
||||
history <count> print last <count> entries in history. count=10 by default.
|
||||
investigator <id> enter interactive investigator management mode for investigator <id>
|
||||
query <uri> send a raw query string, without the base url, to the api
|
||||
search <search> perform a search. see "search help" for more information.
|
||||
showcfg display running configuration
|
||||
status display platform status: connected agents, latest actions, ...
|
||||
|
||||
For example, let's review action id 4999271350274::
|
||||
|
||||
mig> action 4999271350274
|
||||
Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Action: 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '.
|
||||
Launched by 'Julien Vehent' on '2015-09-29 11:40:31.792627 -0400 EDT'.
|
||||
Status 'completed'.
|
||||
2 sent, 2 done, 2 succeeded
|
||||
action 274>
|
||||
|
||||
When entering the action reader mode, a short summary of the action is displayed
|
||||
to the user. We can see that the action was launched by "Julien Vehent" on 2 agents
|
||||
and that it completed. The prompt is changed to **action 274** to indicate that
|
||||
we are now in action reader mode. Entering **help** here provides a list of
|
||||
functionalities specific to that mode::
|
||||
|
||||
The following orders are available:
|
||||
command <id> jump to command reader mode for command <id>
|
||||
|
||||
copy enter action launcher mode using current action as template
|
||||
|
||||
counters display the counters of the action
|
||||
|
||||
details display the details of the action, including status & times
|
||||
|
||||
exit exit this mode (also works with ctrl+d)
|
||||
|
||||
help show this help
|
||||
|
||||
investigators print the list of investigators that signed the action
|
||||
|
||||
json show the json of the action
|
||||
|
||||
list <show> returns the list of commands with their status
|
||||
<show>: * set to "all" to get all results (default)
|
||||
* set to "found" to only display positive results
|
||||
* set to "notfound" for negative results
|
||||
list can be followed by a 'filter' pipe:
|
||||
ex: ls | grep server1.(dom1|dom2) | grep -v example.net
|
||||
|
||||
r refresh the action (get latest version from upstream)
|
||||
|
||||
results <show> <render> display results of all commands
|
||||
<show>: * set to "all" to get all results (default)
|
||||
* set to "found" to only display positive results
|
||||
* set to "notfound" for negative results
|
||||
<render>: * set to "text" to print results in console (default)
|
||||
* set to "map" to generate an open a google map
|
||||
|
||||
times show the various timestamps of the action
|
||||
|
||||
It is possible to review results by entering **results**. If we only want results
|
||||
from agents that have found *something*, we can use **results found**. If we
|
||||
want those results inside a Google Map, enter **results found map**.
|
||||
|
||||
The raw json of the action is available via **json**.
|
||||
|
||||
To get a list of agents on which the action ran, use **list all**. You can filter
|
||||
the list using **list found** and **list notfound** to only get agents that have,
|
||||
or have not, found something. This command also supports a very basic `grep`.
|
||||
|
||||
To continue, let's list the agents that have found *something*::
|
||||
|
||||
action 274> list found
|
||||
..
|
||||
---- Command ID ---- ---- Agent Name & ID----
|
||||
4999268991155 server1.example.net [4942082344151]
|
||||
4999268991154 server2.example.net [4942082360682]
|
||||
2 agents have found things
|
||||
|
||||
We can inspect the command that ran on server1 by entering its ID::
|
||||
|
||||
action 274> command 4999268991155
|
||||
Entering command reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Command 4999268991155 ran on agent 'server1.example.net' based on action 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '
|
||||
command 155>
|
||||
|
||||
As you can see, the console mode has changed from `action 274` to
|
||||
`command 155`. This mode has its own set of functionalities that you
|
||||
can explore via **help**.
|
||||
|
||||
Creating actions
|
||||
----------------
|
||||
|
||||
The console provides a fine grained action generation interface. There
|
||||
are two ways to create a new action:
|
||||
|
||||
1. From the **mig>** mode, create a new empty action using **create action**.
|
||||
|
||||
2. From the **action>** reader mode, when reviewing a previous action,
|
||||
copy it to a new action using **copy**.
|
||||
|
||||
Both methods enter the action launcher mode. Method 2 only enters this
|
||||
mode with an preset action, rather than an empty one::
|
||||
|
||||
mig> create action
|
||||
Entering action launcher with empty template
|
||||
Type exit or press ctrl+d to leave. help may help.
|
||||
launcher> help
|
||||
The following orders are available:
|
||||
addoperation <module> append a new operation of type <module> to the action operations
|
||||
listagents list agents targetted by an action
|
||||
deloperation <opnum> remove operation numbered <opnum> from operations array, count starts at zero
|
||||
details display the action details
|
||||
exit exit this mode
|
||||
help show this help
|
||||
json <pack> show the json of the action
|
||||
launch <nofollow> launch the action. to return before completion, add "nofollow"
|
||||
load <path> load an action from a file at <path>
|
||||
setname <name> set the name of the action
|
||||
settarget <target> set the target
|
||||
settimes <start> <stop> set the validity and expiration dates
|
||||
sign PGP sign the action
|
||||
times show the various timestamps of the action
|
||||
|
||||
Action parameters can be edited prior to launching it:
|
||||
|
||||
* **setname** sets the name field of the action to a new string::
|
||||
|
||||
launcher> setname Test action that pings google.com
|
||||
|
||||
* **settarget** sets the target of the action. The target is evaluated
|
||||
right away, and a list of targeted agents can be obtained via **listagents**::
|
||||
|
||||
launcher> settarget environment->>'os'='linux' and mode='daemon'
|
||||
2 agents will be targetted. To get the list, use 'listagents'
|
||||
|
||||
launcher> listagents
|
||||
---- ID ---- + ---- Name -------
|
||||
4942082360682 server1.example.net
|
||||
4942082344151 server2.example.net
|
||||
|
||||
* **settimes** defines the validity period of the action using
|
||||
<start> and <stop> parameters. <start> can be set to "now", and <stop>
|
||||
can be a duration relative to "now"::
|
||||
|
||||
launcher> settimes now +10m
|
||||
launcher> times
|
||||
Valid from '2015-10-06 13:09:36.189134664 +0000 UTC' until '2015-10-06 13:20:36.189134664 +0000 UTC'
|
||||
Started on '0001-01-01 00:00:00 +0000 UTC'
|
||||
Last updated '0001-01-01 00:00:00 +0000 UTC'
|
||||
Finished on '0001-01-01 00:00:00 +0000 UTC'
|
||||
|
||||
* **addoperation <module>** is used to add a new operation to the action.
|
||||
<module> must be a know MIG module, such as `file`, `netstat`, `memory`,
|
||||
`scribe`, `ping`, `timedrift`, etc... When adding an operation, the console
|
||||
enters a new module-specific mode that takes the operation parameters. For
|
||||
example, this is how to add a ping operation that sends 5 TCP requests to
|
||||
google.com::
|
||||
|
||||
launcher> addoperation ping
|
||||
Ping module checks connectivity between an endpoint and a remote host. It supports
|
||||
icmp, tcp and udp ping. See doc at http://mig.mozilla.org/doc/module_ping.html
|
||||
|
||||
d <ip/fqdn> Destination Address can be ipv4, ipv6 or FQDN
|
||||
example: d www.mozilla.org
|
||||
d 63.245.217.105
|
||||
|
||||
dp <port> For TCP and UDP, specifies the port to test connectivity to
|
||||
example: dp 53
|
||||
|
||||
p <protocol> Protocol to use for the ping. This can be "icmp", "tcp" or "udp"
|
||||
example: p udp
|
||||
|
||||
c <count> Number of ping/connection attempts. Defaults to 3.
|
||||
example: c 5
|
||||
|
||||
t <timeout> Connection timeout in seconds. Defaults to 5.
|
||||
example: t 10
|
||||
ping> p tcp
|
||||
ping> d google.com
|
||||
ping> c 5
|
||||
ping> done
|
||||
Inserting ping operation with parameters:
|
||||
{
|
||||
"module": "ping",
|
||||
"parameters": {
|
||||
"destination": google.com",
|
||||
"protocol": "tcp",
|
||||
"count": 5
|
||||
}
|
||||
}
|
||||
|
||||
* **deloperation** removes an operation from the action operations list.
|
||||
Use **json** to visualize the list, and remove an operation using its
|
||||
position in the list, starting with zero::
|
||||
|
||||
launcher> json
|
||||
{
|
||||
"id": 0,
|
||||
"name": "Test action that pings google.com",
|
||||
"target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
|
||||
"description": {},
|
||||
"threat": {},
|
||||
"validfrom": "2015-10-06T13:09:36.189134664Z",
|
||||
"expireafter": "2015-10-06T13:20:36.189134664Z",
|
||||
"operations": [
|
||||
{
|
||||
"module": "ping",
|
||||
"parameters": {
|
||||
"destination": "google.com",
|
||||
"protocol": "tcp",
|
||||
"count": 5
|
||||
}
|
||||
}
|
||||
],
|
||||
"pgpsignatures": null,
|
||||
"starttime": "0001-01-01T00:00:00Z",
|
||||
"finishtime": "0001-01-01T00:00:00Z",
|
||||
"lastupdatetime": "0001-01-01T00:00:00Z",
|
||||
"counters": {}
|
||||
}
|
||||
|
||||
launcher> deloperation 0
|
||||
|
||||
launcher> json
|
||||
{
|
||||
"id": 0,
|
||||
"name": "Test action that pings google.com",
|
||||
"target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
|
||||
"description": {},
|
||||
"threat": {},
|
||||
"validfrom": "2015-10-06T13:09:36.189134664Z",
|
||||
"expireafter": "2015-10-06T13:20:36.189134664Z",
|
||||
"operations": [],
|
||||
"pgpsignatures": null,
|
||||
"starttime": "0001-01-01T00:00:00Z",
|
||||
"finishtime": "0001-01-01T00:00:00Z",
|
||||
"lastupdatetime": "0001-01-01T00:00:00Z",
|
||||
"counters": {}
|
||||
}
|
||||
|
||||
When ready to launch the action, type **launch**. The console will enter
|
||||
follower mode and print the progress of the action. When completed, the
|
||||
console will directly enter action reader mode, where you can type **results**
|
||||
to view the results::
|
||||
|
||||
launcher> launch
|
||||
Action 'Test action that pings google.com' successfully launched with ID '5033038708749' on target 'environment->>'os'='linux' and mode='daemon''
|
||||
Following action ID 5033038708749.status=inflight...50%.status=completed
|
||||
- 100.0% done in 10.004244071s
|
||||
2 sent, 2 done, 2 succeeded
|
||||
|
||||
Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Action: 'Test action that pings google.com'.
|
||||
Launched by 'Julien Vehent' on '2015-10-06 09:12:31.608546 -0400 EDT'.
|
||||
Status 'completed'.
|
||||
2 sent, 2 done, 2 succeeded
|
||||
action 749> results
|
||||
server2.example.net ping of google.com failed. Target is no reachable.
|
||||
server2.example.net command success
|
||||
server1.example.net ping of google.com failed. Target is no reachable.
|
||||
server1.example.net command success
|
||||
2 agents have all results
|
||||
|
||||
Searching
|
||||
---------
|
||||
|
||||
Return to the base mode using ctrl+d or **exit**::
|
||||
|
||||
command 155> exit
|
||||
exit
|
||||
action 274> exit
|
||||
exit
|
||||
mig>
|
||||
|
||||
When in **mig>** mode, you can use the **search** functionality to
|
||||
explore passed actions and commands, or find investigators. Details
|
||||
of how to run searches can be obtains via **mig> search help**.
|
||||
|
||||
Searches can be slow if your dataset is very large, so make sure to
|
||||
use time windows and limits to help speed up a search::
|
||||
|
||||
mig> search action where investigatorname=%vehent% and agentname=server1% and after=2015-09-01T00:00:00Z and before=2015-10-01T00:00:00Z
|
||||
Searching action after 2015-09-01T00:00:00Z and before 2015-10-01T00:00:00Z, limited to 100 results
|
||||
----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Investigators ---- + - Sent - + - Status - + --- Last Updated ---
|
||||
4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-29T15:40:35Z
|
||||
4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:16Z
|
||||
4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:03Z
|
||||
4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:25:26Z
|
||||
4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:24:57Z
|
||||
4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:39Z
|
||||
4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:25Z
|
||||
4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:45:51Z
|
||||
4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:33:17Z
|
||||
4947944865794 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T14:07:36Z
|
||||
4947909869570 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:58:41Z
|
||||
4947901022223 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:42Z
|
||||
4947901022210 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:26Z
|
||||
4947890798596 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:55:02Z
|
||||
4885615083769 timedrift -c /home/ulfr/.mi... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:02:56Z
|
||||
4885615083755 pkg -c /home/ulfr/.migrc-ln... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:33Z
|
||||
4885615083739 memory -c /home/ulfr/.migrc... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:04Z
|
||||
4885615083724 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T16:58:00Z
|
||||
|
||||
Managing investigators
|
||||
----------------------
|
||||
|
||||
The console can be used to create new investigators, active and
|
||||
disable them, and well as review their activity.
|
||||
|
||||
To review the activity of an investigator, find its ID using the
|
||||
search command and enter investigator mode::
|
||||
|
||||
mig> search investigator where name=%vehent%
|
||||
Searching investigator after 2011-03-31T13:16:55Z and before 2020-04-12T13:16:55Z, limited to 100 results
|
||||
- ID - + ---- Name ---- + --- Status ---
|
||||
2 Julien Vehent active
|
||||
|
||||
mig> investigator 2
|
||||
Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Investigator 2 named 'Julien Vehent'
|
||||
|
||||
inv 2> help
|
||||
The following orders are available:
|
||||
details print the details of the investigator
|
||||
exit exit this mode
|
||||
help show this help
|
||||
lastactions <limit> print the last actions ran by the investigator. limit=10 by default.
|
||||
pubkey show the armored public key of the investigator
|
||||
r refresh the investigator (get latest version from upstream)
|
||||
setstatus <status> changes the status of the investigator to <status> (can be 'active' or 'disabled')
|
||||
|
||||
The command **lastactions** will print the latest activity of this investigator::
|
||||
|
||||
inv 2> lastactions
|
||||
----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Date ---- + -- Status --
|
||||
5033038708749 Test action that pings goog... environment->>'os'='linux' ... 2015-10-06T09:12:31-04:00 completed
|
||||
4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-29T11:40:31-04:00 completed
|
||||
4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:37:12-04:00 completed
|
||||
4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:36:59-04:00 completed
|
||||
4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:25:22-04:00 inflight
|
||||
4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:24:52-04:00 inflight
|
||||
4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:35-04:00 completed
|
||||
4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:21-04:00 completed
|
||||
4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:45:47-04:00 completed
|
||||
4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:33:13-04:00 completed
|
||||
|
||||
To disable him, use **setstatus disabled**. Disabled investigators are no longer
|
||||
allowed to send investigations via the API::
|
||||
|
||||
inv 2> setstatus disabled
|
||||
Investigator status set to disabled
|
||||
|
||||
inv 2> details
|
||||
Investigator ID 2
|
||||
name Julien Vehent
|
||||
status disabled
|
||||
key id E60892BB9BD89A69F759A1A0A3D652173B763E8F
|
||||
created 2015-09-09 09:53:28.989481 -0400 EDT
|
||||
modified 2015-09-09 09:53:28.989481 -0400 EDT
|
||||
|
||||
inv 2> setstatus active
|
||||
Investigator status set to active
|
||||
|
||||
Creating investigators
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To create a new investigator, go back to **mig>** mode and type
|
||||
**create investigator**. The console will prompt the name of the new
|
||||
investigator as well as the location of her public PGP key. You can either
|
||||
provide a local path to the public key file on disk, on provide a fingerprint
|
||||
in the format "0x<40 char sha1 hash>". When a fingerprint is provided, the
|
||||
console will attempt to retrieve the key from the keyserver `gpg.mozilla.org`::
|
||||
|
||||
mig> create investigator
|
||||
Entering investigator creation mode. Please provide the full name
|
||||
and the public key of the new investigator.
|
||||
name> Bob Kelso
|
||||
Name: 'Bob Kelso'
|
||||
Please provide a public key. You can either provide a local path to the
|
||||
armored public key file, or a full length PGP fingerprint.
|
||||
example:
|
||||
pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
retrieving public key from http://gpg.mozilla.org
|
||||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||||
Version: SKS 1.1.5
|
||||
Comment: Hostname: keyserver.mozilla.org
|
||||
|
||||
mQENBEbv+5sBCADNHPvUIajRoxb/qylLrzwm9e+9sB8R/jhY4gxOzGZRDHECPvNeTUd9eogV
|
||||
n24rQDTWowkE+t9sW7vlD3TUWdBEAhXEpDZBfzlTBWIzEb1m3hwPOQM10ZNX6jPS1WlGsfoE
|
||||
LsUC0HmFTtOx4b5os9mIYbsjsDWd/JZjn0yUIv4eb28+fle6BkbgqIotLW4d1gTrxVlFc3be
|
||||
m+4OqimQ/v2LZDV+uObEkbh4UvmTtOCCx8zAOyZohPmICUbmJBc8KWWhzLOo8b9ns/GqP41q
|
||||
/9IuTQDXP2GUAKXzBKSdQiNzJP8Skfu4tPyGsGJSErprPC9t43HPPUgfeW9/sfuaw+vnABEB
|
||||
AAG0JUp1bGllbiBWRUhFTlQgPGp1bGllbkBsaW51eHdhbGwuaW5mbz6JATYEEwECACAFAkbv
|
||||
+5sCGy8GCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRBkUJDmQ2dze5U9CADK4Z6X02TP9afJ
|
||||
AyWF32zM3UdMksJ/F2wuo2HBHT0iOomw4ecNzHyO1P5BTglm5LC5ZZrV+Dx6Jve75JiSDTSD
|
||||
V3AhpR+M83rw8YKkeUrbTvfsy3+qhB7HYNIbCKT0lgAAL05SmDnYwYMQIV+p3T0F8BgGhkGT
|
||||
vHdKLhzEhKNOMaMCwCd1SsiBepA976oBUp9h5Vt6TEFyG6hCcFP90DFjNlK17yMNjbdrgyd6
|
||||
FkGEePKK3RhaLxcPShAgCnYzYYMABLYu1ow5AxxtEJTBHkJFkIzE9XM/lmyekYhWfX/Q4jpn
|
||||
1aiiVlf2klZlFSFy/TXuuQH1JO3YlKo9uHjlvSQb
|
||||
=+Ru1
|
||||
-----END PGP PUBLIC KEY BLOCK-----
|
||||
|
||||
create investigator? (y/n)> y
|
||||
Investigator 'Bob Kelso' successfully created with ID 4
|
||||
|
||||
mig> investigator 4
|
||||
Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Investigator 4 named 'Bob Kelso'
|
||||
|
||||
inv 4> details
|
||||
Investigator ID 4
|
||||
name Bob Kelso
|
||||
status active
|
||||
key id 716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
created 2015-10-06 09:23:14.473307 -0400 EDT
|
||||
modified 2015-10-06 09:23:14.473307 -0400 EDT
|
||||
|
||||
The new investigator now has access to the API.
|
||||
|
|
@ -0,0 +1,605 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
color: #eaeaea;
|
||||
font: 16px;
|
||||
line-height: 1.5em;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
}
|
||||
|
||||
#table-of-contents ul {
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
/* General & 'Reset' Stuff */
|
||||
|
||||
.container {
|
||||
width: 95%;
|
||||
max-width: 1000px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
section {
|
||||
display: block;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
* header - container
|
||||
* h1 - project name
|
||||
* h2 - project description
|
||||
* */
|
||||
|
||||
header {
|
||||
background: rgba(0, 0, 0, 0.1);
|
||||
width: 100%;
|
||||
/*border-bottom: 1px dashed #b5e853;*/
|
||||
/*padding: 20px 0;
|
||||
* margin: 0 0 40px 0;*/
|
||||
padding: 5px 0;
|
||||
margin: 0 0 10px 0;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 30px;
|
||||
line-height: 1.5;
|
||||
margin: 0 0 0 -40px;
|
||||
font-weight: bold;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
/*color: #b5e853;*/
|
||||
color: #089d00;
|
||||
text-shadow: 0 1px 1px rgba(0, 0, 0, 0.1),
|
||||
0 0 5px rgba(181, 232, 83, 0.1),
|
||||
0 0 10px rgba(181, 232, 83, 0.1);
|
||||
letter-spacing: -1px;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
header h1:before {
|
||||
content: "./ ";
|
||||
font-size: 24px;
|
||||
}
|
||||
|
||||
header h2 {
|
||||
font-size: 18px;
|
||||
font-weight: 300;
|
||||
}
|
||||
|
||||
/* Main Content
|
||||
* */
|
||||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
max-width: 100%
|
||||
}
|
||||
|
||||
h2 a {
|
||||
font-weight: bold;
|
||||
color: #8AB638;
|
||||
line-height: 1.4em;
|
||||
font-size: 1.4em;
|
||||
}
|
||||
h3 a, h4 a, h5 a, h6 a {
|
||||
font-weight: bold;
|
||||
color: #934500;
|
||||
line-height: 1.4em;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 30px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 28px;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
h4 {
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
h5 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
h6 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
color: #999;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
dt {
|
||||
font-style: italic;
|
||||
font-weight: bold;
|
||||
}
|
||||
/*
|
||||
ul li {
|
||||
list-style: none;
|
||||
}
|
||||
*/
|
||||
/*
|
||||
ul li:before {
|
||||
content: ">>";
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
font-size: 13px;
|
||||
color: #b5e853;
|
||||
margin-left: -37px;
|
||||
margin-right: 21px;
|
||||
line-height: 16px;
|
||||
}
|
||||
*/
|
||||
|
||||
blockquote {
|
||||
color: #aaa;
|
||||
padding-left: 10px;
|
||||
border-left: 1px dotted #666;
|
||||
}
|
||||
|
||||
|
||||
pre {
|
||||
background: rgba(0, 0, 0, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.15);
|
||||
padding: 10px;
|
||||
font-size: 14px;
|
||||
//color: #b5e853;
|
||||
border-radius: 2px;
|
||||
-moz-border-radius: 2px;
|
||||
-webkit-border-radius: 2px;
|
||||
text-wrap: normal;
|
||||
overflow: auto;
|
||||
overflow-y: hidden;
|
||||
}
|
||||
|
||||
pre.address {
|
||||
margin-bottom: 0 ;
|
||||
margin-top: 0 ;
|
||||
font: inherit }
|
||||
|
||||
pre.literal-block, pre.doctest-block, pre.math, pre.code {
|
||||
margin-left: 2em ;
|
||||
margin-right: 2em }
|
||||
|
||||
code .ln { color: grey; } /* line numbers */
|
||||
/*code, code { background-color: #eeeeee }*/
|
||||
code .comment, code .comment, code .c1 { color: #999; }
|
||||
code .keyword, code .keyword, code .kd, code .kn, code .k, code .o { color: #FC8F3F; font-weight: bold;}
|
||||
code .nb { color: #c45918;}
|
||||
code .s {color: #0a77c4;}
|
||||
code .punctuation, code .p { color: white;}
|
||||
code .literal.string, code .literal.string { color: #40BF32; }
|
||||
code .name, code .name.builtin, code .nx { color: white; }
|
||||
code .deleted, code .deleted { background-color: #DEB0A1}
|
||||
code .inserted, code .inserted { background-color: #A3D289}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
th {
|
||||
text-align: left;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
hr {
|
||||
height: 0;
|
||||
border: 0;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
color: #b5e853;
|
||||
}
|
||||
/* Links
|
||||
* a, a:hover, a:visited
|
||||
* */
|
||||
|
||||
a {
|
||||
color: #63c0f5;
|
||||
/*text-shadow: 0 0 5px rgba(104, 182, 255, 0.5);*/
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
cite {
|
||||
color: #00FF4A;
|
||||
}
|
||||
|
||||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG CONSOLE</h1><div class="contents" id="table-of-contents"><h2>Table of contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#creating-actions" id="id1">1 Creating actions</a></p></li><li><p><a class="reference internal" href="#searching" id="id2">2 Searching</a></p></li><li><p><a class="reference internal" href="#managing-investigators" id="id3">3 Managing investigators</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#creating-investigators" id="id4">3.1 Creating investigators</a></p></li></ul></li></ul></div><p><cite>mig-console</cite> is a terminal-based client to the MIG API. It provides a complete
|
||||
access to data stored in MIG and exposed by the API, in an interface that is
|
||||
easy to use. Similar to the command line client, <cite>mig-console</cite> reads its
|
||||
configuration from $HOME/.migrc and will prompt the user to create one if it
|
||||
doesn't exists.</p><p>The base screen of the API presents a dashboard of the current state of the
|
||||
infrastructure. It includes the number of agents and endpoints, broken down
|
||||
by version of the agent ,and a list of the latest actions that have run.</p><p>Underneath the dasboard, the user is invited to interact in a prompt that
|
||||
starts with <strong>mig></strong>.</p><pre><code class="code bash"><span class="name variable">$ </span>mig-console -c ~/.migrc-lnw
|
||||
|
||||
<span class="comment">## ## _.---._ .---.
|
||||
# # # /-\ ---|| | /\ __...---' .---. '---'-. '.
|
||||
# #| | / || | /--\ .-''__.--' _.'( | )'. '. '._ :
|
||||
# # \_/ ---| \_ \_/ \ .'__-'_ .--'' ._'---'_.-. '. '-'.
|
||||
</span> <span class="comment">### ~ -._ -._''---. -. '-._ '.
|
||||
</span> <span class="comment"># |\ |\ /---------| ~ -.._ _ _ _ ..-_ '. '-._''--.._
|
||||
</span> <span class="comment"># | \| \ / |- |__ | | -~ -._ '-. -. '-._''--.._.--''.
|
||||
</span> <span class="comment">###| \ \/ ---__| | | ~ ~-.__ -._ '-.__ '. '. :
|
||||
</span> <span class="comment">##### ~~ ~---...__ _ ._ .' '.
|
||||
</span> <span class="comment"># /\ --- /-\ |--|---- ~ ~--.....--~
|
||||
</span> <span class="comment"># ### /--\ | | ||-\ //
|
||||
</span> <span class="comment">#####/ \ | \_/ | \//__
|
||||
</span>+------
|
||||
<span class="punctuation">|</span> Agents <span class="punctuation">&</span> Endpoints summary:
|
||||
<span class="punctuation">|</span> * <span class="literal number">2</span> online agents on <span class="literal number">2</span> endpoints
|
||||
<span class="punctuation">|</span> * <span class="literal number">0</span> idle agents on <span class="literal number">0</span> endpoints
|
||||
<span class="punctuation">|</span> * <span class="literal number">0</span> endpoints are running <span class="literal number">2</span> or more agents
|
||||
<span class="punctuation">|</span> * <span class="literal number">0</span> endpoints appeared over the last <span class="literal number">7</span> days
|
||||
<span class="punctuation">|</span> * <span class="literal number">0</span> endpoints disappeared over the last <span class="literal number">7</span> days
|
||||
<span class="punctuation">|</span> * <span class="literal number">0</span> endpoints have been flapping
|
||||
<span class="punctuation">|</span> Online agents by version:
|
||||
<span class="punctuation">|</span> * version 20150909+f2456f5.dev: <span class="literal number">2</span> agents
|
||||
<span class="punctuation">|</span> Idle agents by version:
|
||||
<span class="punctuation">|</span>
|
||||
<span class="punctuation">|</span> Latest Actions:
|
||||
<span class="punctuation">|</span> ---- ID ---- + ---- Name ---- + -Sent- + ---- Date ---- + ---- Investigators ----
|
||||
<span class="punctuation">|</span> <span class="literal number">4999271350274</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-29T15:40:35Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4964811669519</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-23T13:37:16Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4964811669506</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-23T13:37:03Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4964764024853</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-23T13:25:26Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4964764024834</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-23T13:24:57Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4949328330767</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-20T19:59:39Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4949328330754</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-20T19:59:25Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4948324450316</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-20T19:45:51Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4948324450307</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-20T19:33:17Z Julien Vehent
|
||||
<span class="punctuation">|</span> <span class="literal number">4947944865794</span> file -c /home/ulfr/.migrc-l... <span class="literal number">2</span> 2015-09-20T14:07:36Z Julien Vehent
|
||||
+------
|
||||
|
||||
Connected to https://jaffa.linuxwall.info/api/v1/. Exit with ctrl+d. Type <span class="name builtin">help </span><span class="keyword">for</span> help.
|
||||
mig></code></pre><p>Entering <strong>help</strong> here, or on any other mode of the console, provides the user
|
||||
with a list of available functionalities:</p><pre>mig> help
|
||||
The following orders are available:
|
||||
action <id> enter interactive action reader mode for action <id>
|
||||
agent <id> enter interactive agent reader mode for agent <id>
|
||||
create action create a new action
|
||||
create investigator create a new investigator, will prompt for name and public key
|
||||
command <id> enter command reader mode for command <id>
|
||||
exit leave
|
||||
help show this help
|
||||
history <count> print last <count> entries in history. count=10 by default.
|
||||
investigator <id> enter interactive investigator management mode for investigator <id>
|
||||
query <uri> send a raw query string, without the base url, to the api
|
||||
search <search> perform a search. see "search help" for more information.
|
||||
showcfg display running configuration
|
||||
status display platform status: connected agents, latest actions, ...</pre><p>For example, let's review action id 4999271350274:</p><pre>mig> action 4999271350274
|
||||
Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Action: 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '.
|
||||
Launched by 'Julien Vehent' on '2015-09-29 11:40:31.792627 -0400 EDT'.
|
||||
Status 'completed'.
|
||||
2 sent, 2 done, 2 succeeded
|
||||
action 274></pre><p>When entering the action reader mode, a short summary of the action is displayed
|
||||
to the user. We can see that the action was launched by "Julien Vehent" on 2 agents
|
||||
and that it completed. The prompt is changed to <strong>action 274</strong> to indicate that
|
||||
we are now in action reader mode. Entering <strong>help</strong> here provides a list of
|
||||
functionalities specific to that mode:</p><pre>The following orders are available:
|
||||
command <id> jump to command reader mode for command <id>
|
||||
|
||||
copy enter action launcher mode using current action as template
|
||||
|
||||
counters display the counters of the action
|
||||
|
||||
details display the details of the action, including status & times
|
||||
|
||||
exit exit this mode (also works with ctrl+d)
|
||||
|
||||
help show this help
|
||||
|
||||
investigators print the list of investigators that signed the action
|
||||
|
||||
json show the json of the action
|
||||
|
||||
list <show> returns the list of commands with their status
|
||||
<show>: * set to "all" to get all results (default)
|
||||
* set to "found" to only display positive results
|
||||
* set to "notfound" for negative results
|
||||
list can be followed by a 'filter' pipe:
|
||||
ex: ls | grep server1.(dom1|dom2) | grep -v example.net
|
||||
|
||||
r refresh the action (get latest version from upstream)
|
||||
|
||||
results <show> <render> display results of all commands
|
||||
<show>: * set to "all" to get all results (default)
|
||||
* set to "found" to only display positive results
|
||||
* set to "notfound" for negative results
|
||||
<render>: * set to "text" to print results in console (default)
|
||||
* set to "map" to generate an open a google map
|
||||
|
||||
times show the various timestamps of the action</pre><p>It is possible to review results by entering <strong>results</strong>. If we only want results
|
||||
from agents that have found <em>something</em>, we can use <strong>results found</strong>. If we
|
||||
want those results inside a Google Map, enter <strong>results found map</strong>.</p><p>The raw json of the action is available via <strong>json</strong>.</p><p>To get a list of agents on which the action ran, use <strong>list all</strong>. You can filter
|
||||
the list using <strong>list found</strong> and <strong>list notfound</strong> to only get agents that have,
|
||||
or have not, found something. This command also supports a very basic <cite>grep</cite>.</p><p>To continue, let's list the agents that have found <em>something</em>:</p><pre>action 274> list found
|
||||
..
|
||||
---- Command ID ---- ---- Agent Name & ID----
|
||||
4999268991155 server1.example.net [4942082344151]
|
||||
4999268991154 server2.example.net [4942082360682]
|
||||
2 agents have found things</pre><p>We can inspect the command that ran on server1 by entering its ID:</p><pre>action 274> command 4999268991155
|
||||
Entering command reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Command 4999268991155 ran on agent 'server1.example.net' based on action 'file -c /home/ulfr/.migrc-lnw -path /etc -name ^passwd -content julien '
|
||||
command 155></pre><p>As you can see, the console mode has changed from <cite>action 274</cite> to
|
||||
<cite>command 155</cite>. This mode has its own set of functionalities that you
|
||||
can explore via <strong>help</strong>.</p><section id="creating-actions"><header><h2><a href="#id1">1 Creating actions</a></h2></header><p>The console provides a fine grained action generation interface. There
|
||||
are two ways to create a new action:</p><ol class="arabic"><li><p>From the <strong>mig></strong> mode, create a new empty action using <strong>create action</strong>.</p></li><li><p>From the <strong>action></strong> reader mode, when reviewing a previous action,
|
||||
copy it to a new action using <strong>copy</strong>.</p></li></ol><p>Both methods enter the action launcher mode. Method 2 only enters this
|
||||
mode with an preset action, rather than an empty one:</p><pre>mig> create action
|
||||
Entering action launcher with empty template
|
||||
Type exit or press ctrl+d to leave. help may help.
|
||||
launcher> help
|
||||
The following orders are available:
|
||||
addoperation <module> append a new operation of type <module> to the action operations
|
||||
listagents list agents targetted by an action
|
||||
deloperation <opnum> remove operation numbered <opnum> from operations array, count starts at zero
|
||||
details display the action details
|
||||
exit exit this mode
|
||||
help show this help
|
||||
json <pack> show the json of the action
|
||||
launch <nofollow> launch the action. to return before completion, add "nofollow"
|
||||
load <path> load an action from a file at <path>
|
||||
setname <name> set the name of the action
|
||||
settarget <target> set the target
|
||||
settimes <start> <stop> set the validity and expiration dates
|
||||
sign PGP sign the action
|
||||
times show the various timestamps of the action</pre><p>Action parameters can be edited prior to launching it:</p><ul><li><p><strong>setname</strong> sets the name field of the action to a new string:</p><pre>launcher> setname Test action that pings google.com</pre></li><li><p><strong>settarget</strong> sets the target of the action. The target is evaluated
|
||||
right away, and a list of targeted agents can be obtained via <strong>listagents</strong>:</p><pre>launcher> settarget environment->>'os'='linux' and mode='daemon'
|
||||
2 agents will be targetted. To get the list, use 'listagents'
|
||||
|
||||
launcher> listagents
|
||||
---- ID ---- + ---- Name -------
|
||||
4942082360682 server1.example.net
|
||||
4942082344151 server2.example.net</pre></li><li><p><strong>settimes</strong> defines the validity period of the action using
|
||||
<start> and <stop> parameters. <start> can be set to "now", and <stop>
|
||||
can be a duration relative to "now":</p><pre>launcher> settimes now +10m
|
||||
launcher> times
|
||||
Valid from '2015-10-06 13:09:36.189134664 +0000 UTC' until '2015-10-06 13:20:36.189134664 +0000 UTC'
|
||||
Started on '0001-01-01 00:00:00 +0000 UTC'
|
||||
Last updated '0001-01-01 00:00:00 +0000 UTC'
|
||||
Finished on '0001-01-01 00:00:00 +0000 UTC'</pre></li><li><p><strong>addoperation <module></strong> is used to add a new operation to the action.
|
||||
<module> must be a know MIG module, such as <cite>file</cite>, <cite>netstat</cite>, <cite>memory</cite>,
|
||||
<cite>scribe</cite>, <cite>ping</cite>, <cite>timedrift</cite>, etc... When adding an operation, the console
|
||||
enters a new module-specific mode that takes the operation parameters. For
|
||||
example, this is how to add a ping operation that sends 5 TCP requests to
|
||||
google.com:</p><pre>launcher> addoperation ping
|
||||
Ping module checks connectivity between an endpoint and a remote host. It supports
|
||||
icmp, tcp and udp ping. See doc at http://mig.mozilla.org/doc/module_ping.html
|
||||
|
||||
d <ip/fqdn> Destination Address can be ipv4, ipv6 or FQDN
|
||||
example: d www.mozilla.org
|
||||
d 63.245.217.105
|
||||
|
||||
dp <port> For TCP and UDP, specifies the port to test connectivity to
|
||||
example: dp 53
|
||||
|
||||
p <protocol> Protocol to use for the ping. This can be "icmp", "tcp" or "udp"
|
||||
example: p udp
|
||||
|
||||
c <count> Number of ping/connection attempts. Defaults to 3.
|
||||
example: c 5
|
||||
|
||||
t <timeout> Connection timeout in seconds. Defaults to 5.
|
||||
example: t 10
|
||||
ping> p tcp
|
||||
ping> d google.com
|
||||
ping> c 5
|
||||
ping> done
|
||||
Inserting ping operation with parameters:
|
||||
{
|
||||
"module": "ping",
|
||||
"parameters": {
|
||||
"destination": google.com",
|
||||
"protocol": "tcp",
|
||||
"count": 5
|
||||
}
|
||||
}</pre></li><li><p><strong>deloperation</strong> removes an operation from the action operations list.
|
||||
Use <strong>json</strong> to visualize the list, and remove an operation using its
|
||||
position in the list, starting with zero:</p><pre>launcher> json
|
||||
{
|
||||
"id": 0,
|
||||
"name": "Test action that pings google.com",
|
||||
"target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
|
||||
"description": {},
|
||||
"threat": {},
|
||||
"validfrom": "2015-10-06T13:09:36.189134664Z",
|
||||
"expireafter": "2015-10-06T13:20:36.189134664Z",
|
||||
"operations": [
|
||||
{
|
||||
"module": "ping",
|
||||
"parameters": {
|
||||
"destination": "google.com",
|
||||
"protocol": "tcp",
|
||||
"count": 5
|
||||
}
|
||||
}
|
||||
],
|
||||
"pgpsignatures": null,
|
||||
"starttime": "0001-01-01T00:00:00Z",
|
||||
"finishtime": "0001-01-01T00:00:00Z",
|
||||
"lastupdatetime": "0001-01-01T00:00:00Z",
|
||||
"counters": {}
|
||||
}
|
||||
|
||||
launcher> deloperation 0
|
||||
|
||||
launcher> json
|
||||
{
|
||||
"id": 0,
|
||||
"name": "Test action that pings google.com",
|
||||
"target": "environment-\u003e\u003e'os'='linux' and mode='daemon'",
|
||||
"description": {},
|
||||
"threat": {},
|
||||
"validfrom": "2015-10-06T13:09:36.189134664Z",
|
||||
"expireafter": "2015-10-06T13:20:36.189134664Z",
|
||||
"operations": [],
|
||||
"pgpsignatures": null,
|
||||
"starttime": "0001-01-01T00:00:00Z",
|
||||
"finishtime": "0001-01-01T00:00:00Z",
|
||||
"lastupdatetime": "0001-01-01T00:00:00Z",
|
||||
"counters": {}
|
||||
}</pre></li></ul><p>When ready to launch the action, type <strong>launch</strong>. The console will enter
|
||||
follower mode and print the progress of the action. When completed, the
|
||||
console will directly enter action reader mode, where you can type <strong>results</strong>
|
||||
to view the results:</p><pre>launcher> launch
|
||||
Action 'Test action that pings google.com' successfully launched with ID '5033038708749' on target 'environment->>'os'='linux' and mode='daemon''
|
||||
Following action ID 5033038708749.status=inflight...50%.status=completed
|
||||
- 100.0% done in 10.004244071s
|
||||
2 sent, 2 done, 2 succeeded
|
||||
|
||||
Entering action reader mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Action: 'Test action that pings google.com'.
|
||||
Launched by 'Julien Vehent' on '2015-10-06 09:12:31.608546 -0400 EDT'.
|
||||
Status 'completed'.
|
||||
2 sent, 2 done, 2 succeeded
|
||||
action 749> results
|
||||
server2.example.net ping of google.com failed. Target is no reachable.
|
||||
server2.example.net command success
|
||||
server1.example.net ping of google.com failed. Target is no reachable.
|
||||
server1.example.net command success
|
||||
2 agents have all results</pre></section><section id="searching"><header><h2><a href="#id2">2 Searching</a></h2></header><p>Return to the base mode using ctrl+d or <strong>exit</strong>:</p><pre>command 155> exit
|
||||
exit
|
||||
action 274> exit
|
||||
exit
|
||||
mig></pre><p>When in <strong>mig></strong> mode, you can use the <strong>search</strong> functionality to
|
||||
explore passed actions and commands, or find investigators. Details
|
||||
of how to run searches can be obtains via <strong>mig> search help</strong>.</p><p>Searches can be slow if your dataset is very large, so make sure to
|
||||
use time windows and limits to help speed up a search:</p><pre>mig> search action where investigatorname=%vehent% and agentname=server1% and after=2015-09-01T00:00:00Z and before=2015-10-01T00:00:00Z
|
||||
Searching action after 2015-09-01T00:00:00Z and before 2015-10-01T00:00:00Z, limited to 100 results
|
||||
----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Investigators ---- + - Sent - + - Status - + --- Last Updated ---
|
||||
4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-29T15:40:35Z
|
||||
4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:16Z
|
||||
4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-23T13:37:03Z
|
||||
4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:25:26Z
|
||||
4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 inflight 2015-09-23T13:24:57Z
|
||||
4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:39Z
|
||||
4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:59:25Z
|
||||
4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:45:51Z
|
||||
4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T19:33:17Z
|
||||
4947944865794 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T14:07:36Z
|
||||
4947909869570 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:58:41Z
|
||||
4947901022223 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:42Z
|
||||
4947901022210 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:56:26Z
|
||||
4947890798596 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 2 completed 2015-09-20T13:55:02Z
|
||||
4885615083769 timedrift -c /home/ulfr/.mi... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:02:56Z
|
||||
4885615083755 pkg -c /home/ulfr/.migrc-ln... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:33Z
|
||||
4885615083739 memory -c /home/ulfr/.migrc... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T17:01:04Z
|
||||
4885615083724 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... Julien Vehent 3 completed 2015-09-09T16:58:00Z</pre></section><section id="managing-investigators"><header><h2><a href="#id3">3 Managing investigators</a></h2></header><p>The console can be used to create new investigators, active and
|
||||
disable them, and well as review their activity.</p><p>To review the activity of an investigator, find its ID using the
|
||||
search command and enter investigator mode:</p><pre>mig> search investigator where name=%vehent%
|
||||
Searching investigator after 2011-03-31T13:16:55Z and before 2020-04-12T13:16:55Z, limited to 100 results
|
||||
- ID - + ---- Name ---- + --- Status ---
|
||||
2 Julien Vehent active
|
||||
|
||||
mig> investigator 2
|
||||
Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Investigator 2 named 'Julien Vehent'
|
||||
|
||||
inv 2> help
|
||||
The following orders are available:
|
||||
details print the details of the investigator
|
||||
exit exit this mode
|
||||
help show this help
|
||||
lastactions <limit> print the last actions ran by the investigator. limit=10 by default.
|
||||
pubkey show the armored public key of the investigator
|
||||
r refresh the investigator (get latest version from upstream)
|
||||
setstatus <status> changes the status of the investigator to <status> (can be 'active' or 'disabled')</pre><p>The command <strong>lastactions</strong> will print the latest activity of this investigator:</p><pre>inv 2> lastactions
|
||||
----- ID ----- + -------- Action Name ------- + ----------- Target ---------- + ---- Date ---- + -- Status --
|
||||
5033038708749 Test action that pings goog... environment->>'os'='linux' ... 2015-10-06T09:12:31-04:00 completed
|
||||
4999271350274 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-29T11:40:31-04:00 completed
|
||||
4964811669519 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:37:12-04:00 completed
|
||||
4964811669506 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:36:59-04:00 completed
|
||||
4964764024853 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:25:22-04:00 inflight
|
||||
4964764024834 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-23T09:24:52-04:00 inflight
|
||||
4949328330767 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:35-04:00 completed
|
||||
4949328330754 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:59:21-04:00 completed
|
||||
4948324450316 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:45:47-04:00 completed
|
||||
4948324450307 file -c /home/ulfr/.migrc-l... status='online' AND mode='d... 2015-09-20T15:33:13-04:00 completed</pre><p>To disable him, use <strong>setstatus disabled</strong>. Disabled investigators are no longer
|
||||
allowed to send investigations via the API:</p><pre>inv 2> setstatus disabled
|
||||
Investigator status set to disabled
|
||||
|
||||
inv 2> details
|
||||
Investigator ID 2
|
||||
name Julien Vehent
|
||||
status disabled
|
||||
key id E60892BB9BD89A69F759A1A0A3D652173B763E8F
|
||||
created 2015-09-09 09:53:28.989481 -0400 EDT
|
||||
modified 2015-09-09 09:53:28.989481 -0400 EDT
|
||||
|
||||
inv 2> setstatus active
|
||||
Investigator status set to active</pre><section id="creating-investigators"><header><h3><a href="#id4">3.1 Creating investigators</a></h3></header><p>To create a new investigator, go back to <strong>mig></strong> mode and type
|
||||
<strong>create investigator</strong>. The console will prompt the name of the new
|
||||
investigator as well as the location of her public PGP key. You can either
|
||||
provide a local path to the public key file on disk, on provide a fingerprint
|
||||
in the format "0x<40 char sha1 hash>". When a fingerprint is provided, the
|
||||
console will attempt to retrieve the key from the keyserver <cite>gpg.mozilla.org</cite>:</p><pre>mig> create investigator
|
||||
Entering investigator creation mode. Please provide the full name
|
||||
and the public key of the new investigator.
|
||||
name> Bob Kelso
|
||||
Name: 'Bob Kelso'
|
||||
Please provide a public key. You can either provide a local path to the
|
||||
armored public key file, or a full length PGP fingerprint.
|
||||
example:
|
||||
pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
pubkey> 0x716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
retrieving public key from http://gpg.mozilla.org
|
||||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||||
Version: SKS 1.1.5
|
||||
Comment: Hostname: keyserver.mozilla.org
|
||||
|
||||
mQENBEbv+5sBCADNHPvUIajRoxb/qylLrzwm9e+9sB8R/jhY4gxOzGZRDHECPvNeTUd9eogV
|
||||
n24rQDTWowkE+t9sW7vlD3TUWdBEAhXEpDZBfzlTBWIzEb1m3hwPOQM10ZNX6jPS1WlGsfoE
|
||||
LsUC0HmFTtOx4b5os9mIYbsjsDWd/JZjn0yUIv4eb28+fle6BkbgqIotLW4d1gTrxVlFc3be
|
||||
m+4OqimQ/v2LZDV+uObEkbh4UvmTtOCCx8zAOyZohPmICUbmJBc8KWWhzLOo8b9ns/GqP41q
|
||||
/9IuTQDXP2GUAKXzBKSdQiNzJP8Skfu4tPyGsGJSErprPC9t43HPPUgfeW9/sfuaw+vnABEB
|
||||
AAG0JUp1bGllbiBWRUhFTlQgPGp1bGllbkBsaW51eHdhbGwuaW5mbz6JATYEEwECACAFAkbv
|
||||
+5sCGy8GCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRBkUJDmQ2dze5U9CADK4Z6X02TP9afJ
|
||||
AyWF32zM3UdMksJ/F2wuo2HBHT0iOomw4ecNzHyO1P5BTglm5LC5ZZrV+Dx6Jve75JiSDTSD
|
||||
V3AhpR+M83rw8YKkeUrbTvfsy3+qhB7HYNIbCKT0lgAAL05SmDnYwYMQIV+p3T0F8BgGhkGT
|
||||
vHdKLhzEhKNOMaMCwCd1SsiBepA976oBUp9h5Vt6TEFyG6hCcFP90DFjNlK17yMNjbdrgyd6
|
||||
FkGEePKK3RhaLxcPShAgCnYzYYMABLYu1ow5AxxtEJTBHkJFkIzE9XM/lmyekYhWfX/Q4jpn
|
||||
1aiiVlf2klZlFSFy/TXuuQH1JO3YlKo9uHjlvSQb
|
||||
=+Ru1
|
||||
-----END PGP PUBLIC KEY BLOCK-----
|
||||
|
||||
create investigator? (y/n)> y
|
||||
Investigator 'Bob Kelso' successfully created with ID 4
|
||||
|
||||
mig> investigator 4
|
||||
Entering investigator mode. Type exit or press ctrl+d to leave. help may help.
|
||||
Investigator 4 named 'Bob Kelso'
|
||||
|
||||
inv 4> details
|
||||
Investigator ID 4
|
||||
name Bob Kelso
|
||||
status active
|
||||
key id 716CFA6BA8EBB21E860AE231645090E64367737B
|
||||
created 2015-10-06 09:23:14.473307 -0400 EDT
|
||||
modified 2015-10-06 09:23:14.473307 -0400 EDT</pre><p>The new investigator now has access to the API.</p></section></section></body></html>
|
||||
|
|
@ -1,7 +1,6 @@
|
|||
========
|
||||
MIG Data
|
||||
========
|
||||
:Author: Julien Vehent <jvehent@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
@ -227,7 +229,7 @@ cite {
|
|||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG Data</h1><table><tr><td class="field-label">Author</td><td>Julien Vehent <<a class="reference external" href="mailto:jvehent@mozilla.com">jvehent@mozilla.com</a>></td></tr></table><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#postgresql" id="id1">1 Postgresql</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#database-creation-script" id="id2">1.1 Database creation script</a></p></li><li><p><a class="reference internal" href="#structure-tables" id="id3">1.2 Structure & Tables</a></p></li></ul></li><li><p><a class="reference internal" href="#queries" id="id4">2 Queries</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#adding-investigators" id="id5">2.1 Adding Investigators</a></p></li><li><p><a class="reference internal" href="#finding-offline-agents" id="id6">2.2 Finding offline agents</a></p></li><li><p><a class="reference internal" href="#finding-double-agents" id="id7">2.3 Finding double agents</a></p></li></ul></li><li><p><a class="reference internal" href="#scheduler-spool" id="id8">3 Scheduler Spool</a></p></li></ul></div><section id="postgresql"><header><h2><a href="#id1">1 Postgresql</a></h2></header><section id="database-creation-script"><header><h3><a href="#id2">1.1 Database creation script</a></h3></header><p>Two scripts can be used to create a database for MIG.</p><ul><li><p><a class="reference external" href="https://github.com/mozilla/mig/blob/master/src/mig/database/createlocaldb.sh">createlocaldb.sh</a> will create a database on an instance of postgresql
|
||||
</style></head><body><h1>MIG Data</h1><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#postgresql" id="id1">1 Postgresql</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#database-creation-script" id="id2">1.1 Database creation script</a></p></li><li><p><a class="reference internal" href="#structure-tables" id="id3">1.2 Structure & Tables</a></p></li></ul></li><li><p><a class="reference internal" href="#queries" id="id4">2 Queries</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#adding-investigators" id="id5">2.1 Adding Investigators</a></p></li><li><p><a class="reference internal" href="#finding-offline-agents" id="id6">2.2 Finding offline agents</a></p></li><li><p><a class="reference internal" href="#finding-double-agents" id="id7">2.3 Finding double agents</a></p></li></ul></li><li><p><a class="reference internal" href="#scheduler-spool" id="id8">3 Scheduler Spool</a></p></li></ul></div><section id="postgresql"><header><h2><a href="#id1">1 Postgresql</a></h2></header><section id="database-creation-script"><header><h3><a href="#id2">1.1 Database creation script</a></h3></header><p>Two scripts can be used to create a database for MIG.</p><ul><li><p><a class="reference external" href="https://github.com/mozilla/mig/blob/master/src/mig/database/createlocaldb.sh">createlocaldb.sh</a> will create a database on an instance of postgresql
|
||||
running locally. This is used by the standalone installation script.</p></li></ul><ul><li><p><a class="reference external" href="https://github.com/mozilla/mig/blob/master/src/mig/database/createremotedb.sh">createremotedb.sh</a> will connect to an existing MIG database on a remote
|
||||
postgresql server. This is a standard production setup. It assumes that you
|
||||
have created a database beforehand. You can pass the DB credentials by
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
===========
|
||||
MIG Modules
|
||||
===========
|
||||
:Author: Julien Vehent <jvehent@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of Contents
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 900px;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
|
|
@ -29,7 +29,7 @@ section {
|
|||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
margin: 0;
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
|
|
@ -78,6 +78,8 @@ header h2 {
|
|||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
|
|
@ -227,7 +229,7 @@ cite {
|
|||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG Modules</h1><table><tr><td class="field-label">Author</td><td>Julien Vehent <<a class="reference external" href="mailto:jvehent@mozilla.com">jvehent@mozilla.com</a>></td></tr></table><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#module-logic" id="id3">1 Module logic</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#registration" id="id4">1.1 Registration</a></p></li><li><p><a class="reference internal" href="#execution" id="id5">1.2 Execution</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#runner-interface" id="id6">1.2.1 Runner Interface</a></p></li><li><p><a class="reference internal" href="#parameters" id="id7">1.2.2 Parameters</a></p></li><li><p><a class="reference internal" href="#run" id="id8">1.2.3 Run</a></p></li><li><p><a class="reference internal" href="#validate-parameters" id="id9">1.2.4 Validate Parameters</a></p></li></ul></li><li><p><a class="reference internal" href="#results" id="id10">1.3 Results</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#success" id="id11">1.3.1 Success</a></p></li><li><p><a class="reference internal" href="#foundanything" id="id12">1.3.2 FoundAnything</a></p></li><li><p><a class="reference internal" href="#elements" id="id13">1.3.3 Elements</a></p></li><li><p><a class="reference internal" href="#statistics" id="id14">1.3.4 Statistics</a></p></li><li><p><a class="reference internal" href="#errors" id="id15">1.3.5 Errors</a></p></li></ul></li><li><p><a class="reference internal" href="#additional-interfaces" id="id16">1.4 Additional interfaces</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#hasresultsprinter" id="id17">1.4.1 HasResultsPrinter</a></p></li><li><p><a class="reference internal" href="#hasparamscreator" id="id18">1.4.2 HasParamsCreator</a></p></li><li><p><a class="reference internal" href="#hasparamsparser" id="id19">1.4.3 HasParamsParser</a></p></li></ul></li></ul></li><li><p><a class="reference internal" href="#the-example-module" id="id20">2 The Example module</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#headers-and-structs" id="id21">2.1 Headers and structs</a></p></li><li><p><a class="reference internal" href="#id1" id="id22">2.2 Validate Parameters</a></p></li><li><p><a class="reference internal" href="#id2" id="id23">2.3 Run</a></p></li><li><p><a class="reference internal" href="#doing-work-and-building-results" id="id24">2.4 Doing work and building results</a></p></li><li><p><a class="reference internal" href="#printing-results" id="id25">2.5 Printing results</a></p></li><li><p><a class="reference internal" href="#creating-parameters" id="id26">2.6 Creating parameters</a></p></li></ul></li></ul></div><p>In this document, we explain how modules are written and integrated into MIG.</p><p>The reception of a command by an agent triggers the execution of modules. A
|
||||
</style></head><body><h1>MIG Modules</h1><div class="contents" id="table-of-contents"><h2>Table of Contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#module-logic" id="id3">1 Module logic</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#registration" id="id4">1.1 Registration</a></p></li><li><p><a class="reference internal" href="#execution" id="id5">1.2 Execution</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#runner-interface" id="id6">1.2.1 Runner Interface</a></p></li><li><p><a class="reference internal" href="#parameters" id="id7">1.2.2 Parameters</a></p></li><li><p><a class="reference internal" href="#run" id="id8">1.2.3 Run</a></p></li><li><p><a class="reference internal" href="#validate-parameters" id="id9">1.2.4 Validate Parameters</a></p></li></ul></li><li><p><a class="reference internal" href="#results" id="id10">1.3 Results</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#success" id="id11">1.3.1 Success</a></p></li><li><p><a class="reference internal" href="#foundanything" id="id12">1.3.2 FoundAnything</a></p></li><li><p><a class="reference internal" href="#elements" id="id13">1.3.3 Elements</a></p></li><li><p><a class="reference internal" href="#statistics" id="id14">1.3.4 Statistics</a></p></li><li><p><a class="reference internal" href="#errors" id="id15">1.3.5 Errors</a></p></li></ul></li><li><p><a class="reference internal" href="#additional-interfaces" id="id16">1.4 Additional interfaces</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#hasresultsprinter" id="id17">1.4.1 HasResultsPrinter</a></p></li><li><p><a class="reference internal" href="#hasparamscreator" id="id18">1.4.2 HasParamsCreator</a></p></li><li><p><a class="reference internal" href="#hasparamsparser" id="id19">1.4.3 HasParamsParser</a></p></li></ul></li></ul></li><li><p><a class="reference internal" href="#the-example-module" id="id20">2 The Example module</a></p><ul class="auto-toc"><li><p><a class="reference internal" href="#headers-and-structs" id="id21">2.1 Headers and structs</a></p></li><li><p><a class="reference internal" href="#id1" id="id22">2.2 Validate Parameters</a></p></li><li><p><a class="reference internal" href="#id2" id="id23">2.3 Run</a></p></li><li><p><a class="reference internal" href="#doing-work-and-building-results" id="id24">2.4 Doing work and building results</a></p></li><li><p><a class="reference internal" href="#printing-results" id="id25">2.5 Printing results</a></p></li><li><p><a class="reference internal" href="#creating-parameters" id="id26">2.6 Creating parameters</a></p></li></ul></li></ul></div><p>In this document, we explain how modules are written and integrated into MIG.</p><p>The reception of a command by an agent triggers the execution of modules. A
|
||||
module is a Go package that is imported into the agent at compilation, and that
|
||||
performs a very specific set of tasks. For example, the <code>file</code> module
|
||||
provides a way to scan a file system for files that contain regexes, match a
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
==========
|
||||
MIG RUNNER
|
||||
==========
|
||||
:Author: Aaron Meihm <ameihm@mozilla.com>
|
||||
|
||||
.. sectnum::
|
||||
.. contents:: Table of contents
|
||||
|
|
|
|||
|
|
@ -0,0 +1,293 @@
|
|||
<!DOCTYPE html><html><head><meta charset="utf-8"><title></title><style type="text/css">body {
|
||||
width: 95%;
|
||||
max-width: 70%;
|
||||
margin: 20px;
|
||||
padding: 0;
|
||||
background: #151515 url("../images/bkg.png") 0 0;
|
||||
color: #eaeaea;
|
||||
font: 16px;
|
||||
line-height: 1.5em;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
}
|
||||
|
||||
#table-of-contents ul {
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
/* General & 'Reset' Stuff */
|
||||
|
||||
.container {
|
||||
width: 95%;
|
||||
max-width: 1000px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
section {
|
||||
display: block;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
/*margin: 0 0 20px;*/
|
||||
/*margin: 0;*/
|
||||
}
|
||||
|
||||
/* Header, <header>
|
||||
* header - container
|
||||
* h1 - project name
|
||||
* h2 - project description
|
||||
* */
|
||||
|
||||
header {
|
||||
background: rgba(0, 0, 0, 0.1);
|
||||
width: 100%;
|
||||
/*border-bottom: 1px dashed #b5e853;*/
|
||||
/*padding: 20px 0;
|
||||
* margin: 0 0 40px 0;*/
|
||||
padding: 5px 0;
|
||||
margin: 0 0 10px 0;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 30px;
|
||||
line-height: 1.5;
|
||||
margin: 0 0 0 -40px;
|
||||
font-weight: bold;
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
/*color: #b5e853;*/
|
||||
color: #089d00;
|
||||
text-shadow: 0 1px 1px rgba(0, 0, 0, 0.1),
|
||||
0 0 5px rgba(181, 232, 83, 0.1),
|
||||
0 0 10px rgba(181, 232, 83, 0.1);
|
||||
letter-spacing: -1px;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
header h1:before {
|
||||
content: "./ ";
|
||||
font-size: 24px;
|
||||
}
|
||||
|
||||
header h2 {
|
||||
font-size: 18px;
|
||||
font-weight: 300;
|
||||
}
|
||||
|
||||
/* Main Content
|
||||
* */
|
||||
|
||||
body {
|
||||
width: 100%;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
section img {
|
||||
max-width: 100%
|
||||
}
|
||||
|
||||
h2 a {
|
||||
font-weight: bold;
|
||||
color: #8AB638;
|
||||
line-height: 1.4em;
|
||||
font-size: 1.4em;
|
||||
}
|
||||
h3 a, h4 a, h5 a, h6 a {
|
||||
font-weight: bold;
|
||||
color: #934500;
|
||||
line-height: 1.4em;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 30px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 28px;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
h4 {
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
h5 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
h6 {
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
color: #999;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
dt {
|
||||
font-style: italic;
|
||||
font-weight: bold;
|
||||
}
|
||||
/*
|
||||
ul li {
|
||||
list-style: none;
|
||||
}
|
||||
*/
|
||||
/*
|
||||
ul li:before {
|
||||
content: ">>";
|
||||
font-family: Monaco, "Bitstream Vera Sans Mono", "Lucida Console", Terminal, monospace;
|
||||
font-size: 13px;
|
||||
color: #b5e853;
|
||||
margin-left: -37px;
|
||||
margin-right: 21px;
|
||||
line-height: 16px;
|
||||
}
|
||||
*/
|
||||
|
||||
blockquote {
|
||||
color: #aaa;
|
||||
padding-left: 10px;
|
||||
border-left: 1px dotted #666;
|
||||
}
|
||||
|
||||
|
||||
pre {
|
||||
background: rgba(0, 0, 0, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.15);
|
||||
padding: 10px;
|
||||
font-size: 14px;
|
||||
//color: #b5e853;
|
||||
border-radius: 2px;
|
||||
-moz-border-radius: 2px;
|
||||
-webkit-border-radius: 2px;
|
||||
text-wrap: normal;
|
||||
overflow: auto;
|
||||
overflow-y: hidden;
|
||||
}
|
||||
|
||||
pre.address {
|
||||
margin-bottom: 0 ;
|
||||
margin-top: 0 ;
|
||||
font: inherit }
|
||||
|
||||
pre.literal-block, pre.doctest-block, pre.math, pre.code {
|
||||
margin-left: 2em ;
|
||||
margin-right: 2em }
|
||||
|
||||
code .ln { color: grey; } /* line numbers */
|
||||
/*code, code { background-color: #eeeeee }*/
|
||||
code .comment, code .comment, code .c1 { color: #999; }
|
||||
code .keyword, code .keyword, code .kd, code .kn, code .k, code .o { color: #FC8F3F; font-weight: bold;}
|
||||
code .nb { color: #c45918;}
|
||||
code .s {color: #0a77c4;}
|
||||
code .punctuation, code .p { color: white;}
|
||||
code .literal.string, code .literal.string { color: #40BF32; }
|
||||
code .name, code .name.builtin, code .nx { color: white; }
|
||||
code .deleted, code .deleted { background-color: #DEB0A1}
|
||||
code .inserted, code .inserted { background-color: #A3D289}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 0 0 20px 0;
|
||||
}
|
||||
|
||||
th {
|
||||
text-align: left;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
hr {
|
||||
height: 0;
|
||||
border: 0;
|
||||
border-bottom: 1px dashed #b5e853;
|
||||
color: #b5e853;
|
||||
}
|
||||
/* Links
|
||||
* a, a:hover, a:visited
|
||||
* */
|
||||
|
||||
a {
|
||||
color: #63c0f5;
|
||||
/*text-shadow: 0 0 5px rgba(104, 182, 255, 0.5);*/
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
cite {
|
||||
color: #00FF4A;
|
||||
}
|
||||
|
||||
strong {
|
||||
color: #C64216;
|
||||
}
|
||||
</style></head><body><h1>MIG RUNNER</h1><div class="contents" id="table-of-contents"><h2>Table of contents</h2><ul class="auto-toc"><li><p><a class="reference internal" href="#runner-configuration-file" id="id1">1 Runner configuration file</a></p></li><li><p><a class="reference internal" href="#job-configuration" id="id2">2 Job configuration</a></p></li><li><p><a class="reference internal" href="#output-plugins" id="id3">3 Output plugins</a></p></li></ul></div><p>MIG runner is a service that can be deployed to automatically schedule actions
|
||||
to run within the MIG environment and retrieve/process the results.</p><p>The runner interacts directly with the MIG API in the same manner as a client
|
||||
would. When an action is scheduled to run, the runner will deploy the action
|
||||
and schedule a time to gather results (shortly after the action has expired).
|
||||
Once the action has expired, the runner will retrieve results from the API
|
||||
and store these results in the runner directory. The runner can also send the
|
||||
results from MIG to an external program for automatic parsing or formatting,
|
||||
for example to create events for MozDef and send them.</p><section id="runner-configuration-file"><header><h2><a href="#id1">1 Runner configuration file</a></h2></header><p>An example configuration file for use by mig-runner is shown below.</p><pre><code class="code">; Sample MIG runner configuration file
|
||||
|
||||
[runner]
|
||||
directory = "/home/mig-runner/runner" ; The path to the root runner directory
|
||||
checkdirectory = 30 ; How often to check runners/ for job changes
|
||||
|
||||
[logging]
|
||||
mode = "stdout" ; stdout | file | syslog
|
||||
level = "debug"
|
||||
|
||||
[client]
|
||||
clientconfpath = "default" ; Path to client conf, default for $HOME/.migrc
|
||||
delayresults = "30s"; Duration after action expiry to fetch results</code></pre><p>If the GPG key used by mig-runner is protected by a passphrase, the
|
||||
<cite>passphrase</cite> option can be included under the <cite>client</cite> section. If this is
|
||||
specified this passphrase will be used to access the private key.</p><p>The <cite>delayresults</cite> value is optional. If not set, the runner will attempt
|
||||
to fetch action results when the action has expired. If this is set to a
|
||||
duration string value, the runner will wait the specified duration after
|
||||
action expiry before fetching results (for example to ensure all results
|
||||
are written to the database by the scheduler).</p><p>The <cite>checkdirectory</cite> option specifies the number of seconds that elapse
|
||||
between scans of the runner directory for job changes. The runner will
|
||||
automatically add or remove jobs as new jobs are added in the spool
|
||||
directory. When the runner identifies changes to a job, this will be
|
||||
indicated in the log file as the old job configuration will be removed
|
||||
and the new configuration installed.</p><p>The <cite>directory</cite> option specifies the root directory that stores all the
|
||||
mig-runner related control information. A typical runner directory may look
|
||||
something like this.</p><pre><code class="code">runner/
|
||||
|
|
||||
+ runners/
|
||||
| |
|
||||
| + job1/
|
||||
| + job2/
|
||||
|
|
||||
+ plugins/</code></pre></section><section id="job-configuration"><header><h2><a href="#id2">2 Job configuration</a></h2></header><p>Under each job directory, a file entity.cfg defines the parameters used to
|
||||
run this job.</p><pre><code class="code">[configuration]
|
||||
schedule = "<cronexpr>"
|
||||
plugin = "<plugin name>"</code></pre><p>The schedule option should be set to a cron style expression to note when
|
||||
the job should be run.</p><p>The plugin is optional. If set, the value will be interpreted as an
|
||||
executable in the plugins directory. The results of the job will be piped
|
||||
into stdin of this executable in JSON format (mig.RunnerResult). The
|
||||
plugin can then parse and forward the data as needed.</p><p>Optionally the <cite>expiry</cite> setting can be set to a go Duration string to use
|
||||
for action expiry (for example 10m for 10 minutes). If this is not set
|
||||
in the job configuration, a default of 5 minutes will be used.</p><p>Optionally the <cite>sendonly</cite> setting can be set to true, which will result in
|
||||
the runner only dispatching the action, but not attempting to retrieve
|
||||
any results. As an example, this can be used to deploy actions that would
|
||||
be processed by MIG workers, rather than retrieved and processed on the
|
||||
runner side.</p><p>The results are also written into a <cite>results/</cite> subdirectory under the
|
||||
runner directory, using the action ID as a file name. This happens
|
||||
regardless of any plugin configuration for the job.</p><p>In the job directory, the MIG action should should be launched should be
|
||||
called <cite>action.json</cite>. The time validity and expiration fields will be
|
||||
filled in by the runner process before dispatching the action to the
|
||||
API.</p></section><section id="output-plugins"><header><h2><a href="#id3">3 Output plugins</a></h2></header><p>The runner writes JSON output to stdin of any configured output plugin. This
|
||||
is intended to provide flexibility, allowing plugins to be developed in
|
||||
any language. If a plugin is being developed in go, the mig.RunnerResult
|
||||
type can be used to parse incoming data. In other languages the JSON can
|
||||
be parsed as desired.</p></section></body></html>
|
||||
Загрузка…
Ссылка в новой задаче