mozilla
/
mig
зеркало из https://github.com/mozilla/mig.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

[major] Implement support for event workers 

This patch implement a new rabbitmq exchange called migevent where
the scheduler publishes any event of interest. Workers can subscribe
to specific events using binding keys (standard rabbitmq topic
exchange). A simple mozdef-asset worker is implemented, and the base
for an agent verification worker is also present but not yet functional.
This commit is contained in:
Julien Vehent 2015-02-10 22:09:57 -05:00
Родитель 8b1ec01e07
Коммит dcae2a81d7
15 изменённых файлов: 832 добавлений и 105 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

8
Makefile
Просмотреть файл

@ -240,6 +240,14 @@ rpm-api: mig-api
fpm -C tmp -n mig-api --license GPL --vendor mozilla --description "Mozilla InvestiGator API" \
-m "Mozilla OpSec" --url http://mig.mozilla.org --architecture $(FPMARCH) -v $(BUILDREV) -s dir -t rpm .
worker-agent-verif:
$(MKDIR) -p $(BINDIR)
$(GO) build $(GOOPTS) -o $(BINDIR)/mig_agent_verif_worker $(GOLDFLAGS) mig/workers/agent_verif
worker-mozdef-asset:
$(MKDIR) -p $(BINDIR)
$(GO) build $(GOOPTS) -o $(BINDIR)/mig_mozdef_asset_worker $(GOLDFLAGS) mig/workers/mozdef_asset
doc:
make -C doc doc

11
doc/Makefile
Просмотреть файл

@ -8,9 +8,14 @@ doc:
for doc in $$(ls *.rst); do \
$(RST2HTML) --stylesheet=docstyle.css "$$doc" > "$$doc.html"; \
done
for modname in $(ls ../src/mig/modules/); do \
if [ -r "../src/mig/modules/${modname}/doc.rst" ]; then \
$(RST2HTML) --stylesheet=docstyle.css "../src/mig/modules/${modname}/doc.rst" > "module_${modname}.html"; \
for name in `ls ../src/mig/modules/`; do \
if [ -r "../src/mig/modules/$$name/doc.rst" ]; then \
$(RST2HTML) --stylesheet=docstyle.css "../src/mig/modules/$$name/doc.rst" > "module_$$name.html"; \
fi; \
done
for name in `ls ../src/mig/workers/`; do \
if [ -r "../src/mig/workers/$$name/doc.rst" ]; then \
$(RST2HTML) --stylesheet=docstyle.css "../src/mig/workers/$$name/doc.rst" > "worker_$$name.html"; \
fi; \
done
dot -Tsvg -o .files/action_command_flow.svg .files/action_command_flow.dot

37
doc/concepts.rst
Просмотреть файл

@ -48,6 +48,8 @@ Terminology:
* **API**: a REST api that exposes the MIG platform to clients
* **Client**: a program used by an investigator to interface with MIG (like the
MIG Console, or the action generator)
* **Worker**: a worker is a small extension to the scheduler and api that
performs very specific tasks based on events received via the relay.
An investigator uses a client (such as the MIG Console) to communicate with
the API. The API interfaces with the Database and the Scheduler.
@ -344,41 +346,6 @@ control to authorize access to specific functions of modules. For example, an
investigator could be authorized to call the `regex` function of filechecker
module, but only in `/etc`. This functionality is not implemented yet.
Extracting PGP fingerprints from public keys
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Linux, the `gpg` command can easily display the fingerprint of a key using
`gpg --fingerprint <key id>`. For example:
.. code:: bash
$ gpg --fingerprint jvehent@mozilla.com
pub 2048R/3B763E8F 2013-04-30
Key fingerprint = E608 92BB 9BD8 9A69 F759 A1A0 A3D6 5217 3B76 3E8F
uid Julien Vehent (personal) <julien@linuxwall.info>
uid Julien Vehent (ulfr) <jvehent@mozilla.com>
sub 2048R/8026F39F 2013-04-30
You should always verify the trustworthiness of a key before using it:
.. code:: bash
$ gpg --list-sigs jvehent@mozilla.com
pub 2048R/3B763E8F 2013-04-30
uid Julien Vehent (personal) <julien@linuxwall.info>
sig 3 3B763E8F 2013-06-23 Julien Vehent (personal) <julien@linuxwall.info>
sig 3 28A860CE 2013-10-04 Curtis Koenig <ckoenig@mozilla.com>
.....
We want to extract the fingerprint, and obtain a 40 characters hexadecimal
string that can used in permissions.
.. code:: bash
$gpg --fingerprint --with-colons jvehent@mozilla.com |grep '^fpr'|cut -f 10 -d ':'
E60892BB9BD89A69F759A1A0A3D652173B763E8F
Agent initialization process
----------------------------
The agent tries to be as autonomous as possible. One of the goal is to ship

28
doc/concepts.rst.html
Просмотреть файл

@ -19,12 +19,7 @@
<li><a href="#action-commands-workflow">1.3   Action/Commands workflow</a></li>
</ul>
</li>
<li>
<p><a href="#access-control-lists">2   Access Control Lists</a></p>
<ul class="auto-toc">
<li><a href="#extracting-pgp-fingerprints-from-public-keys">2.1   Extracting PGP fingerprints from public keys</a></li>
</ul>
</li>
<li><a href="#access-control-lists">2   Access Control Lists</a></li>
<li>
<p><a href="#agent-initialization-process">3   Agent initialization process</a></p>
<ul class="auto-toc">
@ -67,6 +62,7 @@
<li><strong>Database</strong>: a storage backend used by the scheduler and the api</li>
<li><strong>API</strong>: a REST api that exposes the MIG platform to clients</li>
<li><strong>Client</strong>: a program used by an investigator to interface with MIG (like the MIG Console, or the action generator)</li>
<li><strong>Worker</strong>: a worker is a small extension to the scheduler and api that performs very specific tasks based on events received via the relay.</li>
</ul>
<p>An investigator uses a client (such as the MIG Console) to communicate with the API. The API interfaces with the Database and the Scheduler. When an action is created by an investigator, the API receives it and writes it into the spool of the scheduler (they share it via NFS). The scheduler picks it up, creates one command per target agent, and sends those commands to the relays (running RabbitMQ). Each agent is listening on its own queue on the relay. The agents execute their commands, and return the results through the same relays (same exchange, different queues). The scheduler writes the results into the database, where the investigator can access them through the API. The agents also use the relays to send heartbeat at regular intervals, such that the scheduler always knows how many agents are alive at a given time.</p>
<p>The end-to-end workflow is:</p>
@ -251,26 +247,6 @@ investigator +--------&gt;| data |
<span class="p">}</span></code></pre>
<p>The <cite>default</cite> permission is overridden by module specific permissions.</p>
<p>The ACL is currently applied to modules. In the future, ACL will have finer control to authorize access to specific functions of modules. For example, an investigator could be authorized to call the <cite>regex</cite> function of filechecker module, but only in <cite>/etc</cite>. This functionality is not implemented yet.</p>
<section id="extracting-pgp-fingerprints-from-public-keys">
<h3>2.1   Extracting PGP fingerprints from public keys</h3>
<p>On Linux, the <cite>gpg</cite> command can easily display the fingerprint of a key using <cite>gpg --fingerprint &lt;key id&gt;</cite>. For example:</p>
<pre><code class="bash"><span class="nv">$ </span>gpg --fingerprint jvehent@mozilla.com
pub 2048R/3B763E8F 2013-04-30
Key <span class="nv">fingerprint</span> <span class="o">=</span> E608 92BB 9BD8 9A69 F759 A1A0 A3D6 5217 3B76 3E8F
uid Julien Vehent <span class="o">(</span>personal<span class="o">)</span> &lt;julien@linuxwall.info&gt;
uid Julien Vehent <span class="o">(</span>ulfr<span class="o">)</span> &lt;jvehent@mozilla.com&gt;
sub 2048R/8026F39F 2013-04-30</code></pre>
<p>You should always verify the trustworthiness of a key before using it:</p>
<pre><code class="bash"><span class="nv">$ </span>gpg --list-sigs jvehent@mozilla.com
pub 2048R/3B763E8F 2013-04-30
uid Julien Vehent <span class="o">(</span>personal<span class="o">)</span> &lt;julien@linuxwall.info&gt;
sig 3 3B763E8F 2013-06-23 Julien Vehent <span class="o">(</span>personal<span class="o">)</span> &lt;julien@linuxwall.info&gt;
sig 3 28A860CE 2013-10-04 Curtis Koenig &lt;ckoenig@mozilla.com&gt;
.....</code></pre>
<p>We want to extract the fingerprint, and obtain a 40 characters hexadecimal string that can used in permissions.</p>
<pre><code class="bash"><span class="nv">$gpg</span> --fingerprint --with-colons jvehent@mozilla.com |grep <span class="s1">'^fpr'</span>|cut -f 10 -d <span class="s1">':'</span>
E60892BB9BD89A69F759A1A0A3D652173B763E8F</code></pre>
</section>
</section>
<section id="agent-initialization-process">
<h2>3   Agent initialization process</h2>

86
doc/configuration.rst
Просмотреть файл

@ -570,12 +570,12 @@ Then, make sure than `pam_limits.so` is included in `/etc/pam.d/common-session`:
session required pam_limits.so
1. On the rabbitmq server, create three users:
1. On the rabbitmq server, create users:
* **admin**, with the tag 'administrator'
* **scheduler** and **agent**, with no tag
* **scheduler** , **agent**, **api** and **worker** with no tag
All three should have strong passwords. The scheduler password goes into the
All users should have strong passwords. The scheduler password goes into the
configuration file `conf/mig-scheduler.cfg`, in `[mq] password`. The agent
password goes into `conf/mig-agent-conf.go`, in the agent `AMQPBROKER` dial
string. The admin password is, of course, for yourself.
@ -589,6 +589,10 @@ string. The admin password is, of course, for yourself.
sudo rabbitmqctl add_user agent SomeRandomPassword
sudo rabbitmqctl add_user api SomeRandomPassword
sudo rabbitmqctl add_user worker SomeRandomPassword
You can list the users with the following command:
.. code:: bash
@ -609,37 +613,79 @@ On fresh installation, rabbitmq comes with a `guest` user that as password
sudo rabbitmqctl add_vhost mig
sudo rabbitmqctl list_vhosts
3. Create permissions for the scheduler user. The scheduler is allowed to
publish message (write) to the mig exchange. It can also configure and read
from the heartbeat and sched queues. The command below sets those permissions.
3. Create permissions for the scheduler user. The scheduler is allowed to:
- CONFIGURE:
- create the exchanges `mig` and `migevent`
- create and delete any queues under `migevent.*` and `mig.agt.*`
- WRITE:
- publish into the exchanges `mig` and `migevent`
- bind to the queues `mig.agt.heartbeats` and `mig.agt.results`
- bind to any queue under `migevent.*`
- READ:
- bind to the `mig` and `migevent` exchanges
- read from the queues `mig.agt.heartbeats`, `mig.agt.results`
- read from any queue under `migevent.*`
.. code:: bash
sudo rabbitmqctl set_permissions -p mig scheduler \
'^mig(|\.(heartbeat|sched\..*))' \
'^mig.*' \
'^mig(|\.(heartbeat|sched\..*))'
'^mig(|(event|\.agt)(|\..*))$' \
'^mig(|event(|\..*)|\.(agt\.(heartbeats|results)))$' \
'^mig(|event(|\..*)|\.(agt\.(heartbeats|results)))$'
4. Same thing for the agent. The agent is allowed to configure and read on the
'mig.agt.*' resource, and write to the 'mig' exchange.
4. Create permissions for the agent use. The agent is allowed to:
- CONFIGURE:
- create any queue under `mig.agt.(linux|windows|darwin).*`
- WRITE:
- publish to the `mig` exchange
- bind to any queue under `mig.agt.(linux|windows|darwin).*`
- READ:
- bind to the `mig` exchange
- read from any queue under `mig.agt.(linux|windows|darwin).*`
.. code:: bash
sudo rabbitmqctl set_permissions -p mig agent \
"^mig\.agt\.*" \
"^mig*" \
"^mig(|\.agt\..*)"
'^mig\.agt\.(linux|windows|darwin)\..*$' \
'^mig(|\.agt\.(linux|windows|darwin)\..*)$' \
'^mig(|\.agt\.(linux|windows|darwin)\..*)$'
5. Start the scheduler, it shouldn't return any ACCESS error. You can also list
5. Create permissions for the event workers and api users. The workers and api are allowed to:
- CONFIGURE:
- create any queue under `migevent.*`
- WRITE:
- write into the exchange `migevent`
- bind to any queue under `migevent.*`
- READ:
- bind to the `migevent` exchange
- read from any queue under `migevent.*`
.. code:: bash
sudo rabbitmqctl set_permissions -p mig worker \
'^migevent\..*$' \
'^migevent(|\..*)$' \
'^migevent(|\..*)$'
sudo rabbitmqctl set_permissions -p mig api \
'^migevent\..*$' \
'^migevent(|\..*)$' \
'^migevent(|\..*)$'
6. Start the scheduler, it shouldn't return any ACCESS error. You can also list
the permissions with the command:
.. code:: bash
sudo rabbitmqctl list_permissions -p mig
CONFIGURE WRITE READ
agent ^mig\\.agt\\.* ^mig* ^mig(|\\.agt\\..*)
scheduler ^mig(|\\.(heartbeat|sched\\..*)) ^mig.* ^mig(|\\.(heartbeat|sched\\..*))
$ sudo rabbitmqctl list_permissions -p mig | column -t
Listing permissions in vhost "mig" ...
USER------+CONFIGURE---------------------------------+WRITE-------------------------------------------------+READ-------------------------------------------------+
admin .* .* .*
scheduler ^mig(|(event|\\.agt)(|\\..*))$ ^mig(|event(|\\..*)|\\.(agt\\.(heartbeats|results)))$ ^mig(|event(|\\..*)|\\.(agt\\.(heartbeats|results)))$
agent ^mig\\.agt\\.(linux|windows|darwin)\\..*$ ^mig(|\\.agt\\.(linux|windows|darwin)\\..*)$ ^mig(|\\.agt\\.(linux|windows|darwin)\\..*)$
api ^migevent\\..*$ ^migevent(|\\..*)$ ^migevent(|\\..*)$
worker ^migevent\\..*$ ^migevent(|\\..*)$ ^migevent(|\\..*)$
RabbitMQ TLS configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~

175
doc/configuration.rst.html
Просмотреть файл

@ -382,22 +382,26 @@ BdUCSrvo/r7oAims8SyWE+ZObC+rw7u01Sut0ctnYrvklaM10+zkwGNOTszrduUy
<pre><code class="bash">session required pam_limits.so</code></pre>
<ol type="1">
<li>
<p>On the rabbitmq server, create three users:</p>
<p>On the rabbitmq server, create users:</p>
<blockquote>
<ul>
<li><strong>admin</strong>, with the tag 'administrator'</li>
<li><strong>scheduler</strong> and <strong>agent</strong>, with no tag</li>
<li><strong>scheduler</strong> , <strong>agent</strong>, <strong>api</strong> and <strong>worker</strong> with no tag</li>
</ul>
</blockquote>
</li>
</ol>
<p>All three should have strong passwords. The scheduler password goes into the configuration file <cite>conf/mig-scheduler.cfg</cite>, in <cite>[mq] password</cite>. The agent password goes into <cite>conf/mig-agent-conf.go</cite>, in the agent <cite>AMQPBROKER</cite> dial string. The admin password is, of course, for yourself.</p>
<p>All users should have strong passwords. The scheduler password goes into the configuration file <cite>conf/mig-scheduler.cfg</cite>, in <cite>[mq] password</cite>. The agent password goes into <cite>conf/mig-agent-conf.go</cite>, in the agent <cite>AMQPBROKER</cite> dial string. The admin password is, of course, for yourself.</p>
<pre><code class="bash">sudo rabbitmqctl add_user admin SomeRandomPassword
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl add_user scheduler SomeRandomPassword
sudo rabbitmqctl add_user agent SomeRandomPassword</code></pre>
sudo rabbitmqctl add_user agent SomeRandomPassword
sudo rabbitmqctl add_user api SomeRandomPassword
sudo rabbitmqctl add_user worker SomeRandomPassword</code></pre>
<p>You can list the users with the following command:</p>
<pre><code class="bash">sudo rabbitmqctl list_users</code></pre>
<p>On fresh installation, rabbitmq comes with a <cite>guest</cite> user that as password <cite>guest</cite> and admin privileges. You may you to delete that account.</p>
@ -408,26 +412,165 @@ sudo rabbitmqctl add_user agent SomeRandomPassword</code></pre>
<pre><code class="bash">sudo rabbitmqctl add_vhost mig
sudo rabbitmqctl list_vhosts</code></pre>
<ol start="3" type="1">
<li>Create permissions for the scheduler user. The scheduler is allowed to publish message (write) to the mig exchange. It can also configure and read from the heartbeat and sched queues. The command below sets those permissions.</li>
<li>
<dl>
<dt>Create permissions for the scheduler user. The scheduler is allowed to:</dt>
<dd>
<ul>
<li>
<dl>
<dt>CONFIGURE:</dt>
<dd>
<ul>
<li>create the exchanges <cite>mig</cite> and <cite>migevent</cite></li>
<li>create and delete any queues under <cite>migevent.*</cite> and <cite>mig.agt.*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>WRITE:</dt>
<dd>
<ul>
<li>publish into the exchanges <cite>mig</cite> and <cite>migevent</cite></li>
<li>bind to the queues <cite>mig.agt.heartbeats</cite> and <cite>mig.agt.results</cite></li>
<li>bind to any queue under <cite>migevent.*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>READ:</dt>
<dd>
<ul>
<li>bind to the <cite>mig</cite> and <cite>migevent</cite> exchanges</li>
<li>read from the queues <cite>mig.agt.heartbeats</cite>, <cite>mig.agt.results</cite></li>
<li>read from any queue under <cite>migevent.*</cite></li>
</ul>
</dd>
</dl>
</li>
</ul>
</dd>
</dl>
</li>
</ol>
<pre><code class="bash">sudo rabbitmqctl set_permissions -p mig scheduler <span class="se">\
</span><span class="s1">'^mig(|\.(heartbeat|sched\..*))'</span> <span class="se">\
</span><span class="s1">'^mig.*'</span> <span class="se">\
</span><span class="s1">'^mig(|\.(heartbeat|sched\..*))'</span></code></pre>
</span> <span class="s1">'^mig(|(event|\.agt)(|\..*))$'</span> <span class="se">\
</span> <span class="s1">'^mig(|event(|\..*)|\.(agt\.(heartbeats|results)))$'</span> <span class="se">\
</span> <span class="s1">'^mig(|event(|\..*)|\.(agt\.(heartbeats|results)))$'</span></code></pre>
<ol start="4" type="1">
<li>Same thing for the agent. The agent is allowed to configure and read on the 'mig.agt.*' resource, and write to the 'mig' exchange.</li>
<li>
<dl>
<dt>Create permissions for the agent use. The agent is allowed to:</dt>
<dd>
<ul>
<li>
<dl>
<dt>CONFIGURE:</dt>
<dd>
<ul>
<li>create any queue under <cite>mig.agt.(linux|windows|darwin).*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>WRITE:</dt>
<dd>
<ul>
<li>publish to the <cite>mig</cite> exchange</li>
<li>bind to any queue under <cite>mig.agt.(linux|windows|darwin).*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>READ:</dt>
<dd>
<ul>
<li>bind to the <cite>mig</cite> exchange</li>
<li>read from any queue under <cite>mig.agt.(linux|windows|darwin).*</cite></li>
</ul>
</dd>
</dl>
</li>
</ul>
</dd>
</dl>
</li>
</ol>
<pre><code class="bash">sudo rabbitmqctl set_permissions -p mig agent <span class="se">\
</span><span class="s2">"^mig\.agt\.*"</span> <span class="se">\
</span><span class="s2">"^mig*"</span> <span class="se">\
</span><span class="s2">"^mig(|\.agt\..*)"</span></code></pre>
</span> <span class="s1">'^mig\.agt\.(linux|windows|darwin)\..*$'</span> <span class="se">\
</span> <span class="s1">'^mig(|\.agt\.(linux|windows|darwin)\..*)$'</span> <span class="se">\
</span> <span class="s1">'^mig(|\.agt\.(linux|windows|darwin)\..*)$'</span></code></pre>
<ol start="5" type="1">
<li>
<dl>
<dt>Create permissions for the event workers and api users. The workers and api are allowed to:</dt>
<dd>
<ul>
<li>
<dl>
<dt>CONFIGURE:</dt>
<dd>
<ul>
<li>create any queue under <cite>migevent.*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>WRITE:</dt>
<dd>
<ul>
<li>write into the exchange <cite>migevent</cite></li>
<li>bind to any queue under <cite>migevent.*</cite></li>
</ul>
</dd>
</dl>
</li>
<li>
<dl>
<dt>READ:</dt>
<dd>
<ul>
<li>bind to the <cite>migevent</cite> exchange</li>
<li>read from any queue under <cite>migevent.*</cite></li>
</ul>
</dd>
</dl>
</li>
</ul>
</dd>
</dl>
</li>
</ol>
<pre><code class="bash">sudo rabbitmqctl set_permissions -p mig worker <span class="se">\
</span><span class="s1">'^migevent\..*$'</span> <span class="se">\
</span><span class="s1">'^migevent(|\..*)$'</span> <span class="se">\
</span><span class="s1">'^migevent(|\..*)$'</span>
sudo rabbitmqctl set_permissions -p mig api <span class="se">\
</span><span class="s1">'^migevent\..*$'</span> <span class="se">\
</span><span class="s1">'^migevent(|\..*)$'</span> <span class="se">\
</span><span class="s1">'^migevent(|\..*)$'</span></code></pre>
<ol start="6" type="1">
<li>Start the scheduler, it shouldn't return any ACCESS error. You can also list the permissions with the command:</li>
</ol>
<pre><code class="bash">sudo rabbitmqctl list_permissions -p mig
CONFIGURE WRITE READ
agent ^mig<span class="se">\\</span>.agt<span class="se">\\</span>.* ^mig* ^mig<span class="o">(</span>|<span class="se">\\</span>.agt<span class="se">\\</span>..*<span class="o">)</span>
scheduler ^mig<span class="o">(</span>|<span class="se">\\</span>.<span class="o">(</span>heartbeat|sched<span class="se">\\</span>..*<span class="o">))</span> ^mig.* ^mig<span class="o">(</span>|<span class="se">\\</span>.<span class="o">(</span>heartbeat|sched<span class="se">\\</span>..*<span class="o">))</span></code></pre>
<pre><code class="bash"><span class="nv">$ </span>sudo rabbitmqctl list_permissions -p mig | column -t
Listing permissions in vhost <span class="s2">"mig"</span> ...
USER------+CONFIGURE---------------------------------+WRITE-------------------------------------------------+READ-------------------------------------------------+
admin .* .* .*
scheduler ^mig<span class="o">(</span>|<span class="o">(</span>event|<span class="se">\\</span>.agt<span class="o">)(</span>|<span class="se">\\</span>..*<span class="o">))</span><span class="nv">$ </span> ^mig<span class="o">(</span>|event<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span>|<span class="se">\\</span>.<span class="o">(</span>agt<span class="se">\\</span>.<span class="o">(</span>heartbeats|results<span class="o">)))</span><span class="nv">$ </span> ^mig<span class="o">(</span>|event<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span>|<span class="se">\\</span>.<span class="o">(</span>agt<span class="se">\\</span>.<span class="o">(</span>heartbeats|results<span class="o">)))</span><span class="err">$</span>
agent ^mig<span class="se">\\</span>.agt<span class="se">\\</span>.<span class="o">(</span>linux|windows|darwin<span class="o">)</span><span class="se">\\</span>..*<span class="nv">$ </span> ^mig<span class="o">(</span>|<span class="se">\\</span>.agt<span class="se">\\</span>.<span class="o">(</span>linux|windows|darwin<span class="o">)</span><span class="se">\\</span>..*<span class="o">)</span><span class="nv">$ </span> ^mig<span class="o">(</span>|<span class="se">\\</span>.agt<span class="se">\\</span>.<span class="o">(</span>linux|windows|darwin<span class="o">)</span><span class="se">\\</span>..*<span class="o">)</span><span class="err">$</span>
api ^migevent<span class="se">\\</span>..*<span class="nv">$ </span> ^migevent<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span><span class="nv">$ </span> ^migevent<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span><span class="err">$</span>
worker ^migevent<span class="se">\\</span>..*<span class="nv">$ </span> ^migevent<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span><span class="nv">$ </span> ^migevent<span class="o">(</span>|<span class="se">\\</span>..*<span class="o">)</span><span class="err">$</span></code></pre>
</section>
<section id="rabbitmq-tls-configuration">
<h3>5.3   RabbitMQ TLS configuration</h3>

85
doc/worker_mozdef_asset.html Normal file
Просмотреть файл

@ -0,0 +1,85 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link href="docstyle.css" rel="stylesheet" />
<title>Mozilla InvestiGator: MozDef Asset Worker</title>
<meta content="Julien Vehent &lt;jvehent@mozilla.com&gt;" name="author" />
</head>
<body>
<h1>Mozilla InvestiGator: MozDef Asset Worker</h1>
<aside class="topic contents" id="table-of-contents">
<h1>Table of Contents</h1>
<ul class="auto-toc">
<li>
<p><a href="#configuration">1   Configuration</a></p>
<ul class="auto-toc">
<li><a href="#upstart">1.1   Upstart</a></li>
</ul>
</li>
</ul>
</aside>
<p>The MozDef Asset Worker in a separate program that listens for event about agents that newly joined the platform, and create asset hints that are published to MozDef. This worker serves a very specific purpose in the collection of asset data performed by Mozilla OpSec. It may not be very useful to anyone else.</p>
<section id="configuration">
<h2>1   Configuration</h2>
<p>This worker retrieves agents hearbeats in the MIG Agent format from the MIG Relay, transforms them into Asset Hints, and publishes them to some other rabbitmq endpoint when MozDef will retrieve them.</p>
<pre><code class="">; mozdef rabbitmq endpoint
[mozdef]
host = "hostname.mozdef.rabbitmq.example.net"
port = 5671
user = "mig"
pass = "somepassphrase"
vhost = "mozdef"
usetls = false
cacert = "/path/to/ca.crt"
tlscert = "/path/to/client.crt"
tlskey = "/path/to/client.key"
timeout = "10s"
; mig rabbitmq endpoint
[mq]
host = "hostname.mig.relay.example.net"
port = 5671
user = "migworker"
pass = "somepassphrase"
vhost = "mig"
usetls = true
cacert = "/path/to/ca.crt"
tlscert = "/path/to/client.crt"
tlskey = "/path/to/client.key"
timeout = "10s"
[logging]
mode = "syslog" ; stdout | file | syslog
level = "info" ; debug | info | warning | error | critical
host = "localhost"
port = 514
protocol = "udp"</code></pre>
<section id="upstart">
<h3>1.1   Upstart</h3>
<pre><code class=""># Mozilla InvestiGator MozDef Asset Worker
description "MIG MozDef Asset Worker"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
setuid mig
limit nofile 640000 640000
respawn
respawn limit 10 5
umask 022
console none
pre-start script
test /opt/mig_mozdef_asset_worker || { stop; exit 0; }
end script
# Start
exec /opt/mig_mozdef_asset_worker</code></pre>
</section>
</section>
</body>
</html>

11
src/mig/event/event.go Normal file
Просмотреть файл

@ -0,0 +1,11 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]
package event
const (
Q_Agt_Auth_Fail = "agent.authentication.failure"
Q_Agt_New = "agent.new"
)

12
src/mig/scheduler/agents_queues.go
Просмотреть файл

@ -9,6 +9,7 @@ import (
"encoding/json"
"fmt"
"mig"
"mig/event"
"time"
"github.com/streadway/amqp"
@ -100,6 +101,12 @@ func getHeartbeats(msg amqp.Delivery, ctx Context) (err error) {
if !ok {
desc := fmt.Sprintf("getHeartbeats(): Agent '%s' is not authorized", agt.QueueLoc)
ctx.Channels.Log <- mig.Log{Desc: desc}.Warning()
// send an event to notify workers of the failed agent auth
err = sendEvent(event.Q_Agt_Auth_Fail, msg.Body, ctx)
if err != nil {
panic(err)
}
// agent authorization failed so we drop this heartbeat and return
return
}
@ -115,6 +122,11 @@ func getHeartbeats(msg amqp.Delivery, ctx Context) (err error) {
if err != nil {
ctx.Channels.Log <- mig.Log{Desc: fmt.Sprintf("Heartbeat DB insertion failed with error '%v' for agent '%s'", err, agt.Name)}.Err()
}
// notify the agt.new event queue
err = sendEvent(event.Q_Agt_New, msg.Body, ctx)
if err != nil {
ctx.Channels.Log <- mig.Log{Desc: fmt.Sprintf("Failed to send migevent to %s: %v", err, event.Q_Agt_New)}.Err()
}
} else {
// the agent exists in database. reuse the existing ID, and keep the status if it was
// previously set to destroyed or upgraded. otherwise set status to online

13
src/mig/scheduler/context.go
Просмотреть файл

@ -67,9 +67,8 @@ type Context struct {
UseTLS bool
TLScert, TLSkey, CAcert string
Timeout string
// internal
conn *amqp.Connection
Chan *amqp.Channel
conn *amqp.Connection
Chan *amqp.Channel
}
PGP struct {
Pubring, Secring io.ReadSeeker
@ -288,12 +287,16 @@ func initRelay(orig_ctx Context) (ctx Context, err error) {
if err != nil {
panic(err)
}
// declare the "mig" exchange used for all publications
// declare the "mig" exchange used for communication with the agents
err = ctx.MQ.Chan.ExchangeDeclare("mig", "topic", true, false, false, false, nil)
if err != nil {
panic(err)
}
// declare the "migevent" exchange used for communication between the platform components
err = ctx.MQ.Chan.ExchangeDeclare("migevent", "topic", true, false, false, false, nil)
if err != nil {
panic(err)
}
ctx.Channels.Log <- mig.Log{Desc: "AMQP connection opened"}
return
}

31
src/mig/scheduler/events.go Normal file
Просмотреть файл

@ -0,0 +1,31 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]
package main
import (
"fmt"
"github.com/streadway/amqp"
"mig"
"time"
)
// sendEvent publishes a message to the miginternal rabbitmq exchange
func sendEvent(key string, body []byte, ctx Context) error {
msg := amqp.Publishing{
DeliveryMode: amqp.Persistent,
Timestamp: time.Now(),
ContentType: "text/plain",
Expiration: "6000000", // events expire after 100 minutes if not consumed
Body: body,
}
err := ctx.MQ.Chan.Publish("migevent", key, false, false, msg)
if err != nil {
err = fmt.Errorf("event publication failed. err='%v', key='%s', body='%s'", err, key, msg)
ctx.Channels.Log <- mig.Log{Desc: fmt.Sprintf("%v", err)}.Err()
return err
}
return nil
}

50
src/mig/workers/agent_verif/main.go Normal file
Просмотреть файл

@ -0,0 +1,50 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]
package main
import (
"code.google.com/p/gcfg"
"flag"
"fmt"
"mig/event"
"mig/workers"
"os"
)
const workerName = "agent_verif"
type Config struct {
Mq workers.MqConf
}
func main() {
var (
err error
conf Config
)
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "%s - a worker verifying agents that fail to authenticate\n", os.Args[0])
flag.PrintDefaults()
}
var configPath = flag.String("c", "/etc/mig/agent_verif_worker.cfg", "Load configuration from file")
flag.Parse()
err = gcfg.ReadFileInto(&conf, *configPath)
if err != nil {
panic(err)
}
// set a binding to route events from event.Q_Agt_Auth_Fail into the queue named after the worker
// and return a channel that consumes the queue
workerQueue := "migevent.worker." + workerName
consumerChan, err := workers.InitMqWithConsumer(conf.Mq, workerQueue, event.Q_Agt_Auth_Fail)
if err != nil {
panic(err)
}
fmt.Println("started worker", workerName, "consuming queue", workerQueue, "from key", event.Q_Agt_Auth_Fail)
for event := range consumerChan {
fmt.Printf("%s\n", event.Body)
}
return
}

84
src/mig/workers/mozdef_asset/doc.rst Normal file
Просмотреть файл

@ -0,0 +1,84 @@
=========================================
Mozilla InvestiGator: MozDef Asset Worker
=========================================
:Author: Julien Vehent <jvehent@mozilla.com>
.. sectnum::
.. contents:: Table of Contents
The MozDef Asset Worker in a separate program that listens for event about
agents that newly joined the platform, and create asset hints that are
published to MozDef. This worker serves a very specific purpose in the
collection of asset data performed by Mozilla OpSec. It may not be very useful
to anyone else.
Configuration
-------------
This worker retrieves agents hearbeats in the MIG Agent format from the MIG
Relay, transforms them into Asset Hints, and publishes them to some other
rabbitmq endpoint when MozDef will retrieve them.
.. code::
; mozdef rabbitmq endpoint
[mozdef]
host = "hostname.mozdef.rabbitmq.example.net"
port = 5671
user = "mig"
pass = "somepassphrase"
vhost = "mozdef"
usetls = false
cacert = "/path/to/ca.crt"
tlscert = "/path/to/client.crt"
tlskey = "/path/to/client.key"
timeout = "10s"
; mig rabbitmq endpoint
[mq]
host = "hostname.mig.relay.example.net"
port = 5671
user = "migworker"
pass = "somepassphrase"
vhost = "mig"
usetls = true
cacert = "/path/to/ca.crt"
tlscert = "/path/to/client.crt"
tlskey = "/path/to/client.key"
timeout = "10s"
[logging]
mode = "syslog" ; stdout | file | syslog
level = "info" ; debug | info | warning | error | critical
host = "localhost"
port = 514
protocol = "udp"
Upstart
~~~~~~~
.. code::
# Mozilla InvestiGator MozDef Asset Worker
description "MIG MozDef Asset Worker"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
setuid mig
limit nofile 640000 640000
respawn
respawn limit 10 5
umask 022
console none
pre-start script
test /opt/mig_mozdef_asset_worker || { stop; exit 0; }
end script
# Start
exec /opt/mig_mozdef_asset_worker

183
src/mig/workers/mozdef_asset/main.go Normal file
Просмотреть файл

@ -0,0 +1,183 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]
package main
import (
"code.google.com/p/gcfg"
"encoding/json"
"flag"
"fmt"
"github.com/streadway/amqp"
"mig"
"mig/event"
"mig/workers"
"os"
"regexp"
"time"
)
const workerName = "mozdef_asset"
type Config struct {
Mq workers.MqConf
MozDef workers.MqConf
Logging mig.Logging
}
func main() {
var (
err error
conf Config
hint hint
)
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "%s - a worker that listens to new endpoints and sends them as assets to mozdef\n", os.Args[0])
flag.PrintDefaults()
}
var configPath = flag.String("c", "/etc/mig/mozdef_asset_worker.cfg", "Load configuration from file")
flag.Parse()
err = gcfg.ReadFileInto(&conf, *configPath)
if err != nil {
panic(err)
}
logctx, err := mig.InitLogger(conf.Logging, workerName)
if err != nil {
panic(err)
}
// bind to the MIG even queue
workerQueue := "migevent.worker." + workerName
consumerChan, err := workers.InitMqWithConsumer(conf.Mq, workerQueue, event.Q_Agt_New)
if err != nil {
panic(err)
}
// bind to the mozdef relay exchange
mozdefChan, err := workers.InitMQ(conf.MozDef)
if err != nil {
panic(err)
}
mig.ProcessLog(logctx, mig.Log{Desc: "worker started, consuming queue " + workerQueue + " from key " + event.Q_Agt_New})
for event := range consumerChan {
var agt mig.Agent
err = json.Unmarshal(event.Body, &agt)
if err != nil {
mig.ProcessLog(logctx, mig.Log{Desc: fmt.Sprintf("invalid agent description: %v", err)}.Err())
}
hint, err = makeHintFromAgent(agt)
if err != nil {
mig.ProcessLog(logctx, mig.Log{Desc: fmt.Sprintf("failed to build asset hint: %v", err)}.Err())
}
err = publishHintToMozdef(hint, mozdefChan)
if err != nil {
mig.ProcessLog(logctx, mig.Log{Desc: fmt.Sprintf("failed to publish to mozdef: %v", err)}.Err())
}
mig.ProcessLog(logctx, mig.Log{Desc: "published asset hint for agent '" + hint.Details.Name + "' to mozdef"}.Debug())
}
return
}
// A hint describes informations about an endpoint as gathered by MIG
// The format uses Mozdef's standard event format described in
// http://mozdef.readthedocs.org/en/latest/usage.html#json-format
// {
// "timestamp": "2014-02-14T11:48:19.035762739-05:00",
// "summary": "mig discovered host server1.net.example.com",
// "hostname": "mig-worker1.use1.opsec.mozilla.com",
// "severity": "INFO",
// "category": "asset_hint",
// "tags": [
// "MIG",
// "Asset"
// ],
// "details": {
// "type": "host",
// "name": "opsec1.private.phx1.mozilla.com",
// "ipv4": [
// "10.8.75.110/24"
// ],
// "ipv6": [
// "fe80::250:56ff:febd:6850/64"
// ],
// "arch": "amd64",
// "ident": "Red Hat Enterprise Linux Server release 6.6 (Santiago)",
// "init": "upstart",
// "isproxied": false,
// "operator": "IT"
// }
// }
type hint struct {
Timestamp time.Time `json:"timestamp"`
Summary string `json:"summary"`
Hostname string `json:"hostname"`
Severity string `json:"severity"`
Category string `json:"category"`
Tags []string `json:"tags"`
Details hintDetails `json:"details"`
}
type hintDetails struct {
Type string `json:"type"`
Name string `json:"name"`
IPv4 []string `json:"ipv4"`
IPv6 []string `json:"ipv6"`
OS string `json:"os"`
Arch string `json:"arch"`
Ident string `json:"ident"`
Init string `json:"init"`
IsProxied bool `json:"isproxied"`
Operator string `json:"operator"`
}
func makeHintFromAgent(agt mig.Agent) (hint hint, err error) {
hint.Timestamp = time.Now().UTC()
hint.Summary = "mig discovered host " + agt.Name
hint.Hostname, err = os.Hostname()
if err != nil {
return
}
hint.Severity = "INFO"
hint.Category = "asset_hint"
hint.Tags = append(hint.Tags, "mig")
hint.Tags = append(hint.Tags, "asset")
hint.Details.Type = "host"
hint.Details.Name = agt.Name
reipv4 := regexp.MustCompile(`([0-9]{1,3}\.){3}([0-9]{1,3})`)
for _, ip := range agt.Env.Addresses {
if reipv4.MatchString(ip) {
hint.Details.IPv4 = append(hint.Details.IPv4, ip)
} else {
hint.Details.IPv6 = append(hint.Details.IPv6, ip)
}
}
hint.Details.OS = agt.Env.OS
hint.Details.Arch = agt.Env.Arch
hint.Details.Ident = agt.Env.Ident
hint.Details.Init = agt.Env.Init
hint.Details.IsProxied = agt.Env.IsProxied
if _, ok := agt.Tags.(map[string]interface{})["operator"]; ok {
hint.Details.Operator = agt.Tags.(map[string]interface{})["operator"].(string)
}
return
}
func publishHintToMozdef(hint hint, mozdefChan *amqp.Channel) (err error) {
data, err := json.Marshal(hint)
if err != nil {
return
}
msg := amqp.Publishing{
DeliveryMode: amqp.Persistent,
Timestamp: time.Now(),
ContentType: "text/plain",
Expiration: "6000000", // events expire after 100 minutes if not consumed
Body: data,
}
err = mozdefChan.Publish("mozdef", "event", false, false, msg)
return
}

123
src/mig/workers/workers.go Normal file
Просмотреть файл

@ -0,0 +1,123 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]
package workers
import (
"crypto/rand"
"crypto/tls"
"crypto/x509"
"fmt"
"github.com/streadway/amqp"
"io/ioutil"
"net"
"time"
)
type MqConf struct {
Host, User, Pass, Vhost string
Port int
UseTLS bool
TLScert, TLSkey, CAcert string
Timeout string
}
func InitMQ(conf MqConf) (amqpChan *amqp.Channel, err error) {
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("worker.initMQ() -> %v", e)
}
}()
// create an AMQP configuration with a 10min heartbeat and timeout
// dialing address use format "<scheme>://<user>:<pass>@<host>:<port><vhost>"
var scheme, user, pass, host, port, vhost string
if conf.UseTLS {
scheme = "amqps"
} else {
scheme = "amqp"
}
if conf.User == "" {
panic("MQ User is missing")
}
user = conf.User
if conf.Pass == "" {
panic("MQ Pass is missing")
}
pass = conf.Pass
if conf.Host == "" {
panic("MQ Host is missing")
}
host = conf.Host
if conf.Port < 1 {
panic("MQ Port is missing")
}
port = fmt.Sprintf("%d", conf.Port)
vhost = conf.Vhost
dialaddr := scheme + "://" + user + ":" + pass + "@" + host + ":" + port + "/" + vhost
timeout, _ := time.ParseDuration(conf.Timeout)
var dialConfig amqp.Config
dialConfig.Heartbeat = timeout
dialConfig.Dial = func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
if conf.UseTLS {
// import the client certificates
cert, err := tls.LoadX509KeyPair(conf.TLScert, conf.TLSkey)
if err != nil {
panic(err)
}
// import the ca cert
data, err := ioutil.ReadFile(conf.CAcert)
ca := x509.NewCertPool()
if ok := ca.AppendCertsFromPEM(data); !ok {
panic("failed to import CA Certificate")
}
TLSconfig := tls.Config{Certificates: []tls.Certificate{cert},
RootCAs: ca,
InsecureSkipVerify: false,
Rand: rand.Reader}
dialConfig.TLSClientConfig = &TLSconfig
}
// Setup the AMQP broker connection
amqpConn, err := amqp.DialConfig(dialaddr, dialConfig)
if err != nil {
panic(err)
}
amqpChan, err = amqpConn.Channel()
if err != nil {
panic(err)
}
return
}
func InitMqWithConsumer(conf MqConf, name, key string) (consumer <-chan amqp.Delivery, err error) {
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("worker.InitMqWithConsumer() -> %v", e)
}
}()
amqpChan, err := InitMQ(conf)
if err != nil {
panic(err)
}
_, err = amqpChan.QueueDeclare(name, true, false, false, false, nil)
if err != nil {
panic(err)
}
err = amqpChan.QueueBind(name, key, "migevent", false, nil)
if err != nil {
panic(err)
}
err = amqpChan.Qos(0, 0, false)
if err != nil {
panic(err)
}
consumer, err = amqpChan.Consume(name, "", true, false, false, false, nil)
if err != nil {
panic(err)
}
return
}