зеркало из https://github.com/mozilla/mig.git
516 строки
38 KiB
HTML
516 строки
38 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<link href="docstyle.css" rel="stylesheet" />
|
||
<title>Mozilla InvestiGator Configuration Documentation</title>
|
||
<meta content="Julien Vehent <jvehent@mozilla.com>" name="author" />
|
||
</head>
|
||
<body>
|
||
<h1>Mozilla InvestiGator Configuration Documentation</h1>
|
||
<aside class="topic contents" id="table-of-contents">
|
||
<h1>Table of Contents</h1>
|
||
<ul class="auto-toc">
|
||
<li><a href="#the-quick-compilation-doc">1 The quick compilation doc</a></li>
|
||
<li>
|
||
<p><a href="#agent-configuration">2 Agent Configuration</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#amqps-configuration">2.1 AMQPS configuration</a></li>
|
||
<li><a href="#proxy-support">2.2 Proxy support</a></li>
|
||
<li><a href="#stat-socket">2.3 Stat socket</a></li>
|
||
<li><a href="#logging">2.4 Logging</a></li>
|
||
<li><a href="#access-control-lists">2.5 Access Control Lists</a></li>
|
||
<li><a href="#investigators-s-public-keys">2.6 Investigators's public keys</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p><a href="#build-instructions">3 Build instructions</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#build-agent-with-specific-configuration-file">3.1 Build agent with specific configuration file</a></li>
|
||
<li><a href="#agent-external-configuration-file">3.2 Agent external configuration file</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p><a href="#scheduler-configuration">4 Scheduler Configuration</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#spool-directories">4.1 Spool directories</a></li>
|
||
<li><a href="#whitelist">4.2 Whitelist</a></li>
|
||
<li><a href="#database-creation">4.3 Database creation</a></li>
|
||
<li><a href="#database-tuning">4.4 Database tuning</a></li>
|
||
<li><a href="#id1">4.5 Logging</a></li>
|
||
<li><a href="#id2">4.6 AMQPS configuration</a></li>
|
||
<li><a href="#collector">4.7 Collector</a></li>
|
||
<li><a href="#pgp">4.8 PGP</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p><a href="#rabbitmq-configuration">5 RabbitMQ Configuration</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#installation">5.1 Installation</a></li>
|
||
<li><a href="#rabbitmq-permissions">5.2 RabbitMQ Permissions</a></li>
|
||
<li><a href="#rabbitmq-tls-configuration">5.3 RabbitMQ TLS configuration</a></li>
|
||
<li><a href="#queues-mirroring">5.4 Queues mirroring</a></li>
|
||
<li><a href="#cluster-management">5.5 Cluster management</a></li>
|
||
<li><a href="#serving-amqps-on-port-443">5.6 Serving AMQPS on port 443</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p><a href="#api-configuration">6 API configuration</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#location">6.1 Location</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</aside>
|
||
<p>This document describes the steps to build and configure a complete MIG platform.</p>
|
||
<section id="the-quick-compilation-doc">
|
||
<h2>1 The quick compilation doc</h2>
|
||
<p>First, install Go:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>wget https://storage.googleapis.com/golang/go1.2.2.linux-amd64.tar.gz
|
||
<span class="nv">$ </span>tar -xzvf go1.2.2.linux-amd64.tar.gz
|
||
<span class="nv">$ </span><span class="nb">export </span><span class="nv">GOROOT</span><span class="o">=</span>~/go
|
||
<span class="nv">$ </span><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span>:<span class="nv">$HOME</span>/go/bin
|
||
<span class="nv">$ </span>go version
|
||
go version go1.2.2 linux/amd64</code></pre>
|
||
<p>Then, download MIG:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>git clone git@github.com:mozilla/mig.git</code></pre>
|
||
<p>Download the dependencies:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make go_get_deps
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u code.google.com/p/go.crypto/openpgp
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u github.com/streadway/amqp
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u github.com/lib/pq
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u github.com/howeyc/fsnotify
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u code.google.com/p/gcfg
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u github.com/gorilla/mux
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u github.com/jvehent/cljs
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u bitbucket.org/kardianos/osext
|
||
<span class="nv">GOPATH</span><span class="o">=</span>/home/jvehent/mig go get -u bitbucket.org/kardianos/service</code></pre>
|
||
<p>Build the scheduler or the API:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make mig-scheduler</code></pre>
|
||
<p>That's it. Now to build the agent, you need to perform some configuration first.</p>
|
||
</section>
|
||
<section id="agent-configuration">
|
||
<h2>2 Agent Configuration</h2>
|
||
<p>The MIG Agent configuration must be prepared before build. The configuration is hardwired into the agent, such that no external file is required to run it.</p>
|
||
<p>TLS Certificates, PGP public keys and configuration variables would normally be stored in external files, that would make installing an agent on an endpoint more complex. The approach of building all of the configuration parameters into the agent means that we can ship a single binary that is self-sufficient. Go's approach to statically built binary also helps greatly eliminate the need for external dependencies. One the agent is built, ship it to an endpoint, run it, and you're done.</p>
|
||
<p>A template of agent configuration is in 'conf/mig-agent-conf.go.inc'. Copy this to 'conf/mig-agent-conf.go' and edit the file. Make sure to respect Go syntax format.</p>
|
||
<pre><code class="bash">git clone git@github.com:mozilla/mig.git
|
||
cp conf/mig-agent-conf.go<span class="o">{</span>.inc,<span class="o">}</span>
|
||
vim mig-agent-conf.go</code></pre>
|
||
<p>Later on, when you run 'make mig-agent', the Makefile will copy the agent configuration to the agent source code, and build the binary. If the configuration file is missing, Makefile will alert you. If you have an error in the format of the file, the Go compiler will return a list of compilation errors for you to fix.</p>
|
||
<section id="amqps-configuration">
|
||
<h3>2.1 AMQPS configuration</h3>
|
||
<p>TLS support between agents and rabbitmq is optional, but strongly recommended. If you want to use TLS, you need to import the PEM encoded client certificate, client key and CA certificate into 'mig-agent-conf.go'.</p>
|
||
<ol type="1">
|
||
<li><strong>CACERT</strong> must contain the PEM encoded certificate of the Root CA.</li>
|
||
<li><strong>AGENTCERT</strong> must contain the PEM encoded client certificate of the agent.</li>
|
||
<li><strong>AGENTKEY</strong> must contain the PEM encoded client certificate of the agent.</li>
|
||
</ol>
|
||
<p>You also need to edit the <strong>AMQPBROKER</strong> variable to invoke <strong>amqps</strong> instead of the regular amqp mode. You probably also want to change the port from 5672 (default amqp) to 5671 (default amqps).</p>
|
||
</section>
|
||
<section id="proxy-support">
|
||
<h3>2.2 Proxy support</h3>
|
||
<p>The agent supports connecting to the relay via a CONNECT proxy. It will attempt a direct connection first, and if this fails, will look for the environment variable <cite>HTTP_PROXY</cite> to use as a proxy. A list of proxies can be manually added to the configuration of the agent in the <cite>PROXIES</cite> parameters. These proxies will be used if the two previous connections fail.</p>
|
||
<p>An agent using a proxy will reference the name of the proxy in the environment fields of the heartbeat sent to the scheduler.</p>
|
||
</section>
|
||
<section id="stat-socket">
|
||
<h3>2.3 Stat socket</h3>
|
||
<p>The agent can establish a listening TCP socket on localhost for management purpose. The list of supported operations can be obtained by sending the keyword <cite>help</cite> to this socket.</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>nc localhost 51664 <span class="o"><<<</span> <span class="nb">help
|
||
|
||
</span>Welcome to the MIG agent socket. The commands are:
|
||
pid returns the PID of the running agent</code></pre>
|
||
<p>To obtain the PID of the running agent, use the following command:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>nc localhost 51664 <span class="o"><<<</span> pid ; <span class="nb">echo
|
||
</span>9792</code></pre>
|
||
<p>Leave the <cite>SOCKET</cite> configuration variable empty to disable the stat socket.</p>
|
||
</section>
|
||
<section id="logging">
|
||
<h3>2.4 Logging</h3>
|
||
<p>The agent can log to stdout, to a file or to the system logging. On Windows, the system logging is the Event log. On POSIX systems, it's syslog.</p>
|
||
<p>The <cite>LOGGINGCONF</cite> parameter is used to configure the proper logging level.</p>
|
||
</section>
|
||
<section id="access-control-lists">
|
||
<h3>2.5 Access Control Lists</h3>
|
||
<p>see <a href="concepts.rst">concepts: Access Control Lists</a></p>
|
||
</section>
|
||
<section id="investigators-s-public-keys">
|
||
<h3>2.6 Investigators's public keys</h3>
|
||
<p>The public keys of all investigators must be listed in the <cite>PUBLICPGPKEYS</cite> array. Each key is its own entry in the array. To export a public key in the proper format, use the command:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>gpg --export -a jvehent@mozilla.com
|
||
|
||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||
Version: GnuPG v1
|
||
|
||
mQENBFF/69EBCADe79sqUKJHXTMW3tahbXPdQAnpFWXChjI9tOGbgxmse1eEGjPZ
|
||
QPFOPgu3O3iij6UOVh+LOkqccjJ8gZVLYMJzUQC+2RJ3jvXhti8xZ1hs2iEr65Rj
|
||
zUklHVZguf2Zv2X9Er8rnlW5xzplsVXNWnVvMDXyzx0ufC00dDbCwahLQnv6Vqq8
|
||
etc...</code></pre>
|
||
<p>Then insert the whole, with header and footer, into the array:</p>
|
||
<pre><code class="bash">// PGP public key that is authorized to sign actions
|
||
var <span class="nv">PUBLICPGPKEYS</span> <span class="o">=</span> <span class="o">[</span>...<span class="o">]</span>string<span class="o">{</span>
|
||
<span class="sb">`</span>-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||
Version: GnuPG v1 - bob.kelso@mozilla.com
|
||
|
||
mQENBFF/69EBCADe79sqUKJHXTMW3tahbXPdQAnpFWXChjI9tOGbgxmse1eEGjPZ
|
||
<span class="o">=</span>3tGV
|
||
-----END PGP PUBLIC KEY BLOCK-----
|
||
<span class="sb">`</span>,
|
||
<span class="sb">`</span>
|
||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||
Version: GnuPG v1. Name: sam.axe@mozilla.com
|
||
|
||
mQINBE5bjGABEACnT9K6MEbeDFyCty7KalsNnMjXH73kY4B8aJXbE6SSnRA3gWpa
|
||
-----END PGP PUBLIC KEY BLOCK-----<span class="sb">`</span><span class="o">}</span></code></pre>
|
||
</section>
|
||
</section>
|
||
<section id="build-instructions">
|
||
<h2>3 Build instructions</h2>
|
||
<p>To build MIG, you need Go version 1.2 or superior. External Go dependencies can be resolved by running <cite>make go_get_deps</cite>:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make go_get_deps
|
||
<span class="nv">GOPATH</span><span class="o">=</span>. go get -u code.google.com/p/go.crypto/openpgp
|
||
<span class="nv">GOPATH</span><span class="o">=</span>. go get -u github.com/streadway/amqp
|
||
...</code></pre>
|
||
<p>Each component of MIG can be built independently with 'make mig-action-generator', 'make mig-action-verifier', 'make mig-scheduler', 'make mig-api', 'make mig-console' and 'make mig-agent'. To build the entire platform, simply run 'make'.</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make</code></pre>
|
||
<p>Built binaries will be placed in <strong>bin/linux/amd64/</strong> (or in a similar directory if you are building on a different platform).</p>
|
||
<section id="build-agent-with-specific-configuration-file">
|
||
<h3>3.1 Build agent with specific configuration file</h3>
|
||
<p>Use the AGTCONF make variable to specify a different path than 'conf/mig-agent-conf.go'.</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make mig-agent <span class="nv">AGTCONF</span><span class="o">=</span>conf/mig-agent-conf.dev.go <span class="nv">BUILDENV</span><span class="o">=</span>dev</code></pre>
|
||
<p>To cross-compile for a different platform, use the <cite>ARCH</cite> and <cite>OS</cite> make variables:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>make mig-agent <span class="nv">AGTCONF</span><span class="o">=</span>conf/mig-agent-conf.prod.go <span class="nv">BUILDENV</span><span class="o">=</span>prod <span class="nv">OS</span><span class="o">=</span>windows <span class="nv">ARCH</span><span class="o">=</span>amd64</code></pre>
|
||
</section>
|
||
<section id="agent-external-configuration-file">
|
||
<h3>3.2 Agent external configuration file</h3>
|
||
<p>It is possible to use a configuration file with the agent. The location of the file can be specified using the <cite>-c</cite> flag of the agent's binary. If no flag is specific, the agent will look for a configuration file at <cite>/etc/mig/mig-agent.cfg</cite>. If no file is found at this location, the builtin parameters are used.</p>
|
||
<p>The following parameters are <strong>not</strong> controlable by the configuration file:</p>
|
||
<ul>
|
||
<li>list of investigators public keys in <cite>PUBLICPGPKEYS</cite></li>
|
||
<li>list of access control lists in <cite>AGENTACL</cite></li>
|
||
<li>list of proxies in <cite>PROXIES</cite></li>
|
||
</ul>
|
||
<p>All other parameters can be overriden in the configuration file. Check out the sample file <cite>mig-agent.cfg.inc</cite> in the <strong>conf</strong> folder.</p>
|
||
</section>
|
||
</section>
|
||
<section id="scheduler-configuration">
|
||
<h2>4 Scheduler Configuration</h2>
|
||
<p>The scheduler template configuration is in 'conf/mig-scheduler.cfg.inc'. It must be copied to a location of your choice, and edited.</p>
|
||
<section id="spool-directories">
|
||
<h3>4.1 Spool directories</h3>
|
||
<p>The scheduler and the API share a spool for actions and commands that are active in the MIG platform. You need to create that spool on your server, the recommended location is <cite>/var/cache/mig</cite>, but feel free to update that to your needs.</p>
|
||
<pre><code class="bash">sudo mkdir -p /var/cache/mig/<span class="o">{</span>action/new,action/done,action/inflight,action/invalid,command/done,command/inflight,command/ready,command/returned<span class="o">}</span>
|
||
|
||
sudo chown mig-user /var/cache/mig -R</code></pre>
|
||
</section>
|
||
<section id="whitelist">
|
||
<h3>4.2 Whitelist</h3>
|
||
<p>Agents' names must be listed in a whitelist file for the scheduler to accept their registrations. The location of the whitelist is configurable, but a good place for it is in <cite>/var/cache/mig/agents_whitelist.txt</cite>. The file contains one agent name on each line. The agent name is taken from the hostname the agent runs on.</p>
|
||
<blockquote>
|
||
<dl>
|
||
<dt>::</dt>
|
||
<dd>agent123.example.net db4.sub.example.com ...</dd>
|
||
</dl>
|
||
</blockquote>
|
||
</section>
|
||
<section id="database-creation">
|
||
<h3>4.3 Database creation</h3>
|
||
<p>The dabase for MIG is PostgreSQL. If you are using a local postgres database, you can run the script in <a href=".files/createdb.sh">doc/.files/createdb.sh</a>, which will create the database and 3 users: <cite>migadmin</cite>, <cite>migscheduler</cite> and <cite>migapi</cite>. Each user has different permissions on the database.</p>
|
||
<p>If you are using a remote database, create the database <cite>mig</cite> and user <cite>migadmin</cite>, the run the script from <a href=".files/createremotedb.sh">doc/.files/createremotedb.sh</a> that will create the tables, users and permissions. This approach works well with Amazon RDS.</p>
|
||
<p>Edit the variables in the script <cite>createremotedb.sh</cite>:</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>vim createremotedb.sh
|
||
|
||
<span class="nv">PGDATABASE</span><span class="o">=</span><span class="s1">'mig'</span>
|
||
<span class="nv">PGUSER</span><span class="o">=</span><span class="s1">'migadmin'</span>
|
||
<span class="nv">PGPASS</span><span class="o">=</span><span class="s1">'MYDATABASEPASSWORD'</span>
|
||
<span class="nv">PGHOST</span><span class="o">=</span><span class="s1">'192.168.0.1'</span>
|
||
<span class="nv">PGPORT</span><span class="o">=</span>5432</code></pre>
|
||
<p>Then run it against your database server.</p>
|
||
<pre><code class="bash"><span class="nv">$ </span>which psql
|
||
/usr/bin/psql
|
||
|
||
<span class="nv">$ </span>bash createremotedb.sh
|
||
|
||
<span class="o">[</span>... bunch of sql queries ...<span class="o">]</span>
|
||
|
||
created users: migscheduler/4NvQFdwdQ8UOU4ekEOgWDWi3gzG5cg2X migapi/xcJyJhLg1cldIp7eXcxv0U-UqV80tMb-</code></pre>
|
||
<p>The <cite>migscheduler</cite> and <cite>migapi</cite> users can now be added to the configuration files or the scheduler and the api.</p>
|
||
<blockquote>
|
||
<pre>[postgres]
|
||
host = "192.168.0.1"
|
||
port = 5432
|
||
dbname = "mig"
|
||
user = "migapi"
|
||
password = "xcJyJhLg1cldIp7eXcxv0U-UqV80tMb-"
|
||
sslmode = "verify-full"</pre>
|
||
</blockquote>
|
||
<p>Note that <cite>sslmode</cite> can take the values <cite>disable</cite>, <cite>require</cite> (no cert verification) and <cite>verify-full</cite> (requires cert verification).</p>
|
||
</section>
|
||
<section id="database-tuning">
|
||
<h3>4.4 Database tuning</h3>
|
||
<p>The scheduler has an extra parameter to control the max number of database connections. It's important to keep that number relatively low, and increase it with the size of your infrastructure. The default value is set to <cite>10</cite>, and a good production value is <cite>100</cite>.</p>
|
||
<blockquote>
|
||
<pre>[postgres]
|
||
...
|
||
maxconn = 10</pre>
|
||
</blockquote>
|
||
<p>If the DB insertion rate is lower than the agent heartbeats rate, the scheduler will receive more heartbeats per seconds than it can insert in the database. When that happens, you will see the insertion lag increase in the query below:</p>
|
||
<pre><code class="sql"><span class="n">mig</span><span class="o">=></span> <span class="k">select</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="n">heartbeattime</span> <span class="k">as</span> <span class="ss">"insertion lag"</span>
|
||
<span class="n">mig</span><span class="o">-></span> <span class="k">from</span> <span class="n">agents</span> <span class="k">order</span> <span class="k">by</span> <span class="n">heartbeattime</span> <span class="k">desc</span> <span class="k">limit</span> <span class="mi">1</span><span class="p">;</span>
|
||
<span class="n">insertion</span> <span class="n">lag</span>
|
||
<span class="c1">-----------------
|
||
</span> <span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">.</span><span class="mi">212257</span>
|
||
<span class="p">(</span><span class="mi">1</span> <span class="k">row</span><span class="p">)</span></code></pre>
|
||
<p>A healthy insertion lag should be below one second. If the lag increases, and your DB server still isn't stuck at 100% CPU, try increasing the value of <cite>maxconn</cite>. It will cause the scheduler to use more insertion threads.</p>
|
||
</section>
|
||
<section id="id1">
|
||
<h3>4.5 Logging</h3>
|
||
<p>The scheduler can log to stdout, syslog, or a target file. It will run in foreground if the logging mode is set to 'stdout'. For the scheduler to run as a daemon, set the mode to 'file' or 'syslog'.</p>
|
||
<blockquote>
|
||
<pre>[logging]
|
||
; select a mode between 'stdout', 'file' and 'syslog
|
||
; for syslog, logs go into local3
|
||
mode = "syslog"
|
||
level = "debug"
|
||
host = "localhost"
|
||
port = 514
|
||
protocol = "udp"</pre>
|
||
</blockquote>
|
||
</section>
|
||
<section id="id2">
|
||
<h3>4.6 AMQPS configuration</h3>
|
||
<p>TLS support between the scheduler and rabbitmq is optional but strongly recommended. To enable it, generate a client certificate and set the [mq] configuration section of the scheduler as follow:</p>
|
||
<blockquote>
|
||
<pre>[mq]
|
||
host = "relay1.mig.example.net"
|
||
port = 5671
|
||
user = "scheduler"
|
||
pass = "secretrabbitmqpassword"
|
||
vhost = "mig"
|
||
|
||
; TLS options
|
||
usetls = true
|
||
cacert = "/etc/mig/scheduler/cacert.pem"
|
||
tlscert = "/etc/mig/scheduler/scheduler-amqps.pem"
|
||
tlskey = "/etc/mig/scheduler/scheduler-amqps-key.pem"</pre>
|
||
</blockquote>
|
||
<p>Make sure to use <strong>fully qualified paths</strong> otherwise the scheduler will fail to load them after going in the background.</p>
|
||
</section>
|
||
<section id="collector">
|
||
<h3>4.7 Collector</h3>
|
||
<p>The Collector is a routine ran periodically by the scheduler to inspect the content of its spool. It will load files that may have been missed by the file notification routine, and delete old files after a grace period.</p>
|
||
<blockquote>
|
||
<pre>[collector]
|
||
; frequency at which the collector runs
|
||
freq = "60s"
|
||
|
||
; period during which done actions and commands,
|
||
; and invalid actions are kept
|
||
deleteafter = "72h"</pre>
|
||
</blockquote>
|
||
</section>
|
||
<section id="pgp">
|
||
<h3>4.8 PGP</h3>
|
||
<p>The scheduler uses a PGP key to sign agent destruction actions during the agent upgrade protocol. Due to the limited scope of that key, it is stored in the database to facilitate deployment and provisioning of multiple schedulers.</p>
|
||
<p>Upon startup, the scheduler will look for an investigator named <cite>migscheduler</cite> and retrieve its private key to use it in action signing. If no investigator is found, it generates one and inserts it into the database, such that other schedulers can use it as well.</p>
|
||
<p>At the time, the scheduler public key must be manually added into the agent configuration. This will be changed in the future when ACLs and investigators can be dynamically distributed to agents.</p>
|
||
<p>In the ACL of the agent configuration file <cite>conf/mig-agent-conf.go</cite>:</p>
|
||
<blockquote>
|
||
<pre>var AGENTACL = [...]string{
|
||
`{
|
||
"agentdestroy": {
|
||
"minimumweight": 1,
|
||
"investigators": {
|
||
"MIG Scheduler": {
|
||
"fingerprint": "1E644752FB76B77245B1694E556CDD7B07E9D5D6",
|
||
"weight": 1
|
||
}
|
||
}
|
||
}
|
||
}`,
|
||
}</pre>
|
||
</blockquote>
|
||
<p>And add the public PGP key of the scheduler as well:</p>
|
||
<blockquote>
|
||
<pre>// PGP public keys that are authorized to sign actions
|
||
var PUBLICPGPKEYS = [...]string{
|
||
`
|
||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||
Version: GnuPG v1. Name: MIG Scheduler
|
||
|
||
mQENBFF/69EBCADe79sqUKJHXTMW3tahbXPdQAnpFWXChjI9tOGbgxmse1eEGjPZ
|
||
QPFOPgu3O3iij6UOVh+LOkqccjJ8gZVLYMJzUQC+2RJ3jvXhti8xZ1hs2iEr65Rj
|
||
zUklHVZguf2Zv2X9Er8rnlW5xzplsVXNWnVvMDXyzx0ufC00dDbCwahLQnv6Vqq8
|
||
BdUCSrvo/r7oAims8SyWE+ZObC+rw7u01Sut0ctnYrvklaM10+zkwGNOTszrduUy
|
||
.....
|
||
`
|
||
}</pre>
|
||
</blockquote>
|
||
</section>
|
||
</section>
|
||
<section id="rabbitmq-configuration">
|
||
<h2>5 RabbitMQ Configuration</h2>
|
||
<p>All communications between scheduler and agents rely on RabbitMQ's AMQP protocol. While MIG does not rely on the security of RabbitMQ to pass orders to agents, an attacker that gains control to the message broker would be able to listen to all message, or shut down MIG entirely. To prevent this, RabbitMQ must provide a reasonable amount of protection, at two levels:</p>
|
||
<ul>
|
||
<li>All communications on the public internet are authenticated using client and server certificates. Since all agents share a single client certificate, this provides minimal security, and should only be used to make it harder for attackers to establish an AMQP connection with rabbitmq.</li>
|
||
<li>A given agent can listen and write to its own queue, and no other. We accomplish this by adding a random number to the queue ID, which is generated by an agent, and hard to guess by another agent.</li>
|
||
</ul>
|
||
<p>Note that, even if a random agent manages to connect to the relay, the scheduler will accept its registration only if it is present in the scheduler's whitelist.</p>
|
||
<section id="installation">
|
||
<h3>5.1 Installation</h3>
|
||
<p>Install the RabbitMQ server from your distribution's packaging system. If your distribution does not provide a RabbitMQ package, install <cite>erlang</cite> from yum or apt, and then install RabbitMQ using the packages from rabbitmq.com</p>
|
||
</section>
|
||
<section id="rabbitmq-permissions">
|
||
<h3>5.2 RabbitMQ Permissions</h3>
|
||
<ol type="1">
|
||
<li>
|
||
<p>On the rabbitmq server, create three users:</p>
|
||
<blockquote>
|
||
<ul>
|
||
<li><strong>admin</strong>, with the tag 'administrator'</li>
|
||
<li><strong>scheduler</strong> and <strong>agent</strong>, with no tag</li>
|
||
</ul>
|
||
</blockquote>
|
||
</li>
|
||
</ol>
|
||
<p>All three should have strong passwords. The scheduler password goes into the configuration file <cite>conf/mig-scheduler.cfg</cite>, in <cite>[mq] password</cite>. The agent password goes into <cite>conf/mig-agent-conf.go</cite>, in the agent <cite>AMQPBROKER</cite> dial string. The admin password is, of course, for yourself.</p>
|
||
<pre><code class="bash">sudo rabbitmqctl add_user admin SomeRandomPassword
|
||
sudo rabbitmqctl set_user_tags admin administrator
|
||
|
||
sudo rabbitmqctl add_user scheduler SomeRandomPassword
|
||
|
||
sudo rabbitmqctl add_user agent SomeRandomPassword</code></pre>
|
||
<p>You can list the users with the following command:</p>
|
||
<pre><code class="bash">sudo rabbitmqctl list_users</code></pre>
|
||
<p>On fresh installation, rabbitmq comes with a <cite>guest</cite> user that as password <cite>guest</cite> and admin privileges. You may you to delete that account.</p>
|
||
<pre><code class="bash">sudo rabbitmqctl delete_user guest</code></pre>
|
||
<ol start="2" type="1">
|
||
<li>Create a 'mig' virtual host.</li>
|
||
</ol>
|
||
<pre><code class="bash">sudo rabbitmqctl add_vhost mig
|
||
sudo rabbitmqctl list_vhosts</code></pre>
|
||
<ol start="3" type="1">
|
||
<li>Create permissions for the scheduler user. The scheduler is allowed to publish message (write) to the mig exchange. It can also configure and read from the heartbeat and sched queues. The command below sets those permissions.</li>
|
||
</ol>
|
||
<pre><code class="bash">sudo rabbitmqctl set_permissions -p mig scheduler <span class="se">\
|
||
</span><span class="s1">'^mig(|\.(heartbeat|sched\..*))'</span> <span class="se">\
|
||
</span><span class="s1">'^mig.*'</span> <span class="se">\
|
||
</span><span class="s1">'^mig(|\.(heartbeat|sched\..*))'</span></code></pre>
|
||
<ol start="4" type="1">
|
||
<li>Same thing for the agent. The agent is allowed to configure and read on the 'mig.agt.*' resource, and write to the 'mig' exchange.</li>
|
||
</ol>
|
||
<pre><code class="bash">sudo rabbitmqctl set_permissions -p mig agent <span class="se">\
|
||
</span><span class="s2">"^mig\.agt\.*"</span> <span class="se">\
|
||
</span><span class="s2">"^mig*"</span> <span class="se">\
|
||
</span><span class="s2">"^mig(|\.agt\..*)"</span></code></pre>
|
||
<ol start="5" type="1">
|
||
<li>Start the scheduler, it shouldn't return any ACCESS error. You can also list the permissions with the command:</li>
|
||
</ol>
|
||
<pre><code class="bash">sudo rabbitmqctl list_permissions -p mig
|
||
CONFIGURE WRITE READ
|
||
agent ^mig<span class="se">\\</span>.agt<span class="se">\\</span>.* ^mig* ^mig<span class="o">(</span>|<span class="se">\\</span>.agt<span class="se">\\</span>..*<span class="o">)</span>
|
||
scheduler ^mig<span class="o">(</span>|<span class="se">\\</span>.<span class="o">(</span>heartbeat|sched<span class="se">\\</span>..*<span class="o">))</span> ^mig.* ^mig<span class="o">(</span>|<span class="se">\\</span>.<span class="o">(</span>heartbeat|sched<span class="se">\\</span>..*<span class="o">))</span></code></pre>
|
||
</section>
|
||
<section id="rabbitmq-tls-configuration">
|
||
<h3>5.3 RabbitMQ TLS configuration</h3>
|
||
<p>The documentation from rabbitmq has a thorough explanation of SSL support in rabbit at <a href="http://www.rabbitmq.com/ssl.html">http://www.rabbitmq.com/ssl.html</a> . Without going into too much details, we need three things:</p>
|
||
<ol type="1">
|
||
<li>a PKI (and its public cert)</li>
|
||
<li>a server certificate and private key for rabbitmq itself</li>
|
||
<li>a client certificate and private key for the agents</li>
|
||
</ol>
|
||
<p>You can obtain these three things on you own, or follow the openssl tutorial from the rabbitmq documentation. Come back here when you have all three.</p>
|
||
<p>On the rabbitmq server, place the certificates under <strong>/etc/rabbitmq/certs/</strong>.</p>
|
||
<blockquote>
|
||
<pre>/etc/rabbitmq/certs/
|
||
├── cacert.pem
|
||
├── migrelay1.example.net.key
|
||
└── migrelay1.example.net.pem</pre>
|
||
</blockquote>
|
||
<p>Edit (or create) the configuration file of rabbitmq to reference the certificates.</p>
|
||
<blockquote>
|
||
<pre>[
|
||
{rabbit, [
|
||
{ssl_listeners, [5671]},
|
||
{ssl_options, [{cacertfile,"/etc/rabbitmq/certs/cacert.pem"},
|
||
{certfile,"/etc/rabbitmq/certs/migrelay1.example.net.pem"},
|
||
{keyfile,"/etc/rabbitmq/certs/migrelay1.example.net.key"},
|
||
{verify,verify_peer},
|
||
{fail_if_no_peer_cert,true},
|
||
{ciphers, [{dhe_rsa,aes_128_cbc,sha},
|
||
{dhe_rsa,aes_256_cbc,sha},
|
||
{dhe_rsa,'3des_ede_cbc',sha},
|
||
{rsa,aes_128_cbc,sha},
|
||
{rsa,aes_256_cbc,sha},
|
||
{rsa,'3des_ede_cbc',sha}]},
|
||
{versions, [tlsv1]}
|
||
]}
|
||
]}
|
||
].</pre>
|
||
</blockquote>
|
||
<p>Use this command to list the ciphers supported by a rabbitmq server:</p>
|
||
<pre><code class="bash">rabbitmqctl <span class="nb">eval</span> <span class="s1">'ssl:cipher_suites().'</span></code></pre>
|
||
<p>Note: erlang r14B doesn't support TLS 1.1 and 1.2, as returned by the command:</p>
|
||
<pre><code class="bash"><span class="c"># rabbitmqctl eval 'ssl:versions().'
|
||
</span><span class="o">[{</span>ssl_app,<span class="s2">"4.1.6"</span><span class="o">}</span>,<span class="o">{</span>supported,<span class="o">[</span>tlsv1,sslv3<span class="o">]}</span>,<span class="o">{</span>available,<span class="o">[</span>tlsv1,sslv3<span class="o">]}]</span>
|
||
...done.</code></pre>
|
||
<p>That is it for rabbitmq. Go back to the MIG Agent configuration section of this page in order to add the client certificate into your agents.</p>
|
||
</section>
|
||
<section id="queues-mirroring">
|
||
<h3>5.4 Queues mirroring</h3>
|
||
<p>By default, queues within a RabbitMQ cluster are located on a single node (the node on which they were first declared). If that node goes down, the queue will become unavailable. To mirror all MIG queues to all nodes of a rabbitmq cluster, use the following policy:</p>
|
||
<pre><code class="bash"><span class="c"># rabbitmqctl -p mig set_policy mig-mirror-all "^mig\." '{"ha-mode":"all"}'
|
||
</span>Setting policy <span class="s2">"mig-mirror-all"</span> <span class="k">for </span>pattern <span class="s2">"^mig\\."</span> to <span class="s2">"{\"ha-mode\":\"all\"}"</span> with priority <span class="s2">"0"</span> ...
|
||
...done.</code></pre>
|
||
</section>
|
||
<section id="cluster-management">
|
||
<h3>5.5 Cluster management</h3>
|
||
<p>To create a cluster, all rabbitmq nodes must share a secret called erlang cookie. The erlang cookie is located in <cite>/var/lib/rabbitmq/.erlang.cookie</cite>. Make sure the value of the cookie is identical on all members of the cluster, then tell one node to join another one:</p>
|
||
<pre><code class="bash"><span class="c"># rabbitmqctl stop_app
|
||
</span>Stopping node <span class="s1">'rabbit@ip-172-30-200-73'</span> ...
|
||
...done.
|
||
|
||
<span class="c"># rabbitmqctl join_cluster rabbit@ip-172-30-200-42
|
||
</span>Clustering node <span class="s1">'rabbit@ip-172-30-200-73'</span> with <span class="s1">'rabbit@ip-172-30-200-42'</span> ...
|
||
...done.
|
||
|
||
<span class="c"># rabbitmqctl start_app
|
||
</span>Starting node <span class="s1">'rabbit@ip-172-30-200-73'</span> ...
|
||
...done.</code></pre>
|
||
<p>To remove a dead node from the cluster, use the following command from any active node of the running cluster.</p>
|
||
<pre><code class="bash"><span class="c"># rabbitmqctl forget_cluster_node rabbit@ip-172-30-200-84</span></code></pre>
|
||
<p>If one node of the cluster goes down, and the agents have trouble reconnecting, they may throw the error <cite>NOT_FOUND - no binding mig.agt....</cite>. That happens when the binding in question exists but the 'home' node of the (durable) queue is not alive. In case of a mirrored queue that would imply that all mirrors are down. Essentially both the queue and associated bindings are in a limbo state at that point - they neither exist nor do they not exist. <a href="http://rabbitmq.1065348.n5.nabble.com/Can-t-Bind-After-Upgrading-from-3-1-1-to-3-1-5-td29793.html">source</a></p>
|
||
<p>The safest thing to do is to delete all the queues on the cluster, and restart the scheduler. The agents will restart themselves.</p>
|
||
<pre><code class="bash"><span class="c"># for queue in $(rabbitmqctl list_queues -p mig|grep ^mig|awk '{print $1}')
|
||
</span><span class="k">do
|
||
</span><span class="nb">echo </span>curl -i -u admin:adminpassword -H <span class="s2">"content-type:application/json"</span> <span class="se">\
|
||
</span> -XDELETE http://localhost:15672/api/queues/mig/<span class="nv">$queue</span>;
|
||
<span class="k">done</span></code></pre>
|
||
<p>(remove the <cite>echo</cite> in the command above, it's there as a safety for copy/paste people).</p>
|
||
</section>
|
||
<section id="serving-amqps-on-port-443">
|
||
<h3>5.6 Serving AMQPS on port 443</h3>
|
||
<p>To prevent yours agents from getting blocked by firewalls, it may be a good idea to use port 443 for connections between agents and rabbitmq. However, rabbitmq is not designed to run on a privileged port. The solution, then, is to use iptables to redirect the port on the rabbitmq server.</p>
|
||
<pre><code class="bash">iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 5671 -m comment --comment <span class="s2">"Serve RabbitMQ on HTTPS port"</span></code></pre>
|
||
</section>
|
||
</section>
|
||
<section id="api-configuration">
|
||
<h2>6 API configuration</h2>
|
||
<p>The REST API exposes functions to create, delete and query actions remotely. It is the primary interface to the Scheduler.</p>
|
||
<section id="location">
|
||
<h3>6.1 Location</h3>
|
||
<p>Most likely, the API will be deployed behind some form of reverse proxy. The API doesn't attempt to guess its location. Instead, you can configure it in <cite>mig-api.cfg</cite>, as follow:</p>
|
||
<blockquote>
|
||
<pre>[server]
|
||
ip = "127.0.0.1"
|
||
port = 12345
|
||
host = "http://localhost:12345"
|
||
baseroute = "/api/v1"</pre>
|
||
</blockquote>
|
||
<p><cite>ip</cite> and <cite>port</cite> define the socket the API will be listening on. <cite>host</cite> is the public URL of the API, that clients will be connecting to. <cite>baseroute</cite> is the location of the base of the API, without the trailing slash.</p>
|
||
<p>In this example, to reach the home of the API, we would point our browser to <cite>http://localhost:12345/api/v1/</cite>.</p>
|
||
<p>Note that the API does not support SSL, or authentication (for now). This need to be configured on a reverse proxy in front of it.</p>
|
||
</section>
|
||
</section>
|
||
</body>
|
||
</html> |