mig/doc/data.rst.html

114 строки
8.6 KiB
HTML
Исходник Обычный вид История

2014-09-26 01:29:56 +04:00
<!DOCTYPE html>
<html>
2014-09-11 07:56:53 +04:00
<head>
2014-09-26 01:29:56 +04:00
<meta charset="utf-8" />
<link href="docstyle.css" rel="stylesheet" />
<title>MIG Data</title>
<meta content="Julien Vehent &lt;jvehent@mozilla.com&gt;" name="author" />
2014-09-11 07:56:53 +04:00
</head>
<body>
2014-09-26 01:29:56 +04:00
<h1>MIG Data</h1>
<aside class="topic contents" id="table-of-contents">
<h1>Table of Contents</h1>
<ul class="auto-toc">
<li>
2015-01-26 02:45:39 +03:00
<p><a href="#postgresql">1   Postgresql</a></p>
2014-09-26 01:29:56 +04:00
<ul class="auto-toc">
2015-01-26 02:45:39 +03:00
<li><a href="#database-creation-script">1.1   Database creation script</a></li>
<li><a href="#structure-tables">1.2   Structure &amp; Tables</a></li>
2014-09-26 01:29:56 +04:00
</ul>
</li>
<li>
2015-01-26 02:45:39 +03:00
<p><a href="#queries">2   Queries</a></p>
2014-09-26 01:29:56 +04:00
<ul class="auto-toc">
2015-01-26 02:45:39 +03:00
<li><a href="#adding-investigators">2.1   Adding Investigators</a></li>
<li><a href="#finding-offline-agents">2.2   Finding offline agents</a></li>
<li><a href="#finding-double-agents">2.3   Finding double agents</a></li>
2014-09-26 01:29:56 +04:00
</ul>
</li>
2015-01-26 02:45:39 +03:00
<li><a href="#scheduler-spool">3   Scheduler Spool</a></li>
2014-09-26 01:29:56 +04:00
</ul>
</aside>
2015-01-26 02:45:39 +03:00
<section id="postgresql">
<h2>1   Postgresql</h2>
<section id="database-creation-script">
<h3>1.1   Database creation script</h3>
<p>Two scripts can be used to create a database for MIG.</p>
<ul>
<li><a href="https://github.com/mozilla/mig/blob/master/src/mig/database/createlocaldb.sh">createlocaldb.sh</a> will create a database on an instance of postgresql running locally. This is used by the standalone installation script.</li>
</ul>
<ul>
<li><a href="https://github.com/mozilla/mig/blob/master/src/mig/database/createremotedb.sh">createremotedb.sh</a> will connect to an existing MIG database on a remote postgresql server. This is a standard production setup. It assumes that you have created a database beforehand. You can pass the DB credentials by editing the bash variables at the top of the script before running it.</li>
</ul>
2014-09-26 01:29:56 +04:00
</section>
<section id="structure-tables">
2015-01-26 02:45:39 +03:00
<h3>1.2   Structure &amp; Tables</h3>
<p>The full database schema is available as a SQL file in <a href="https://github.com/mozilla/mig/blob/master/src/mig/database/schema.sql">src/mig/database/schema.sql</a>.</p>
2014-09-26 01:29:56 +04:00
<p>The <cite>actions</cite> table contains the detail of each action ran by the MIG platform. Its structure contains the base action fields found in the json format of an action, plus a number of additional fields such as timestamps and counters.</p>
<p>The <cite>agents</cite> table contains the registrations of each agents known of the MIG platform.</p>
<p>The <cite>commands</cite> table contains each action sent to each agent.</p>
<p><cite>investigators</cite> have a table that contains their public PGP key, and can be used when verifying signatures and generating ACLs.</p>
<p>The <cite>signatures</cite> table is a junction between an action and the investigators that signed the action.</p>
</section>
</section>
<section id="queries">
2015-01-26 02:45:39 +03:00
<h2>2   Queries</h2>
<p>MIG queries are stored separately from the rest of the source code. You can inspect and modify all queries directly in the Go files in <a href="https://github.com/mozilla/mig/tree/master/src/mig/database">src/mig/database</a>.</p>
2014-09-26 01:29:56 +04:00
<section id="adding-investigators">
2015-01-26 02:45:39 +03:00
<h3>2.1   Adding Investigators</h3>
2014-09-26 01:29:56 +04:00
<p>In the future, this will probably be automated via the API. But for now, and until we have a strong authentication mechanism for API calls, it must be done manually in the database.</p>
<p>Adapt the query below to add a new investigator.</p>
<pre><code class="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">investigators</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">pgpfingerprint</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">'Bob Kelso'</span><span class="p">,</span> <span class="s1">'E608......'</span><span class="p">);</span></code></pre>
</section>
<section id="finding-offline-agents">
2015-01-26 02:45:39 +03:00
<h3>2.2   Finding offline agents</h3>
2014-09-26 01:29:56 +04:00
<p>The following query retrieves a list of agents that have been online over the last 30 days, but have not sent a heartbeat in the last 5 minutes.</p>
<pre><code class="sql"><span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
<span class="k">WHERE</span> <span class="n">name</span> <span class="k">IN</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">&gt;=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">interval</span> <span class="s1">'30 days'</span>
<span class="p">)</span> <span class="k">AND</span> <span class="n">name</span> <span class="k">NOT</span> <span class="k">IN</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">&gt;=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">interval</span> <span class="s1">'5 minutes'</span>
<span class="p">)</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="n">name</span><span class="p">;</span></code></pre>
</section>
<section id="finding-double-agents">
2015-01-26 02:45:39 +03:00
<h3>2.3   Finding double agents</h3>
2014-09-26 01:29:56 +04:00
<p>Sometimes during upgrades the older agent isn't shut down. You can find these endpoints with double agents in the database because each agent sends separate heartbeats for the same endpoint:</p>
<pre><code class="sql"><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">queueloc</span><span class="p">),</span> <span class="n">queueloc</span> <span class="k">FROM</span> <span class="n">agents</span>
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">&gt;=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">INTERVAL</span> <span class="s1">'10 minutes'</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">queueloc</span> <span class="k">HAVING</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">queueloc</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">count</span><span class="p">(</span><span class="n">queueloc</span><span class="p">)</span> <span class="k">DESC</span><span class="p">;</span></code></pre>
<p>This query will list all the agents sorted by the count of agents heartbeatting on each endpoint:</p>
<pre>| count | name
2014-09-11 07:56:53 +04:00
|--------+--------------------------------------
| 3 | puppet3.private.dc1.example.net
| 2 | mv1.mv.example.net
| 2 | memcache1.webapp.dc1.example.net
| 2 | ip2.dc.example.net
2015-01-26 02:45:39 +03:00
|</pre>
2014-09-26 01:29:56 +04:00
</section>
</section>
2015-01-26 02:45:39 +03:00
<section id="scheduler-spool">
<h2>3   Scheduler Spool</h2>
<p>MIG data is stored both on the file system of the scheduler, and in the database. On the scheduler, each action and command are stored individually in a text file in /var/cache/mig (by default).</p>
<pre><code class="bash"> <span class="nv">$ </span>tree -d /var/cache/mig/
/var/cache/mig/
├── action
│   ├── <span class="k">done</span>
│   ├── inflight
│   ├── invalid
│   └── new
└── <span class="nb">command</span>
├── <span class="k">done</span>
├── inflight
├── ready
└── returned
10 directories
2 | command.private.corp.dc1.example.net</code></pre>
</section>
2014-09-11 07:56:53 +04:00
</body>
2014-09-26 01:29:56 +04:00
</html>