зеркало из https://github.com/mozilla/mig.git
114 строки
8.6 KiB
HTML
114 строки
8.6 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<link href="docstyle.css" rel="stylesheet" />
|
||
<title>MIG Data</title>
|
||
<meta content="Julien Vehent <jvehent@mozilla.com>" name="author" />
|
||
</head>
|
||
<body>
|
||
<h1>MIG Data</h1>
|
||
<aside class="topic contents" id="table-of-contents">
|
||
<h1>Table of Contents</h1>
|
||
<ul class="auto-toc">
|
||
<li>
|
||
<p><a href="#postgresql">1 Postgresql</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#database-creation-script">1.1 Database creation script</a></li>
|
||
<li><a href="#structure-tables">1.2 Structure & Tables</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p><a href="#queries">2 Queries</a></p>
|
||
<ul class="auto-toc">
|
||
<li><a href="#adding-investigators">2.1 Adding Investigators</a></li>
|
||
<li><a href="#finding-offline-agents">2.2 Finding offline agents</a></li>
|
||
<li><a href="#finding-double-agents">2.3 Finding double agents</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#scheduler-spool">3 Scheduler Spool</a></li>
|
||
</ul>
|
||
</aside>
|
||
<section id="postgresql">
|
||
<h2>1 Postgresql</h2>
|
||
<section id="database-creation-script">
|
||
<h3>1.1 Database creation script</h3>
|
||
<p>Two scripts can be used to create a database for MIG.</p>
|
||
<ul>
|
||
<li><a href="https://github.com/mozilla/mig/blob/master/src/mig/database/createlocaldb.sh">createlocaldb.sh</a> will create a database on an instance of postgresql running locally. This is used by the standalone installation script.</li>
|
||
</ul>
|
||
<ul>
|
||
<li><a href="https://github.com/mozilla/mig/blob/master/src/mig/database/createremotedb.sh">createremotedb.sh</a> will connect to an existing MIG database on a remote postgresql server. This is a standard production setup. It assumes that you have created a database beforehand. You can pass the DB credentials by editing the bash variables at the top of the script before running it.</li>
|
||
</ul>
|
||
</section>
|
||
<section id="structure-tables">
|
||
<h3>1.2 Structure & Tables</h3>
|
||
<p>The full database schema is available as a SQL file in <a href="https://github.com/mozilla/mig/blob/master/src/mig/database/schema.sql">src/mig/database/schema.sql</a>.</p>
|
||
<p>The <cite>actions</cite> table contains the detail of each action ran by the MIG platform. Its structure contains the base action fields found in the json format of an action, plus a number of additional fields such as timestamps and counters.</p>
|
||
<p>The <cite>agents</cite> table contains the registrations of each agents known of the MIG platform.</p>
|
||
<p>The <cite>commands</cite> table contains each action sent to each agent.</p>
|
||
<p><cite>investigators</cite> have a table that contains their public PGP key, and can be used when verifying signatures and generating ACLs.</p>
|
||
<p>The <cite>signatures</cite> table is a junction between an action and the investigators that signed the action.</p>
|
||
</section>
|
||
</section>
|
||
<section id="queries">
|
||
<h2>2 Queries</h2>
|
||
<p>MIG queries are stored separately from the rest of the source code. You can inspect and modify all queries directly in the Go files in <a href="https://github.com/mozilla/mig/tree/master/src/mig/database">src/mig/database</a>.</p>
|
||
<section id="adding-investigators">
|
||
<h3>2.1 Adding Investigators</h3>
|
||
<p>In the future, this will probably be automated via the API. But for now, and until we have a strong authentication mechanism for API calls, it must be done manually in the database.</p>
|
||
<p>Adapt the query below to add a new investigator.</p>
|
||
<pre><code class="sql"><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">investigators</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">pgpfingerprint</span><span class="p">)</span>
|
||
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">'Bob Kelso'</span><span class="p">,</span> <span class="s1">'E608......'</span><span class="p">);</span></code></pre>
|
||
</section>
|
||
<section id="finding-offline-agents">
|
||
<h3>2.2 Finding offline agents</h3>
|
||
<p>The following query retrieves a list of agents that have been online over the last 30 days, but have not sent a heartbeat in the last 5 minutes.</p>
|
||
<pre><code class="sql"><span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
|
||
<span class="k">WHERE</span> <span class="n">name</span> <span class="k">IN</span> <span class="p">(</span>
|
||
<span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
|
||
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">>=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">interval</span> <span class="s1">'30 days'</span>
|
||
<span class="p">)</span> <span class="k">AND</span> <span class="n">name</span> <span class="k">NOT</span> <span class="k">IN</span> <span class="p">(</span>
|
||
<span class="k">SELECT</span> <span class="k">DISTINCT</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">agents</span>
|
||
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">>=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">interval</span> <span class="s1">'5 minutes'</span>
|
||
<span class="p">)</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="n">name</span><span class="p">;</span></code></pre>
|
||
</section>
|
||
<section id="finding-double-agents">
|
||
<h3>2.3 Finding double agents</h3>
|
||
<p>Sometimes during upgrades the older agent isn't shut down. You can find these endpoints with double agents in the database because each agent sends separate heartbeats for the same endpoint:</p>
|
||
<pre><code class="sql"><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">queueloc</span><span class="p">),</span> <span class="n">queueloc</span> <span class="k">FROM</span> <span class="n">agents</span>
|
||
<span class="k">WHERE</span> <span class="n">heartbeattime</span> <span class="o">>=</span> <span class="n">NOW</span><span class="p">()</span> <span class="o">-</span> <span class="nb">INTERVAL</span> <span class="s1">'10 minutes'</span>
|
||
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">queueloc</span> <span class="k">HAVING</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">queueloc</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span>
|
||
<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">count</span><span class="p">(</span><span class="n">queueloc</span><span class="p">)</span> <span class="k">DESC</span><span class="p">;</span></code></pre>
|
||
<p>This query will list all the agents sorted by the count of agents heartbeatting on each endpoint:</p>
|
||
<pre>| count | name
|
||
|--------+--------------------------------------
|
||
| 3 | puppet3.private.dc1.example.net
|
||
| 2 | mv1.mv.example.net
|
||
| 2 | memcache1.webapp.dc1.example.net
|
||
| 2 | ip2.dc.example.net
|
||
|</pre>
|
||
</section>
|
||
</section>
|
||
<section id="scheduler-spool">
|
||
<h2>3 Scheduler Spool</h2>
|
||
<p>MIG data is stored both on the file system of the scheduler, and in the database. On the scheduler, each action and command are stored individually in a text file in /var/cache/mig (by default).</p>
|
||
<pre><code class="bash"> <span class="nv">$ </span>tree -d /var/cache/mig/
|
||
/var/cache/mig/
|
||
├── action
|
||
│ ├── <span class="k">done</span>
|
||
│ ├── inflight
|
||
│ ├── invalid
|
||
│ └── new
|
||
└── <span class="nb">command</span>
|
||
├── <span class="k">done</span>
|
||
├── inflight
|
||
├── ready
|
||
└── returned
|
||
|
||
10 directories
|
||
|
||
2 | command.private.corp.dc1.example.net</code></pre>
|
||
</section>
|
||
</body>
|
||
</html> |