зеркало из https://github.com/mozilla/mig.git
[doc] Added examples to mongodb query cookbook
This commit is contained in:
Julien Vehent
Родитель
c808bc9d0e
Коммит
1efcfb1fdb
273
doc/data.rst
273
doc/data.rst
|
|
@ -23,8 +23,8 @@ the scheduler, each action and command are stored individually in a text file in
|
||||||
|
|
||||||
10 directories
|
10 directories
|
||||||
|
|
||||||
MongoDB
|
MongoDB Cookbook
|
||||||
-------
|
----------------
|
||||||
|
|
||||||
Data is also stored in MongoDB. Actions and Commands have their own separate
|
Data is also stored in MongoDB. Actions and Commands have their own separate
|
||||||
collections. MongoDB is also used to store agent's keepalives.
|
collections. MongoDB is also used to store agent's keepalives.
|
||||||
|
|
@ -67,12 +67,10 @@ See the **concepts** documentation for details on the action and command fields.
|
||||||
|
|
||||||
.. code:: javascript
|
.. code:: javascript
|
||||||
|
|
||||||
> var actionFoundSomething = db.commands.find(
|
> var actionFoundSomething = db.commands.find({
|
||||||
... {
|
'action.id': 5978800913477866376,
|
||||||
... 'action.id': 5978800913477866376,
|
'results.0.FoundAnything': true
|
||||||
... 'results.0.FoundAnything': true
|
});
|
||||||
... }
|
|
||||||
... );
|
|
||||||
|
|
||||||
The results are stored in a variable called 'actionFoundSomething', that is
|
The results are stored in a variable called 'actionFoundSomething', that is
|
||||||
created inside the mongo shell. We can iterate through the results of the query,
|
created inside the mongo shell. We can iterate through the results of the query,
|
||||||
|
|
@ -81,10 +79,10 @@ and print the list of agents, using a second shell command:
|
||||||
.. code:: javascript
|
.. code:: javascript
|
||||||
|
|
||||||
> actionFoundSomething.forEach(
|
> actionFoundSomething.forEach(
|
||||||
... function(currentcommand){
|
function(currentcommand){
|
||||||
... ... print(currentcommand.agentname);
|
print(currentcommand.agentname);
|
||||||
... }
|
}
|
||||||
... );
|
);
|
||||||
|
|
||||||
To print the result of the commands ran on the agent, we need to know the
|
To print the result of the commands ran on the agent, we need to know the
|
||||||
parameters of the action that was sent. In this case, the 'filechecker' module
|
parameters of the action that was sent. In this case, the 'filechecker' module
|
||||||
|
|
@ -209,5 +207,256 @@ to parse, so the generic mongodb query below will do it for you:
|
||||||
|
|
||||||
{ test identifier } {mode} { test value } {counter} { file } {agent name}
|
{ test identifier } {mode} { test value } {counter} { file } {agent name}
|
||||||
|
|
||||||
|
**Find the agents where a given action has not completed**
|
||||||
|
|
||||||
|
One action spans one command per agent. Since each command is stored
|
||||||
|
individually in the database, we can write a query that count commands per
|
||||||
|
status.
|
||||||
|
|
||||||
|
.. code:: javascript
|
||||||
|
|
||||||
|
> db.commands.group({
|
||||||
|
key: {status: 1},
|
||||||
|
cond: { 'action.id': 5979546396770985756},
|
||||||
|
reduce: function(cur, result){ result.count++ },
|
||||||
|
initial: { count: 0}
|
||||||
|
})
|
||||||
|
|
||||||
|
[ { "status" : "succeeded", "count" : 149 } ]
|
||||||
|
|
||||||
|
If the action was still running on some agents, the result would be:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"status" : "sent",
|
||||||
|
"count" : 11
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"status" : "succeeded",
|
||||||
|
"count" : 138
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
If the action had timed out on some agent, we would get:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"status" : "timeout",
|
||||||
|
"count" : 4
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"status" : "succeeded",
|
||||||
|
"count" : 145
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
We can list the names of the agents where the action timed out:
|
||||||
|
|
||||||
|
.. code:: javascript
|
||||||
|
|
||||||
|
> var actionTimedOut = db.commands.find({'action.id': 5979555683008369409,'status': 'timeout'});
|
||||||
|
|
||||||
|
> actionTimedOut.forEach(function(cmd){ print(cmd.agentname);});
|
||||||
|
|
||||||
|
someagent123.example.net
|
||||||
|
someagent567.datacenter1.example.com
|
||||||
|
someagent2912.datacenter2.example.net
|
||||||
|
server5.example.com
|
||||||
|
|
||||||
|
**List the commands that have run on a specific agent in the last 10 minutes**
|
||||||
|
|
||||||
|
The query below returns the detail of each command that was ran on agent
|
||||||
|
'server1234.example.net' in the last 10 minutes. You can see the full command,
|
||||||
|
including the action embedded in it, in the output.
|
||||||
|
|
||||||
|
.. code:: javascript
|
||||||
|
|
||||||
|
> db.commands.find({
|
||||||
|
'action.validfrom': {
|
||||||
|
$gt: new Date(ISODate().getTime() - 1000 * 60 * 10)
|
||||||
|
},
|
||||||
|
'agentname': 'server1234.example.net'
|
||||||
|
}).pretty()
|
||||||
|
|
||||||
|
{
|
||||||
|
"_id" : ObjectId("52fba637fbfe511da91fd447"),
|
||||||
|
"id" : NumberLong("5979555681133474350"),
|
||||||
|
"action" : {
|
||||||
|
"id" : NumberLong("5979555683008369409"),
|
||||||
|
"name" : "secrets,keys and other goodies stored in homedir",
|
||||||
|
"target" : "linux",
|
||||||
|
"description" : {
|
||||||
|
"author" : "Julien Vehent",
|
||||||
|
"email" : "ulfr@mozilla.com",
|
||||||
|
"url" : "",
|
||||||
|
"revision" : NumberLong("201402121138")
|
||||||
|
},
|
||||||
|
"threat" : {
|
||||||
|
"level" : "info",
|
||||||
|
"family" : "compliance"
|
||||||
|
},
|
||||||
|
"validfrom" : ISODate("2014-02-12T16:49:53.551Z"),
|
||||||
|
"expireafter" : ISODate("2014-02-12T17:19:53.551Z"),
|
||||||
|
"operations" : [
|
||||||
|
{
|
||||||
|
"module" : "filechecker",
|
||||||
|
"parameters" : {
|
||||||
|
"/home" : {
|
||||||
|
"filename" : {
|
||||||
|
"DB file" : [
|
||||||
|
"\\.db\\$"
|
||||||
|
],
|
||||||
|
"Dump file" : [
|
||||||
|
"\\.dump\\$"
|
||||||
|
],
|
||||||
|
"SQL file" : [
|
||||||
|
"\\.sql\\$"
|
||||||
|
],
|
||||||
|
"gnupg secring" : [
|
||||||
|
"secring.gpg"
|
||||||
|
],
|
||||||
|
"key file" : [
|
||||||
|
"\\.key\\$"
|
||||||
|
],
|
||||||
|
"log file" : [
|
||||||
|
"\\.log\\$"
|
||||||
|
],
|
||||||
|
"password file" : [
|
||||||
|
"password"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"regex" : {
|
||||||
|
"cleartext RSA private key" : [
|
||||||
|
"-----BEGIN RSA PRIVATE KEY-----"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"pgpsignature" : "iQEcBAEBCAAGBQJS+6YxAAoJEKPWUhc7dj6PHSEIAM5AAF4yIZqvV8bLjY0xjjUfjubIEsH0s0ZlXtQOaqbE8x0DL4CmeDR4d8Z8zGwFO+VGrcRa/axAe3486DjBijpKWnNxYbFhrsLroExrSwxJOGogglJeZMT+/grnxVwU7r2z0aY90tld5KXNhEbwdE84re+4fKq9tdS4C3f+sWR1ZwZ2L2wKrNWP6uByI031z2WDcn8osLJjbyZUYNW7HdkrMiM+n/oZzMpKvqyoKvAPrlYNNQgztOhgw3OHDoeXYKhV2MsQJuFcNMGf87ebUJkhUbbzvf6RjkJTmBUfygYeGXPq3ZapuDLVdV2fThFQFDqc9eQxSG9Ua54DCdQpFIQ==nIL4",
|
||||||
|
"pgpsignaturedate" : ISODate("2014-02-12T16:49:53.575Z"),
|
||||||
|
"syntaxversion" : 1
|
||||||
|
},
|
||||||
|
"agentname" : "server1234.example.net",
|
||||||
|
"agentqueueloc" : "linux.server1234.example.net.55tjipop5r2h1",
|
||||||
|
"status" : "succeeded",
|
||||||
|
"results" : [
|
||||||
|
{
|
||||||
|
"Elements" : {
|
||||||
|
"/home" : {
|
||||||
|
"filename" : {
|
||||||
|
"DB file" : {
|
||||||
|
"\.db\$" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Dump file" : {
|
||||||
|
"\.dump\$" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"SQL file" : {
|
||||||
|
"\.sql\$" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"gnupg secring" : {
|
||||||
|
"secring.gpg" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"key file" : {
|
||||||
|
"\.key\$" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"log file" : {
|
||||||
|
"\.log\$" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"password file" : {
|
||||||
|
"password" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"regex" : {
|
||||||
|
"cleartext RSA private key" : {
|
||||||
|
"-----BEGIN RSA PRIVATE KEY-----" : {
|
||||||
|
"Filecount" : 972,
|
||||||
|
"Files" : {
|
||||||
|
},
|
||||||
|
"Matchcount" : 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Extra" : {
|
||||||
|
"Statistics" : {
|
||||||
|
"Checkcount" : 8,
|
||||||
|
"Checksmatch" : 0,
|
||||||
|
"Exectime" : "83.966495ms",
|
||||||
|
"Filescount" : 972,
|
||||||
|
"Openfailed" : 0,
|
||||||
|
"Totalhits" : 0,
|
||||||
|
"Uniquefiles" : 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"FoundAnything" : false
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"starttime" : ISODate("2014-02-12T16:49:58.790Z"),
|
||||||
|
"finishtime" : ISODate("2014-02-12T16:49:59.279Z")
|
||||||
|
}
|
||||||
|
|
||||||
|
**Get a list of agents that have not sent a heartbeat recently**
|
||||||
|
|
||||||
|
The following query will search the **registrations** collection to list agents
|
||||||
|
that haven't checked in for the last two hours. It uses javascript's date
|
||||||
|
operation to substract 120 minutes to the current date, and query on that.
|
||||||
|
|
||||||
|
.. code:: javascript
|
||||||
|
|
||||||
|
> db.registrations.find({ heartbeatts: {$lt: new Date(ISODate().getTime() - 1000 * 60 * 120)}});
|
||||||
|
|
||||||
|
We can use javascript to print the agent name, queueloc and timestamp of last
|
||||||
|
heartbeat.
|
||||||
|
|
||||||
|
.. code:: javascript
|
||||||
|
|
||||||
|
> var lateagents = db.registrations.find({ heartbeatts: {$lt: new Date(ISODate().getTime() - 1000 * 60 * 120)}});
|
||||||
|
|
||||||
|
> lateagents.forEach(function(agent){ print(agent.name, agent.queueloc, agent.heartbeatts);});
|
||||||
|
agentXYZ.example.net linux.agentXYZ.example.net.55t93uhd7m69p Wed Feb 12 2014 15:49:12 GMT+0000 (UTC)
|
||||||
|
database123.example.com linux.database123.example.com.55tjdn0fsrdaf Wed Feb 12 2014 15:49:43 GMT+0000 (UTC)
|
||||||
|
firewall55.example.net linux.firewall55.example.net.55ub9eh81igbi Wed Feb 12 2014 15:48:29 GMT+0000 (UTC)
|
||||||
|
|
||||||
See MongoDB reference documentation for a full explanation of the query language.
|
See MongoDB reference documentation for a full explanation of the query language.
|
||||||
|
|
|
||||||
Загрузка…
Ссылка в новой задаче