Merge pull request #30 from mozilla/filemodule

rewrite of file module
This commit is contained in:
Julien Vehent 2014-11-30 05:19:20 -05:00
Родитель 045db2cad7 f52c14454e
Коммит f185a6889b
14 изменённых файлов: 1662 добавлений и 1008 удалений

Просмотреть файл

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.34.0 (20140110.0949)
<!-- Generated by graphviz version 2.34.0 (20141111.1304)
-->
<!-- Title: mig Pages: 1 -->
<svg width="1332pt" height="949pt"

До

Ширина:  |  Высота:  |  Размер: 26 KiB

После

Ширина:  |  Высота:  |  Размер: 26 KiB

Просмотреть файл

@ -2,4 +2,9 @@ doc:
for doc in $$(ls *.rst); do \
rst2html5 --stylesheet=docstyle.css "$$doc" > "$$doc.html"; \
done
for modname in $(ls ../src/mig/modules/); do \
if [ -r "../src/mig/modules/$modname/doc.rst" ]; then \
rst2html5 --stylesheet=docstyle.css "../src/mig/modules/$modname/doc.rst" > "module_$modname.html"; \
fi; \
done
dot -Tsvg -o .files/action_command_flow.svg .files/action_command_flow.dot

122
doc/module_file.html Normal file
Просмотреть файл

@ -0,0 +1,122 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link href="docstyle.css" rel="stylesheet" />
<title>Mozilla InvestiGator: File module</title>
<meta content="Julien Vehent &lt;jvehent@mozilla.com&gt;" name="author" />
</head>
<body>
<h1>Mozilla InvestiGator: File module</h1>
<aside class="topic contents" id="table-of-contents">
<h1>Table of Contents</h1>
<ul class="auto-toc">
<li>
<p><a href="#usage">1   Usage</a></p>
<ul class="auto-toc">
<li><a href="#search-paths">1.1   Search Paths</a></li>
<li><a href="#search-filters">1.2   Search Filters</a></li>
<li><a href="#search-options">1.3   Search Options</a></li>
</ul>
</li>
<li>
<p><a href="#search-algorithm">2   Search algorithm</a></p>
<ul class="auto-toc">
<li><a href="#search-activation-deactivation">2.1   Search activation &amp; deactivation</a></li>
</ul>
</li>
</ul>
</aside>
<p>The file module (FM) provides a basic tools to inspect a file system. It is inspired by "find" on Unix, and implements a subset of its functionalities with a focus on speed of execution.</p>
<section id="usage">
<h2>1   Usage</h2>
<p>FM implements searches that are defined by a search label. A search can have a number of search parameters and options, defined below. There is no maximum number of searches that can be performed by a single invocation of FM. However, heavy invocations are frowned upon, because the MIG Agent will most likely kill modules that run for more than 5 minutes (configurable).</p>
<p>In JSON format, searches are defined as a json object where each search has a label (key) and search parameters (value).</p>
<p>A search label is a string between 1 and 64 characters, composed of letter [a-zA-z], numbers [0-9], underscore [_] or dashes [-].</p>
<p>A search must have at least one search path.</p>
<pre><code class="json"><span class="p">{</span>
<span class="nt">"searches"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"somesearchlabel"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"paths"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"/etc/shadow"</span>
<span class="p">],</span>
<span class="nt">"contents"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"^root"</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="nt">"another_search"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"paths"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"/usr"</span>
<span class="p">],</span>
<span class="nt">"sizes"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"&lt;371m"</span>
<span class="p">],</span>
<span class="nt">"modes"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"^-r-xr-x--"</span>
<span class="p">]</span>
<span class="s2">"sha256"</span><span class="p">:</span> <span class="p">[</span>
<span class="s2">"fff415292dc59cc99d43e70fd69347d09b9bd7a581f4d77b6ec0fa902ebaaec8"</span>
<span class="p">],</span>
<span class="nt">"options"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"matchall"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nt">"maxdepth"</span><span class="p">:</span> <span class="mi">3</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre>
<section id="search-paths">
<h3>1.1   Search Paths</h3>
<p>A search can have an unlimited number of search paths. Each path is treated as a string. No path expansion or regular expression is permitted in a path string.</p>
<p>A path can indicate a directory or a file. In the case of a directory, FM will enter the directory structure recursively until its end is reached, or until <cite>maxdepth</cite> is exceeded.</p>
<p>For each path defined in a search, all search filters will be evaluated.</p>
</section>
<section id="search-filters">
<h3>1.2   Search Filters</h3>
<p>Search filters can be used to locate a file on its metadata (fileinfo) or content. Filters can be applied in two ways: either <cite>matchall</cite> is set and all filters must match on a given file to include it in the results, or <cite>matchall</cite> is not set and filters are treated individually.</p>
<p>Note: all regular expressions used in search filters use the regexp syntax provided by the Go language, which is very similar to Posix. A full description of the syntax is available at <a href="http://golang.org/pkg/regexp/syntax/">http://golang.org/pkg/regexp/syntax/</a>.</p>
<p>Metadata filters:</p>
<ul>
<li><strong>name</strong>: a regular expression that is applying on the base name of a file.</li>
<li><strong>size</strong>: a size filter indicates whether we want files that are larger or smaller than a given size. The syntax uses a prefix <cite>&lt;</cite> or <cite>&gt;</cite> to indicate smaller than and greater than. The file size is assumed to be in bytes, and multipliers can be provided as suffix: <cite>k</cite>, <cite>m</cite>, <cite>g</cite> and <cite>t</cite> for kilobytes, megabytes, gigabytes and terabytes. For example, the filter <cite>&lt;10m</cite> will match on files that have a size inferior than 10 megabytes. When <cite>matchall</cite> is set, several size filters can provide an efficient way to bound the search to a given file size window.</li>
<li><strong>mode</strong>: mode filters on both the type and permission of a file. The filter uses a regular expression that applies on the stringified filemode returned by Go. The mode string first contains the type of the file, followed by the permissions of the file. For example, a regular file with 640 permissions would return <cite>-rw-r-----</cite> and a regular expression on that string can be used to match the file. If the file has special attributes, such as setuid or sticky bits, those are prepended to the mode string: <cite>gtrwx--x--x</cite>. The meaning of each letter is defined in the Go documentation at <a href="http://golang.org/pkg/os/#FileMode">http://golang.org/pkg/os/#FileMode</a>.</li>
<li><strong>mtime</strong>: mtime filters on the modification time of a file. It takes a period parameter that checks if the file has been modified since a given perior, or before a given period. For example, the mtime filter <cite>&lt;90d</cite> will match of files that have been modified over the last nighty days, while the filter <cite>&gt;5h</cite> will match modified more than 5 hours ago. The mtime syntax takes a prefix <cite>&lt;</cite> or <cite>&gt;</cite>, a integer that represents the period, and a suffix <cite>d</cite>, <cite>h</cite> or <cite>m</cite> for days, hours and minutes.</li>
</ul>
<p>Content filters:</p>
<ul>
<li><strong>content</strong>: a regular expression that matches against the content of the file. Inspection stops at the first occurence of the regular expression that matches on the file.</li>
<li><strong>md5</strong>: a md5 checksum</li>
<li><strong>sha1</strong>: a sha1 checksum</li>
<li><strong>sha256</strong>: a sha256 checksum</li>
<li><strong>sha384</strong>: a sha384 checksum</li>
<li><strong>sha512</strong>: a sha512 checksum</li>
<li><strong>sha3_224</strong>: a sha3_224 checksum</li>
<li><strong>sha3_256</strong>: a sha3_256 checksum</li>
<li><strong>sha3_384</strong>: a sha3_384 checksum</li>
<li><strong>sha3_512</strong>: a sha3_512 checksum</li>
</ul>
</section>
<section id="search-options">
<h3>1.3   Search Options</h3>
<p>Several options can be applied to a search:</p>
<ul>
<li><strong>maxdepth</strong> controls the maximum number of directories that can be traversed by a search. For example, is a search has path <cite>/home</cite>, and <cite>maxdepth</cite> is set to the value 3, the deepest directory that can be visited is <cite>/home/dir1/dir2/dir3</cite>.</li>
<li><strong>matchall</strong> indicates that within a given search, all search filters must match on one file for it to be included in the results. Being a boolean, <cite>matchall</cite> is not set by default. The MIG command line sets it automatically, the console does not.</li>
<li><strong>matchlimit</strong> controls how many files can be returned by a single search. This safeguard prevents a single run of the file module from crashing before of the amount of results it is returning. The default value is 1,000, which is already significant. If you plan on returning more than 1,000 results in a single file search, you should probably consider breaking it down into smaller searches, or running the search locally instead of through MIG.</li>
</ul>
</section>
</section>
<section id="search-algorithm">
<h2>2   Search algorithm</h2>
<p>FM traverse a directory tree starting from a root path and until no search are longer active. FM traverses a given path only once, regardless of the number of searches that are being performed. When FM enters a directory, it activates searches that apply to the directory, and deactivates the ones that don't. As soon as no searches are active, FM either tries another root path, or exits.</p>
<p>Inside a given directory, FM evaluates all files one by one. The filters on fileinfo are first applied: name, size, mode and mtime. If the matchall option is set, and at least one of the fileinfo filter does not match, the file is discarded. If matchall is not set, or if all fileinfo filters match, the filters on file content are then evaluated: content regex and checksums.</p>
<p>The case of content regex is particular, because evaluation of the file stops at the first positive occurence of the regex in a file. This is meant to speed up searches on large files that may match a large number of times.</p>
<p>Once all searches are deactivated, FM builds a result object from the internal checks results. For each search, each file that matched is included once. If the search was set to <cite>matchall</cite>, the search parameters are not included in the results (we now that all of them must have matched). If <cite>matchall</cite> was not set, then each file returns the list of checks that matched it. It is thus possible to have, in one same search, a file match of a file size filter, and another one match on a sha256 checksum.</p>
<section id="search-activation-deactivation">
<h3>2.1   Search activation &amp; deactivation</h3>
<p>While processing the directory structure, FM compares the current path with the search paths of each search. A single search can have multiple paths, and if one of them matches the current path, the search is activated.</p>
<p>For example, if the current path is <cite>/var/lib/postgres</cite>, and a search has a path set to <cite>/var</cite>, the search will be activated for the current directory.</p>
<p>Unless the value of <cite>maxdepth</cite> indicates that the search should not go beyond a certain number of subdirectories, and that number is reached. In which case, the search is deactivated.</p>
</section>
</section>
</body>
</html>

Просмотреть файл

@ -17,112 +17,104 @@
"module": "file",
"parameters": {
"searches": {
"pkeyhomedir": {
"description": "private ssh key in homedir",
"paths": [
"/home/*/.ssh/*"
"s1": {
"md5": [
"6a7f0de60c4f43522882f2fb8a69209a"
],
"regexes": [
"-----BEGIN RSA PRIVATE KEY-----"
]
},
"localuser": {
"description": "test the presence of some users",
"paths": [
"/etc/passwd"
],
"regexes": [
"^ulfr",
"^kang"
]
},
"strongrootpasswd": {
"description": "verify that root uses a strong salted password",
"paths": [
"/etc/shadow"
],
"regexes": [
"^root:\\$(2(a|y)|5|6)\\$"
]
},
"rootmd5": {
"description": "flag if shadow uses md5 for root",
"paths": [
"/etc/shadow"
],
"regexes": [
"^root:\\$1\\$"
]
},
"etcnorecursive": {
"description": "look into /etc/ only, no recursive walk down the paths",
"paths": [
"/etc/"
],
"regexes": [
"^something interesting$"
"names": [
"fusermount"
],
"options": {
"maxdepth": 0
}
},
"etcrecursive": {
"description": "look into /etc/ and all subsequent directories",
"matchall": true,
"maxdepth": 2
},
"paths": [
"/etc/"
],
"regexes": [
"$another interesting thing$"
],
"options": {
"maxdepth": -1,
"crossfs": false
}
},
"sqlrootdir": {
"description": "Flag sql dumps in root home",
"paths": [
"/root/"
],
"filedescriptions": [
".sql"
"/usr"
]
},
"dangerousbin": {
"description": "detect known dangerous binaries",
"paths": [
"/bin/*",
"/sbin/*",
"/usr/bin/*",
"/usr/sbin/*",
"/opt/*"
"s2": {
"contents": [
"gpgcheck=1"
],
"sha256": [
"1e2699ff1f9238c58390c1ada53f4f21032ca5e0946bfb54a5a144452e6efc82",
"286c39ec3d8e4f15f353dca350ca7575e0269dba808206f3ce8d1a3ea142b353",
"39823089fa324ceba00d5939d2e7b308fec28ee0f16c6caa4739a53ad6ecee64",
"3efee976d6565edd1492aa1047ffa10be6025de18206f6c68f91dd218801778f",
"467f34eee9d133653467a60ab0fe938d7c26918465a2ac938d2ffc6f2525b1ff",
"4735f97b31ddb8a1bbc61e8d66b4dbc08d8092142d8ae7564f9058e0a20bbbb6",
"5cba4433237e2ff202a5b20aad00a12d25bfc5564c3620a9463767eec2150cc1",
"6114624bf5d7e29f738f939bcc2bc794de9bf377a571fe1e84ae9159794308cf",
"72071c89c07da8229be29da807c0340c870d431796c382c006bf08f63b2d9b46",
"72589dd25b491ed53670bc7d04f4874075fc7d16361fc295c31fc86118d84cbd",
"72a44f3e7c4d9c9b72b1bda77d687346447d8e398983965b8e690eeeadebdc76",
"7c9816b5f1b840eb8c5ecfc0fed29972877ca5bd909469d03f26d3b8f837043d",
"81dac9c6dc5e4ed615d496aea74fddc85925b00a6a54ddcbb90603c1469ce04c",
"89a400077d74d1d76103180f41f40de6bcfffc89de461f497eef2ea763a68d73",
"89b68f8ea6a32d525fbf491878980180ffa395b042ea3104b11da229bade71db",
"903c13171c7467271fd79244ad8281ded9f51e3cf27c3399b42a175c53806a99",
"939cc74b5343bde1a17dfa270f8e6dc719a4bc6b3143f4581b401c81fd9a110d",
"adbee847c12c73605ff657e668c8096df138f824eb542027a10c0b5c07619c8d",
"dbe7fc18667cd75317d494ed3b32cfe3cd077c870d015dc18b406a4a39747f55",
"fc48883e129225dc8fc9e340a495fbd834c97f5ff7fa70ab6089ec216a465328",
"fd702be65b1d3abed4c0197854c0c777a2bb50632932e1e389129b19b14a1e69"
"names": [
"yum.conf"
],
"options": {
"matchall": true,
"maxdepth": 1
},
"paths": [
"/etc/"
]
},
"s3": {
"contents": [
"RedHat"
],
"names": [
"Concertation",
"cariboumaurice12345"
],
"options": {
"maxdepth": 1
},
"paths": [
"/home/ulfr"
]
},
"s4": {
"paths": [
"/nowhere"
],
"names": [
"nofilenamenotfound"
],
"options": {
"matchall": true
}
},
"s5": {
"names": [
"known_hosts"
],
"options": {
"maxdepth": 2
},
"paths": [
"/home/ulfr"
]
},
"s6": {
"mtimes": [
"<90d"
],
"options": {
"matchall": true,
"maxdepth": 2
},
"paths": [
"/home/ulfr"
],
"sizes": [
">1g"
]
},
"s7": {
"modes": [
"gt"
],
"names": [
"somefile"
],
"options": {
"matchall": true,
"maxdepth": 1
},
"paths": [
"/home/ulfr"
]
}
},
"condition": "pkeyhomedir or !localuser or !strongrootpasswd or rootmd5 or dangerousbin"
}
}
},
{

Просмотреть файл

@ -1,5 +1,6 @@
#!/usr/bin/env python
import os
import sys
import gnupg
from time import gmtime, strftime
import random
@ -25,10 +26,10 @@ def makeToken(gpghome, keyid):
return token
if __name__ == '__main__':
token = makeToken("/home/ulfr/.gnupg",
"E60892BB9BD89A69F759A1A0A3D652173B763E8F")
r = requests.get("http://localhost:12345/api/v1/dashboard",
headers={'X-PGPAUTHORIZATION': token})
token = makeToken("/home/ulfr/.gnupg", "E60892BB9BD89A69F759A1A0A3D652173B763E8F")
r = requests.get(sys.argv[1],
headers={'X-PGPAUTHORIZATION': token},
verify=False)
print token
print r.text

Просмотреть файл

@ -118,7 +118,7 @@ func main() {
case "agent":
err := configLoad(*config)
if err != nil {
fmt.Fprintf(os.Stderr, "[info] Using builtin conf. %v\n", err)
fmt.Fprintf(os.Stderr, "[info] Just FYI (not an error): using the builtin conf because %v\n", err)
} else {
fmt.Fprintf(os.Stderr, "[info] Using external conf from %s\n", *config)
}

Просмотреть файл

@ -9,7 +9,7 @@ import (
"encoding/json"
"fmt"
"mig"
"mig/modules/filechecker"
"mig/modules/file"
"time"
)
@ -62,29 +62,114 @@ func commandsToComplianceItems(commands []mig.Command) (items []ComplianceItem,
continue
}
switch cmd.Action.Operations[i].Module {
case "filechecker":
var r filechecker.Results
case "file":
var r file.Results
err = json.Unmarshal(buf, &r)
if err != nil {
return items, err
}
for path, _ := range r.Elements {
bitem.Check.Location = path
for method, _ := range r.Elements[path] {
bitem.Check.Test.Type = method
for id, _ := range r.Elements[path][method] {
bitem.Check.Name = id
for value, _ := range r.Elements[path][method][id] {
bitem.Check.Test.Value = value
if r.Elements[path][method][id][value].Matchcount > 0 {
bitem.Compliance = true
} else {
bitem.Compliance = false
}
item := bitem
items = append(items, item)
for label, sr := range r.Elements {
for _, mf := range sr {
bitem.Check.Location = mf.File
bitem.Check.Name = label
bitem.Check.Test.Type = "file"
bitem.Check.Test.Value = ""
for _, v := range mf.Search.Names {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("name='%s'", v)
}
for _, v := range mf.Search.Sizes {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("size='%s'", v)
}
for _, v := range mf.Search.Modes {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("mode='%s'", v)
}
for _, v := range mf.Search.Mtimes {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("mtime='%s'", v)
}
for _, v := range mf.Search.Contents {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("content='%s'", v)
}
for _, v := range mf.Search.MD5 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("md5='%s'", v)
}
for _, v := range mf.Search.SHA1 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha1='%s'", v)
}
for _, v := range mf.Search.SHA256 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha256='%s'", v)
}
for _, v := range mf.Search.SHA384 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha384='%s'", v)
}
for _, v := range mf.Search.SHA512 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha512='%s'", v)
}
for _, v := range mf.Search.SHA3_224 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha3_224='%s'", v)
}
for _, v := range mf.Search.SHA3_256 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha3_256='%s'", v)
}
for _, v := range mf.Search.SHA3_384 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha3_384='%s'", v)
}
for _, v := range mf.Search.SHA3_512 {
if len(bitem.Check.Test.Value) > 0 {
bitem.Check.Test.Value += " and "
}
bitem.Check.Test.Value += fmt.Sprintf("sha3_512='%s'", v)
}
if mf.File == "" {
for i, p := range mf.Search.Paths {
if i > 0 {
bitem.Check.Location += ", "
}
bitem.Check.Location += p
}
bitem.Compliance = false
} else {
bitem.Compliance = true
}
items = append(items, bitem)
}
}
}

Просмотреть файл

@ -112,7 +112,7 @@ func main() {
os.Exit(2)
}
op.Parameters, err = modRunner.(mig.HasParamsParser).ParamsParser(modargs)
if err != nil {
if err != nil || op.Parameters == nil {
panic(err)
}
a.Operations = append(a.Operations, op)

Просмотреть файл

@ -26,7 +26,7 @@ func main() {
var err error
defer func() {
if e := recover(); e != nil {
fmt.Printf("FATAL: %v\n", e)
fmt.Fprintf(os.Stderr, "FATAL: %v\n", e)
}
}()
homedir := client.FindHomedir()

Просмотреть файл

@ -0,0 +1,203 @@
=================================
Mozilla InvestiGator: File module
=================================
:Author: Julien Vehent <jvehent@mozilla.com>
.. sectnum::
.. contents:: Table of Contents
The file module (FM) provides a basic tools to inspect a file system. It is
inspired by "find" on Unix, and implements a subset of its functionalities
with a focus on speed of execution.
Usage
-----
FM implements searches that are defined by a search label. A search can have a
number of search parameters and options, defined below. There is no maximum
number of searches that can be performed by a single invocation of FM. However,
heavy invocations are frowned upon, because the MIG Agent will most likely kill
modules that run for more than 5 minutes (configurable).
In JSON format, searches are defined as a json object where each search has a
label (key) and search parameters (value).
A search label is a string between 1 and 64 characters, composed of letter
[a-zA-z], numbers [0-9], underscore [_] or dashes [-].
A search must have at least one search path.
.. code:: json
{
"searches": {
"somesearchlabel": {
"paths": [
"/etc/shadow"
],
"contents": [
"^root"
]
},
"another_search": {
"paths": [
"/usr"
],
"sizes": [
"<371m"
],
"modes": [
"^-r-xr-x--"
]
"sha256": [
"fff415292dc59cc99d43e70fd69347d09b9bd7a581f4d77b6ec0fa902ebaaec8"
],
"options": {
"matchall": true,
"maxdepth": 3
}
}
}
}
Search Paths
~~~~~~~~~~~~
A search can have an unlimited number of search paths. Each path is treated as
a string. No path expansion or regular expression is permitted in a path string.
A path can indicate a directory or a file. In the case of a directory, FM will
enter the directory structure recursively until its end is reached, or until
`maxdepth` is exceeded.
For each path defined in a search, all search filters will be evaluated.
Search Filters
~~~~~~~~~~~~~~
Search filters can be used to locate a file on its metadata (fileinfo) or
content. Filters can be applied in two ways: either `matchall` is set and all
filters must match on a given file to include it in the results, or `matchall`
is not set and filters are treated individually.
Note: all regular expressions used in search filters use the regexp syntax
provided by the Go language, which is very similar to Posix. A full description
of the syntax is available at http://golang.org/pkg/regexp/syntax/.
Metadata filters:
* **name**: a regular expression that is applying on the base name of a file.
* **size**: a size filter indicates whether we want files that are larger or
smaller than a given size. The syntax uses a prefix `<` or `>` to indicate
smaller than and greater than. The file size is assumed to be in bytes, and
multipliers can be provided as suffix: `k`, `m`, `g` and `t` for kilobytes,
megabytes, gigabytes and terabytes. For example, the filter `<10m` will match
on files that have a size inferior than 10 megabytes. When `matchall` is set,
several size filters can provide an efficient way to bound the search to a
given file size window.
* **mode**: mode filters on both the type and permission of a file. The filter
uses a regular expression that applies on the stringified filemode returned by
Go. The mode string first contains the type of the file, followed by the
permissions of the file.
For example, a regular file with 640 permissions would return `-rw-r-----`
and a regular expression on that string can be used to match the file.
If the file has special attributes, such as setuid or sticky bits, those are
prepended to the mode string: `gtrwx--x--x`. The meaning of each letter is
defined in the Go documentation at http://golang.org/pkg/os/#FileMode.
* **mtime**: mtime filters on the modification time of a file. It takes a
period parameter that checks if the file has been modified since a given
perior, or before a given period. For example, the mtime filter `<90d` will
match of files that have been modified over the last nighty days, while the
filter `>5h` will match modified more than 5 hours ago.
The mtime syntax takes a prefix `<` or `>`, a integer that represents the
period, and a suffix `d`, `h` or `m` for days, hours and minutes.
Content filters:
* **content**: a regular expression that matches against the content of the
file. Inspection stops at the first occurence of the regular expression that
matches on the file.
* **md5**: a md5 checksum
* **sha1**: a sha1 checksum
* **sha256**: a sha256 checksum
* **sha384**: a sha384 checksum
* **sha512**: a sha512 checksum
* **sha3_224**: a sha3_224 checksum
* **sha3_256**: a sha3_256 checksum
* **sha3_384**: a sha3_384 checksum
* **sha3_512**: a sha3_512 checksum
Search Options
~~~~~~~~~~~~~~
Several options can be applied to a search:
* **maxdepth** controls the maximum number of directories that can be traversed
by a search. For example, is a search has path `/home`, and `maxdepth` is set
to the value 3, the deepest directory that can be visited is
`/home/dir1/dir2/dir3`.
* **matchall** indicates that within a given search, all search filters must
match on one file for it to be included in the results. Being a boolean,
`matchall` is not set by default. The MIG command line sets it automatically,
the console does not.
* **matchlimit** controls how many files can be returned by a single search.
This safeguard prevents a single run of the file module from crashing before
of the amount of results it is returning. The default value is 1,000, which is
already significant. If you plan on returning more than 1,000 results in a
single file search, you should probably consider breaking it down into smaller
searches, or running the search locally instead of through MIG.
Search algorithm
----------------
FM traverse a directory tree starting from a root path and until no search are
longer active. FM traverses a given path only once, regardless of the number of
searches that are being performed. When FM enters a directory, it activates
searches that apply to the directory, and deactivates the ones that don't.
As soon as no searches are active, FM either tries another root path, or exits.
Inside a given directory, FM evaluates all files one by one. The filters on
fileinfo are first applied: name, size, mode and mtime. If the matchall option
is set, and at least one of the fileinfo filter does not match, the file is
discarded. If matchall is not set, or if all fileinfo filters match, the
filters on file content are then evaluated: content regex and checksums.
The case of content regex is particular, because evaluation of the file stops
at the first positive occurence of the regex in a file. This is meant to speed
up searches on large files that may match a large number of times.
Once all searches are deactivated, FM builds a result object from the internal
checks results. For each search, each file that matched is included once. If
the search was set to `matchall`, the search parameters are not included in the
results (we now that all of them must have matched). If `matchall` was not set,
then each file returns the list of checks that matched it. It is thus possible
to have, in one same search, a file match of a file size filter, and another
one match on a sha256 checksum.
Search activation & deactivation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While processing the directory structure, FM compares the current path with the
search paths of each search. A single search can have multiple paths, and if
one of them matches the current path, the search is activated.
For example, if the current path is `/var/lib/postgres`, and a search has a
path set to `/var`, the search will be activated for the current directory.
Unless the value of `maxdepth` indicates that the search should not go beyond a
certain number of subdirectories, and that number is reached. In which case,
the search is deactivated.

Разница между файлами не показана из-за своего большого размера Загрузить разницу

Просмотреть файл

@ -14,28 +14,64 @@ import (
"strings"
)
const help string = `To add search parameters, use the following syntax:
path <path> add a path to search.
example: > /etc/yum.*/*.repo
func printHelp(isCmd bool) {
dash := ""
if isCmd {
dash = "-"
}
fmt.Printf(`Search parameters
-----------------
%spath <string> - search path
ex: path /etc
%sname <regex> - regex to match against the name of a file
ex: name \.sql$
%ssize <size> - match files with a size smaller or greater that <size>
prefix with '<' for lower than, and '>' for greater than
suffix with k, m, g or t for kilo, mega, giga and terabytes
ex: size <10m (match files larger than 10 megabytes)
%smode <regex> - filter on the filemode, provided as a regex on the mode string
ex: mode -r(w|-)xr-x---
%smtime <period> - match files modified before or since <period>
prefix with '<' for modified since, and '>' for modified before
suffix with d, h, m for days, hours and minutes
ex: mtime <90d (match files modified since last 90 days)
%scontent <regex> - regex to match against the content of a file
ex: content ^root:\$1\$10CXRS19\$/h
%smd5 <hash> .
%ssha1 <hash> .
%ssha256 <hash> .
%ssha384 <hash> .
%ssha512 <hash> .
%ssha3_224 <hash> .
%ssha3_256 <hash> .
%ssha3_384 <hash> .
%ssha3_512 <hash> - compare file against given hash
content <regex> add a regex to check against files content
example: > ^root:\$1\$10CXRS19\$/h
name <regex> add a regex to search against filenames
example: > \.sql$
Options
-------
%smaxdepth <int> - limit search to that many subdirectories
ex: maxdepth 3
%smatchall - all search parameters must match on a given file for it to
return as a match. off by default. deactivates 'matchany' if set.
ex: matchall
%smatchany - any search parameter must match on a given file for it to
return as a match. on by default. deactivates 'matchall' if set.
ex: matchany
%smatchlimit <int> - limit the number of files that can be matched by a search.
the default limit is set to 1000. search will stop once the limit
is reached.
<hashType> <hash> add an hash to compare files against
Available hash types:
md5, sha1, sha256, sha384, sha512,
sha3_224, sha3_256, sha3_384, sha3_512
example: > md5 a12496cb3fd77a535df2d6ddc2e4ef53
When done, enter 'done'`
detailled doc at http://mig.mozilla.org/doc/module_file.html
`, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash, dash)
return
}
// ParamsCreator implements an interactive parameters creation interface, which
// receives user input, stores it into a Parameters structure, validates it,
// and returns that structure as an interface. It is mainly used by the MIG Console
func (r Runner) ParamsCreator() (interface{}, error) {
fmt.Println("initializing filechecker parameters creation")
var err error
p := newParameters()
scanner := bufio.NewScanner(os.Stdin)
@ -43,7 +79,7 @@ func (r Runner) ParamsCreator() (interface{}, error) {
var label string
var search search
for {
fmt.Println("create a new search by entering a search label, or 'done' to exit")
fmt.Println("Give a name to this search, or 'done' to exit")
fmt.Printf("label> ")
scanner.Scan()
if err := scanner.Err(); err != nil {
@ -55,6 +91,10 @@ func (r Runner) ParamsCreator() (interface{}, error) {
// no label to add, exit
goto exit
}
if label == "help" {
fmt.Println(`A search must first have a name before search parameters can be defined.`)
continue
}
err = validateLabel(label)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
@ -77,7 +117,8 @@ func (r Runner) ParamsCreator() (interface{}, error) {
}
break
}
fmt.Printf("creating new search with label: %s\n%s\n", label, help)
fmt.Printf("Creating new search with label '%s'\n"+
"Enter 'help' to list available parameters.\n", label)
for {
fmt.Printf("search '%s'> ", label)
@ -91,12 +132,12 @@ func (r Runner) ParamsCreator() (interface{}, error) {
break
}
if input == "help" {
fmt.Printf("%s\n", help)
printHelp(false)
continue
}
arr := strings.SplitN(input, " ", 2)
if len(arr) != 2 {
fmt.Printf("Invalid input format!\n%s\n", help)
fmt.Printf("Invalid input format!\n")
continue
}
checkType := arr[0]
@ -104,13 +145,6 @@ func (r Runner) ParamsCreator() (interface{}, error) {
switch checkType {
case "path":
search.Paths = append(search.Paths, checkValue)
case "content":
err = validateRegex(checkValue)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
continue
}
search.Contents = append(search.Contents, checkValue)
case "name":
err = validateRegex(checkValue)
if err != nil {
@ -118,6 +152,34 @@ func (r Runner) ParamsCreator() (interface{}, error) {
continue
}
search.Names = append(search.Names, checkValue)
case "size":
err = validateSize(checkValue)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
continue
}
search.Sizes = append(search.Sizes, checkValue)
case "mode":
err = validateRegex(checkValue)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
continue
}
search.Modes = append(search.Modes, checkValue)
case "mtime":
err = validateMtime(checkValue)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
continue
}
search.Mtimes = append(search.Mtimes, checkValue)
case "content":
err = validateRegex(checkValue)
if err != nil {
fmt.Printf("ERROR: %v\nTry again.\n", err)
continue
}
search.Contents = append(search.Contents, checkValue)
case "md5":
err = validateHash(checkValue, checkMD5)
if err != nil {
@ -182,10 +244,10 @@ func (r Runner) ParamsCreator() (interface{}, error) {
}
search.SHA3_512 = append(search.SHA3_512, checkValue)
default:
fmt.Printf("Invalid method!\n%s\n", help)
fmt.Printf("Invalid method!\n")
continue
}
fmt.Printf("Stored %s %s\n", checkType, checkValue)
fmt.Printf("Stored %s %s\nEnter a new parameter, or 'done' to exit.\n", checkType, checkValue)
}
p.Searches[label] = search
fmt.Println("Stored search", label)
@ -194,37 +256,28 @@ exit:
return p, nil
}
const cmd_help string = `~~~ FILE module ~~~
-path <string> inspects the given path recursively.
At least one path must be given on invocation.
-name <regex> looks for files that have a name that match this regex
-content <regex> look for files that have a content that match this regex
-md5, -sha1, -sha256,
-sha384, -sha512,
-sha3_224, -sha3_256,
-sha3_384, -sha3_512 <hash> look for files that match a given checksum
`
// ParamsParser implements a command line parameters parser that takes a string
// and returns a Parameters structure in an interface. It will display the module
// help if the arguments string spell the work 'help'
func (r Runner) ParamsParser(args []string) (interface{}, error) {
var (
err error
paths, names, contents, md5s, sha1s, sha256s, sha384s,
sha512s, sha3_224s, sha3_256s, sha3_384s, sha3_512s flagParam
fs flag.FlagSet
paths, names, sizes, modes, mtimes, contents, md5s, sha1s, sha256s,
sha384s, sha512s, sha3_224s, sha3_256s, sha3_384s, sha3_512s flagParam
maxdepth, matchlimit float64
matchall, matchany bool
fs flag.FlagSet
)
if len(args) < 1 || args[0] == "" || args[0] == "help" {
fmt.Println(cmd_help)
return nil, fmt.Errorf("help printed")
printHelp(true)
return nil, nil
}
fs.Init("file", flag.ContinueOnError)
fs.Var(&paths, "path", "see help")
fs.Var(&names, "name", "see help")
fs.Var(&sizes, "size", "see help")
fs.Var(&modes, "mode", "see help")
fs.Var(&mtimes, "mtime", "see help")
fs.Var(&contents, "content", "see help")
fs.Var(&md5s, "md5", "see help")
fs.Var(&sha1s, "sha1", "see help")
@ -235,6 +288,10 @@ func (r Runner) ParamsParser(args []string) (interface{}, error) {
fs.Var(&sha3_256s, "sha3_256", "see help")
fs.Var(&sha3_384s, "sha3_384", "see help")
fs.Var(&sha3_512s, "sha3_512", "see help")
fs.Float64Var(&maxdepth, "maxdepth", 0, "see help")
fs.Float64Var(&matchlimit, "matchlimit", 0, "see help")
fs.BoolVar(&matchall, "matchall", true, "see help")
fs.BoolVar(&matchany, "matchany", false, "see help")
err = fs.Parse(args)
if err != nil {
return nil, err
@ -242,6 +299,9 @@ func (r Runner) ParamsParser(args []string) (interface{}, error) {
var s search
s.Paths = paths
s.Names = names
s.Sizes = sizes
s.Modes = modes
s.Mtimes = mtimes
s.Contents = contents
s.MD5 = md5s
s.SHA1 = sha1s
@ -252,8 +312,19 @@ func (r Runner) ParamsParser(args []string) (interface{}, error) {
s.SHA3_256 = sha3_256s
s.SHA3_384 = sha3_384s
s.SHA3_512 = sha3_512s
s.Options.MaxDepth = maxdepth
s.Options.MatchLimit = matchlimit
s.Options.MatchAll = matchall
if matchany {
s.Options.MatchAll = false
}
p := newParameters()
p.Searches["s1"] = s
r.Parameters = *p
err = r.ValidateParameters()
if err != nil {
return nil, err
}
return p, nil
}

Просмотреть файл

@ -38,8 +38,7 @@ connectedip <ip|cidr> search for connections with the given IP address or within
listeningport <port> search for an open socket on the local system listening on <port>, tcp and udp
example: > listeningport 443
When done, enter 'done'`
`
// ParamsCreator implements an interactive parameters creation interface, which
// receives user input, stores it into a Parameters structure, validates it,
@ -80,7 +79,7 @@ func (r Runner) ParamsCreator() (interface{}, error) {
continue
}
p.LocalMAC = append(p.LocalMAC, checkValue)
fmt.Printf("Stored %s '%s'. Enter another search or 'done'.\n", checkType, checkValue)
fmt.Printf("Stored %s '%s'. Enter another parameter, or 'done' to exit.\n", checkType, checkValue)
case "neighbormac":
err = validateMAC(checkValue)
if err != nil {
@ -129,7 +128,7 @@ func (r Runner) ParamsCreator() (interface{}, error) {
return p, nil
}
const cmd_help string = `~~~ NETSTAT module ~~~
const cmd_help string = `
-lm <regex> search for local mac addresses that match <regex>
example: -lm ^8c:70:[0-9a-f]

Просмотреть файл

@ -25,11 +25,11 @@ func ArmoredPubKeysToKeyring(pubkeys []string) (keyring io.ReadSeeker, keycount
}()
var buf bytes.Buffer
// iterate over the keys, and load them into an io.Reader keyring
for _, key := range pubkeys {
for i, key := range pubkeys {
// Load PGP public key
el, err := openpgp.ReadArmoredKeyRing(bytes.NewBufferString(key))
if err != nil {
panic(err)
panic(fmt.Errorf("key num.%d failed to load with error %v", i, err))
}
keycount += 1
if len(el) != 1 {