The matchlimit option to the file module was not working correctly due
to a couple reasons.
First, it relied on the Totalhits value in the module statistics to
compare the number of hits to the match limit. This value was compiled
in buildResults, so was 0 throughout module execution. Because of this
matchlimit would never be exceeded.
Also, the comparison to Totalhits was only occuring on directory entry.
This means if the match limit was hit while scanning a single directory,
it would continue to scan files exceeding the match limit.
This modifies the way results are processed by Search types, so
individual checks submit matched files via a channel to the parent
Search entry. The Search entry can then maintain a list of matches, and
increment Totalhits as required while the processing is occuring instead
of in buildResults.
Closes#382
Reworks test tables so all tests execute from a single table, and test
file system creation is separated from the individual tests themselves.
This should permit more complicated test cases. This also removes
various Linux specific path elements so the tests should work on all
platforms.
If a file symlink existed in the root of a search path (e.g., /tmp was
being searched and /tmp/file was a symlink) path separators were not
being added correctly, resulting in no separator between the directory
path and the file name.
This caused subsequent checks (e.g., trying to follow and open the
symlink for content inspection) to fail, resulting in errors being
included in the module error log.
fswatch is a simple file system monitoring module that takes advantage
of the persistent module framework in MIG to do continuous file
integrity scans of specific paths on the file system. When changes are
detected, alerts are generated and appear in the agent's log file.
Adds the ability for persistent modules to read a configuration file
that can be used to control their behavior.
In some cases, we may want a persistent module to be configurable. A new
configuration variable PERSISTCONFIGDIR has been added which indicates a
directory the agent will look in for persistent module configs. When the
agent starts a persistent module, it will determine if modulename.cfg
exists in the config directory; if so the agent will load the config and
send it to the newly executed module, which it will read on stdin. The
module can then read this configuration, and configure itself as needed
before it begins it's primary execution functions.
Modules specify the configuration format within the module code itself,
so each module that makes use of this can have it's own configuration
parameters.
When a message between a module and the agent is read by either,
ReadInput is used which uses buffered IO to read a single line. A bug
existed here where a new bufio Reader was being allocated each time.
Under normal circumstances there would be no issue, since a single line
is generally sent to the module from the agent and read. However, in a
case where a large number of messages were being exchanged between the
agent and the module (e.g., persistent modules) it was possible the
bufio reader could have buffered more than one line. Since a new buffer
was being allocated each time, this resulted in loss of partial message
data, and usually ended up generating JSON unmarshaling errors since
part of the JSON structure was missing.
This was resolved by introducing a new ModuleReader and ModuleWriter
type that is used to read/write between the agent and modules.
ModuleReader allocates the buffered reader on creation once, so we only
ever have the single buffered reader for the lifetime of the module
execution.
We will want to use different socket types for persistent module
communication depending on the platform, e.g., unix sockets are not
available on Windows. This adds support for more dynamic registration of
the socket for a given running module. When the persistent module is
initialized, we allocate the listener and then notify the agent where we
area listening, so queries can dial the address as needed.
Persistent modules are modules that are executed continuously by the
agent. They can perform background tasks, and can be queried using
mig-cmd or mig-console in the same way as regular modules.
The modules are executed by the agent on startup, and a management
channel is used to validate the module is running and collect logs from
the module.
Queries into the persistent module are handled initially through the
same code path as a normal module query; but in the case of a persistent
module an adapter is used to simulate the module IO and collect results
using a domain socket.
EPM (extra privacy mode) is an agent option that if enabled, will
inform modules they should apply additional masking on returned results.
For example, by default if the file module matches a search, the name of
the matching file will be returned. With this enabled, it may inform the
investigator that something on the system matched, but will not provide
meta-data.
net.ParseIP will return IPv4 addresses as a 16 byte value (result comes
from net.IPv4()). Rather than use the length of the return value, try to
convert it to an IPv4 address to test the address family and set the
mask accordingly.
Closes#187
Connections to for example tcp6 sockets are stored in IPv6 related /proc
net files (e.g., /proc/net/tcp6). Also scan these files when searching
for an IPv4 address.
Aaron Meihm
Rob Murtha
Julien Vehent
Yves AGOSTINI