mozilla
/
MozDef
зеркало из https://github.com/mozilla/MozDef.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
5 424 коммитов 4 Ветки 53 Теги 68 MiB
Граф коммитов

509 Коммитов

Автор SHA1 Сообщение Дата
Brandon Myers 7e7c10fdbb
Rename common file to lib/aws 2019-01-22 12:37:46 -06:00
Brandon Myers 7576a55ed7
Merge pull request #990 from ryandeivert/ryandeivert-dry-get-creds 
deduplicating get_aws_credentials function
 2019-01-22 12:35:23 -06:00
Michal Purzynski 529dfa45e4 Changed the data model, added heuristics to figure the destination in case of denies 2019-01-22 10:21:46 -08:00
Michal Purzynski 40d6c12ca3 A new plugin - parse Squid access log messages, coming from syslog-ng via AMQP. Replaces the squid2mozdef script 2019-01-18 16:51:44 -08:00
Brandon Myers 0f014f152f
Fixup filterlog mq plugin 2019-01-14 12:12:43 -06:00
Brandon Myers d8d88a5d35
Merge pull request #1020 from mozilla/lower_keys_fixes 
lowercasing tags for fxa
 2018-12-27 13:22:26 -05:00
Michal Purzynski 319532aed7 Remove the netaddr import 2018-12-26 14:50:32 -08:00
Michal Purzynski d93b2cbb29 Work around the lower_case plugin changes 2018-12-26 14:43:29 -08:00
Phrozyn 15b174743c
lowercasing tags for fxa, this fixes nothing. 2018-12-26 16:03:55 -06:00
Phrozyn 2963b703c9
moving this to run after lower_keys.py 2018-12-19 14:52:15 -06:00
Phrozyn 5da575f246
Correcting registration for fxa events, and removing replacement code. 2018-12-19 14:49:42 -06:00
Phrozyn 6e4d12c717
Resolving areas where keys are manipulated after lower_keys is run. 2018-12-19 11:27:00 -06:00
A Smith 9abad28a43
Merge pull request #1004 from mozilla/key_update_for_pulseguardian 
updating key fields for pulseguardian events to move source_ip to sou…
 2018-12-18 17:41:47 -06:00
A Smith 7215580095
Merge pull request #964 from mozilla/lower_keys 
Lower keys
 2018-12-18 17:41:27 -06:00
Brandon Myers 97409a248c
Merge pull request #995 from mozilla/add_port_details_root 
Move source port and destination port to details root
 2018-12-18 12:48:56 -06:00
Phrozyn 365c565023
updating key fields for pulseguardian events to move source_ip to sourceipaddress. 2018-12-17 10:58:39 -06:00
Brandon Myers 46be867d2f
Fixup unused variables check 2018-12-14 14:06:21 -06:00
Brandon Myers df84a1942d
Fixup block comments not having a space after hash 2018-12-14 13:40:07 -06:00
Brandon Myers be7788089d
Fixup missing whitespace around arithmetic operator 2018-12-14 12:49:25 -06:00
Brandon Myers 09989706a0
Fixup closing bracket indentation not matching original 2018-12-14 12:39:23 -06:00
Brandon Myers d04485c850
Fixup pep8 undefined library 2018-12-14 12:27:57 -06:00
Brandon Myers fc771bd531
Remove unused import statements 2018-12-14 11:34:42 -06:00
Brandon Myers e77b791c8a
Merge pull request #934 from mpurzynski/githubevent_pr 
A MozDef plugin that parses GitHub's Webhook events to create meaning…
 2018-12-13 15:52:41 -05:00
Michal Purzynski 9693dfa58e Address nits from the review - use mozdef_util instead of changing the path, remove unnecessary config file 2018-12-12 12:47:12 -08:00
Brandon Myers 4e28602162
Move source port and destination port to details root 2018-12-10 01:55:54 -05:00
Jeff Bryner 410eb27e1b explicitly accept/map 'source' field 2018-12-03 15:38:24 -05:00
Michal Purzynski 43f1fa2f53 Dynamically resolve path to the config file 2018-11-29 18:06:36 -08:00
Ryan Deivert 42032a99a7 deduplicating get_aws_credentials function 2018-11-29 15:37:45 -08:00
Michal Purzynski ebfacbe147 Move the mapping configuration to a plugin directory 2018-11-29 13:53:43 -08:00
Michal Purzynski 2548178183 Merge remote-tracking branch 'upstream/master' into githubevent_pr 2018-11-29 13:44:16 -08:00
A Smith 03dabc7524
Merge branch 'master' into lower_keys 2018-11-29 10:44:50 -06:00
Phrozyn 307d65165d
lowering keys that the lower_keys plugin will affect, and removing unused details.Random field. 2018-11-26 18:38:51 -06:00
Jeff Bryner 839d545dd6 pull ip from an occasionally present list 2018-11-23 09:26:45 -08:00
andrewkrug 440d50478d
fix flake 8 error 2018-11-21 07:43:37 -08:00
andrewkrug 5845d59dbb
ensure mozdef always polls the SQS queue we create 2018-11-21 06:55:46 -08:00
andrewkrug a14f51fd0e
standardize es_worker credential handling 2018-11-21 06:13:48 -08:00
Michal Purzynski fd5ffafbca Move the configuration file where it can be found 2018-11-20 15:37:22 -08:00
Brandon Myers 21aacc57a0
Add Principal key to cloudtrail plugin 2018-11-14 13:51:55 -06:00
Brandon Myers 006b708693
Sort cloudtrail keys in mq plugin 2018-11-14 13:51:17 -06:00
Phrozyn f9af2dc8f0
Updated code that works on subkeys. 2018-11-14 09:57:47 -06:00
Phrozyn 33e21788bf
initial commit 2018-11-13 16:10:09 -06:00
Brandon Myers 4d07a1e470
Merge pull request #933 from mpurzynski/large_strings_github 
Truncate, if present, the GitHub Webhook's pr_body field
 2018-11-05 15:35:47 -06:00
Michal Purzynski 90b746e5c6 remove newline at the end of the file 2018-11-05 12:11:58 -08:00
Brandon Myers acc00029fe
Merge pull request #932 from mpurzynski/fixup_fxafixup 
Make sure the key eventsource exists before referencing it
 2018-11-05 14:09:33 -06:00
Michal Purzynski 3b751ee9b6 the pep check sometimes wants the empty line at the end of the file and sometimes it does not. go figure. 2018-11-05 12:01:15 -08:00
Michal Purzynski 4ca98e512a python hates me 2018-11-05 11:53:54 -08:00
Michal Purzynski 260b0ec957 python hates newlines 2018-11-05 11:49:39 -08:00
Michal Purzynski 8ac8ff1e29 Make sure the key eventsource exists before referencing it 2018-11-05 11:09:01 -08:00
Michal Purzynski a39f3c2010 Truncate, if present, the GitHub Webhook's pr_body field 2018-11-05 11:05:22 -08:00
Michal Purzynski d61168a3fc A MozDef plugin that parses GitHub's Webhook events to create meaningful IR data 2018-11-05 11:03:40 -08:00
Brandon Myers 3b07f12cc9
Resolve E128 continuation line under indented 2018-10-31 18:11:08 -05:00
Brandon Myers 3fbeae4611
Resolve E127 continuation line over indented 2018-10-31 17:30:18 -05:00
Brandon Myers db5c6c92cc
Resolve E126 continuation of over-indented lines 2018-10-31 17:17:49 -05:00
Brandon Myers 800f595023
Merge pull request #905 from mozilla/fix_geo_db_location 
Fix geolite db location
 2018-10-31 14:25:41 -05:00
Brandon Myers 4c80290a2b
Resolve E302 expected 2 blank lines found 1 2018-10-30 18:08:59 -05:00
Brandon Myers c856c29160
Resolve E713 test for membership 2018-10-30 18:01:19 -05:00
Brandon Myers 82be09f217
Resolve E114 indentation not a multiple of four 2018-10-30 17:51:54 -05:00
Brandon Myers 34ab0000ec
Resolve E116 unexpected indentation in comments 2018-10-30 17:30:02 -05:00
Brandon Myers 434788e9a8
Fix geodb missing path 2018-10-30 15:08:41 -05:00
Brandon Myers bf68a1e9ef
Merge pull request #891 from mozilla/fixup_sqs_connection_mq 
Fixup sqs connection with credentials
 2018-10-26 18:40:07 -05:00
Brandon Myers 420efe9f4c
Merge pull request #889 from mozilla/fix_missing_imports_papertrail 
Fix imports missing in papertrail worker
 2018-10-26 18:37:54 -05:00
Brandon Myers a91bc930e8
Fixup sqs connection with credentials 2018-10-26 17:37:37 -05:00
Gene Wood 22398eab5d
Change connect_sqs parameter names to match boto 2018-10-26 12:59:44 -07:00
Brandon Myers 22d669e417
Fix imports missing in papertrail worker 2018-10-26 14:55:32 -05:00
Gene Wood 1f536dcdd3
Add apiversion as a field forced into string type 
This should resolve this error in the esworker_cloudtrail
`RequestError: TransportError(400, u'mapper_parsing_exception', u'failed to parse [details.apiversion]')`
 2018-10-25 14:31:20 -07:00
Gene Wood 5428d76f52
Merge pull request #867 from gene1wood/cloudify-cloudtrail-worker 
Enable use of boto native access resolution and make role assumption optional in CloudTrail ES worker
 2018-10-25 10:27:55 -07:00
Gene Wood fbc682f852
Update lib.sqs to handle missing AWS API keys 2018-10-25 10:27:05 -07:00
Brandon Myers 8ef1e1ae48
Merge remote-tracking branch 'origin/master' into infosec_workweek 2018-10-25 12:14:19 -05:00
Brandon Myers 9b66dee995
Add logstreamname key to cloudtrail plugin 2018-10-24 19:06:45 -05:00
Brandon Myers a5b4970fc3
Merge remote-tracking branch 'origin/master' into fixup_merge_conflicts 2018-10-24 14:08:01 -05:00
Brandon Myers b4a77b1449
Merge pull request #755 from mpurzynski/ipfixup_clusterip 
If cluster_client_ip is present there seems to be no reason to use th…
 2018-10-24 13:15:30 -05:00
Brandon Myers 663fd76ab2
Merge remote-tracking branch 'origin/infosec_workweek' into virtualenv_path_change 2018-10-24 13:05:30 -05:00
Gene Wood f5c8499517
Enable use of boto native access resolution and make role assumption optional 
Previously the default region to look for the CloudTrail SQS queue in was set
to us-west-1 as a default in the es_worker. This is now set to '' in the
es_worker and as a result will default to whatever region boto determines

This commit introduces a new function, get_aws_credentials, which accepts AWS
API key arguments, checks if they are set to defaults (either defaults in the
code or in the conf file) and if they are, does not return them. This enables
you to conditionally pass or not pass API keys to boto functions depending on
whether or not they were set to something other than the defaults. The result
is that by not setting API keys, MozDef will instead rely on the boto access
resoltion methods which include checking local ~/.aws/ files as well as
instance metadata.

This commit also allows the `cloudtrail_arn` variable, which is actually the
ARN of an IAM role, to be optional. If the value is set to the default, it is
ignored and no IAM Role Assumption is done when attempting to fetch data from
the S3 bucket. Instead the native credentials are used.
 2018-10-23 17:58:37 -07:00
andrewkrug 209e292bd8
fix nits 2018-10-23 10:31:27 -07:00
Zack Mullaly 71f397fd5a Fixed some broken imports 2018-10-17 16:03:00 -07:00
Zack Mullaly 13a6c7401b Replace all the imports to use mozdef_util 2018-10-16 12:45:04 -07:00
Brandon Myers 816e62b698
Add domainname to cloudtrail mapping plugin 2018-10-11 11:21:52 -05:00
Brandon Myers 03d18f914c
Reorder exceptions for network related errors in cloudtrail worker 2018-10-09 17:10:58 -04:00
Brandon Myers 6dfd213f17
Remove redundant backslach between brackets 2018-10-05 18:05:46 -04:00
Brandon Myers 82f88cf5aa
Fixup blank lines with whitespace 2018-10-05 17:51:09 -04:00
Brandon Myers 20bc4a6aba
Remove trailing whitespace 2018-10-05 17:47:49 -04:00
Brandon Myers 7689ea0d20
Remove too many blank lines 2018-10-05 17:46:00 -04:00
Brandon Myers c255c94c67
Remove whitespace before parenthesis 2018-10-05 17:37:47 -04:00
Brandon Myers 06f0e78c5a
Remove whitespace after parenthesis 2018-10-05 17:34:36 -04:00
Phrozyn da03c9f821
Fixing indentation error 2018-10-03 17:24:52 -05:00
A Smith 589cf2c0d0
Merge pull request #756 from mozilla/modify_sqs_drop_nondict 
Modify sqs worker to drop non dict messages
 2018-10-03 17:13:55 -04:00
Brandon Myers fed01844d9
Modify sqs worker to drop non dict messages 2018-10-02 14:53:33 -04:00
Michal Purzynski 004047c471 Second part to actually add the ip address 2018-10-02 18:26:20 +02:00
Michal Purzynski b5f3afad0c If cluster_client_ip is present there seems to be no reason to use the sourceipaddress. The cluster_client_ip should overwrite as the 'true' client's IP. This is to enable anomaly detection, like Geo, on traffic going through load balancers. 2018-10-02 18:20:49 +02:00
Phrozyn 62ac957471
Correcting typo 2018-10-01 13:50:40 -05:00
Phrozyn 29ce658a2e
Fixing details.dhost to be hostname 2018-10-01 11:32:38 -05:00
A Smith 8962bcaf1d
Merge pull request #752 from mozilla/fixup_sqs_worker 
Fixup sqs workers to handle network errors
 2018-09-26 13:58:56 -04:00
A Smith 012bd89906
Merge pull request #746 from mozilla/hostname_field_normalization_phaseI 
hostname field normalization phase I
 2018-09-26 13:14:00 -04:00
Brandon Myers ceebae3c6c
Modify mq workers to stop when ctrl-c 2018-09-25 19:59:07 -05:00
Brandon Myers 43d499efb7
Modify sqs workers to handle network connection error 2018-09-25 19:57:39 -05:00
Brandon Myers 144f5b4fe1
Merge pull request #749 from mpurzynski/suricatafixup 
Rename details.alert to details.suricata_alert to avoid conflicts
 2018-09-19 14:28:58 -05:00
Michal Purzynski b04469d0c1 Rename details.alert to details.suricata_alert to avoid conflicts 2018-09-19 12:14:34 -07:00
Brandon Myers 44a1840a2e
Merge pull request #745 from mpurzynski/suricatafixup 
Initial version of the plugin that parses Suricata eve-log alerts and…
 2018-09-19 13:23:37 -05:00
Michal Purzynski 16a5146ae9 Remove unsed code. 2018-09-17 11:43:59 -07:00
Phrozyn fe7e6cb988
moved hostname out of details. 2018-09-13 15:53:38 -05:00
Phrozyn 44a81da8d6
hostname field normalization phase I 2018-09-13 14:04:22 -05:00
Michal Purzynski ba05341f19 Initial version of the plugin that parses Suricata eve-log alerts and matches field names to Bro 2018-09-12 19:25:05 -07:00
Brandon Myers 1150857fd9
Add callerReference to cloudtrail plugin handler 2018-09-06 13:59:27 -05:00
Phrozyn 638a2220bc
changing modification of sourceip to eventsourceipaddress instead of sourceipaddress. 2018-08-19 19:36:39 -05:00
A Smith 371158e5db
Merge pull request #733 from mozilla/properly_kill_bulk_queue 
Modify workers to stop bulk queue on errors
 2018-08-08 14:06:10 -05:00
Brandon Myers 80e3cc78b9
Removed unused sys exit in sqs worker 2018-08-06 13:11:39 -05:00
Brandon Myers a4980a249f
Modify workers to stop bulk queue on errors 2018-08-06 13:09:58 -05:00
Brandon Myers 67cc8be0fe
Add more keys to cloudtrail plugin 2018-08-06 11:14:03 -05:00
Brandon Myers dec8c1ec51
Add parsing for request source in cloudtrail plugin 2018-08-02 12:39:20 -05:00
Jeff Bryner 0e1ef26a90
Add details.requestparameters.instanceType 
Log errors say details.requestparameters.instanceType is sometimes an object:
"instanceType": {"value": "t2.medium"}}
 2018-07-28 12:09:06 -07:00
Brandon Myers 9e05f32acc
Add responseelements lastModified in cloudtrail plugin 2018-07-20 12:08:26 -05:00
Brandon Myers b77e38f8b0
Modify bro plugin to properly handle unicode for smtp 2018-07-19 10:50:45 -05:00
Jeff Bryner 2fe84fad0a
rename details.service to details.finding 
As per: https://github.com/mozilla/guardDuty2MozDef/pull/1/files
 2018-07-11 09:22:59 -07:00
Brandon Myers e4c096a680
Merge pull request #712 from mozilla/GuardDuty-Plugin 
Guard duty plugin
 2018-06-07 18:21:50 -05:00
Jeff Bryner ee14fb2c76
Pull in required fields 
If the sqs message contains, source, summary or processname use them.
 2018-06-07 10:32:59 -07:00
Jeff Bryner edd2f40db5 dot dict import 2018-06-06 12:25:54 -07:00
Jeff Bryner daf5a7db83 guard duty fixup for dates and ip addresses 2018-06-06 12:24:49 -07:00
Brandon Myers 07ed39a39d
Convert value to string for cloudtrail plugin 2018-05-08 18:12:34 -05:00
Brandon Myers 7634112ac6
Lower severity of few logger statements papertrail 2018-05-08 16:10:55 -05:00
A Smith a987b32893
Merge pull request #680 from mozilla/retry_papertrail_error 
Add retry error handling to papertrail worker
 2018-05-08 09:49:10 -07:00
A Smith 6e9d49bd81
Merge pull request #682 from mozilla/add_long_message_plugin 
Add plugin to cut off long message fields
 2018-05-08 09:48:15 -07:00
Brandon Myers 46d6bd1420
Add few more keys to cloudtrail plugin 2018-05-07 21:33:47 -05:00
Brandon Myers 98302918e0
Convert cloudtrail over to dynamic string mapping modification 2018-05-07 21:27:45 -05:00
Brandon Myers 382cd8b50c
Add plugin to cut off long message fields 2018-05-07 16:49:26 -05:00
Brandon Myers 9294d97e3e
Change severity of log line in papertrail worker 2018-05-07 15:44:00 -05:00
Brandon Myers 6dc3944886
Add retry error handling to papertrail worker 2018-05-07 11:48:38 -05:00
A Smith 85c6fdf12b
Merge pull request #677 from mozilla/fixup_sso_feedback 
Fixup worker and alert for sso feedback events
 2018-04-30 15:34:10 -05:00
Brandon Myers 26701ffa15
Fixup alert and worker for SSO feedback events 2018-04-30 12:43:59 -05:00
Brandon Myers d4514e943b
Update ini files to use new virtualenv path 2018-04-20 13:23:36 -05:00
Michal Purzynski 9a85cadd2e Always truncate the details.uri to prevent scanners from crashing us 2018-04-10 14:22:21 -07:00
A Smith 2651326fb2
Merge pull request #646 from mozilla/add_error_handling_papertrail 
Add error handling to papertrail worker
 2018-04-04 10:21:36 -05:00
Michal Purzynski 4699a05b68 Fixup BroFixup by moving the software's log fields away so they don't conflict with a details.version 2018-04-02 10:41:06 -07:00
Brandon Myers e3cd22c585
Add error handling to papertrail worker 2018-03-22 12:39:51 -05:00
Brandon Myers 3445ebdae3
Add handling of securitygroups in cloudtrail plugin 2018-03-15 12:52:07 -05:00
Brandon Myers a98b7136a1
Merge pull request #593 from mpurzynski/master 
A new and better version of brofixup for syslog-ng plus some tiny cle…
 2018-03-08 16:43:56 -06:00
Phrozyn e9a46b5aff
reverting processname edits but leaving regex changes 2018-03-08 10:29:28 -06:00
Tristan Weir a7ce5126c5 Added additional logic to check for field before analysing 2018-03-03 07:51:24 -08:00
A Smith 67da3b7ad2
Merge pull request #622 from mozilla/add_ebs_cloudtrail_mapping 
Add mapping exception for ebsoptimized in cloudtrail plugin
 2018-03-02 14:10:14 -06:00
Phrozyn 75ebf49cdc
removing session_closed regex since it's handled by session_open regex. 2018-03-02 13:41:29 -06:00
Phrozyn e86347b5f3
correcting if statement for details.program. 2018-03-02 12:47:24 -06:00
Brandon Myers c75349e0a3
Add mapping exception for ebsoptimized in cloudtrail plugin 2018-02-26 14:09:58 -06:00
Michal Purzynski 6ca16ae21d Contributors 2018-02-26 10:55:59 -08:00
Michal Purzynski 648d088731 Changes as requested 2018-02-23 16:56:59 -08:00
Phrozyn 1a87bd7764
Updates to parse_sshd.py to account for other fingerprint types. 2018-02-23 18:26:12 -06:00
Brandon Myers 27f928daba
Modify cloudtrail plugin to convert objects to string 2018-02-15 14:45:20 -06:00
Brandon Myers b5e118c0c0
Modify cloudtrail plugin to handle details.responseelements.endpoint 2018-02-15 13:32:35 -06:00
Michal Purzynski 951fcf61c0 A completely new version of the brofixup code with unit tests 2018-02-14 21:01:34 -08:00
Michal Purzynski ea6e080504 Merge remote-tracking branch 'upstream/master' 2018-02-14 21:00:40 -08:00
Brandon Myers 79fd605d3d
Add rule and subnets to cloudtrail plugin 2018-02-14 11:07:40 -06:00
Brandon Myers 55b9f2e840
Improve cloudtrail plugin parsing of string fields 2018-02-13 14:53:43 -06:00