mozilla
/
MozDef
зеркало из https://github.com/mozilla/MozDef.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
5 612 коммитов 4 Ветки 53 Теги 68 MiB
Граф коммитов

548 Коммитов

Автор SHA1 Сообщение Дата
Brandon Myers a91bc930e8
Fixup sqs connection with credentials 2018-10-26 17:37:37 -05:00
Gene Wood 22398eab5d
Change connect_sqs parameter names to match boto 2018-10-26 12:59:44 -07:00
Brandon Myers 22d669e417
Fix imports missing in papertrail worker 2018-10-26 14:55:32 -05:00
Gene Wood 1f536dcdd3
Add apiversion as a field forced into string type 
This should resolve this error in the esworker_cloudtrail
`RequestError: TransportError(400, u'mapper_parsing_exception', u'failed to parse [details.apiversion]')`
 2018-10-25 14:31:20 -07:00
Gene Wood 5428d76f52
Merge pull request #867 from gene1wood/cloudify-cloudtrail-worker 
Enable use of boto native access resolution and make role assumption optional in CloudTrail ES worker
 2018-10-25 10:27:55 -07:00
Gene Wood fbc682f852
Update lib.sqs to handle missing AWS API keys 2018-10-25 10:27:05 -07:00
Brandon Myers 8ef1e1ae48
Merge remote-tracking branch 'origin/master' into infosec_workweek 2018-10-25 12:14:19 -05:00
Brandon Myers 9b66dee995
Add logstreamname key to cloudtrail plugin 2018-10-24 19:06:45 -05:00
Brandon Myers a5b4970fc3
Merge remote-tracking branch 'origin/master' into fixup_merge_conflicts 2018-10-24 14:08:01 -05:00
Brandon Myers b4a77b1449
Merge pull request #755 from mpurzynski/ipfixup_clusterip 
If cluster_client_ip is present there seems to be no reason to use th…
 2018-10-24 13:15:30 -05:00
Brandon Myers 663fd76ab2
Merge remote-tracking branch 'origin/infosec_workweek' into virtualenv_path_change 2018-10-24 13:05:30 -05:00
Gene Wood f5c8499517
Enable use of boto native access resolution and make role assumption optional 
Previously the default region to look for the CloudTrail SQS queue in was set
to us-west-1 as a default in the es_worker. This is now set to '' in the
es_worker and as a result will default to whatever region boto determines

This commit introduces a new function, get_aws_credentials, which accepts AWS
API key arguments, checks if they are set to defaults (either defaults in the
code or in the conf file) and if they are, does not return them. This enables
you to conditionally pass or not pass API keys to boto functions depending on
whether or not they were set to something other than the defaults. The result
is that by not setting API keys, MozDef will instead rely on the boto access
resoltion methods which include checking local ~/.aws/ files as well as
instance metadata.

This commit also allows the `cloudtrail_arn` variable, which is actually the
ARN of an IAM role, to be optional. If the value is set to the default, it is
ignored and no IAM Role Assumption is done when attempting to fetch data from
the S3 bucket. Instead the native credentials are used.
 2018-10-23 17:58:37 -07:00
andrewkrug 209e292bd8
fix nits 2018-10-23 10:31:27 -07:00
Zack Mullaly 71f397fd5a Fixed some broken imports 2018-10-17 16:03:00 -07:00
Zack Mullaly 13a6c7401b Replace all the imports to use mozdef_util 2018-10-16 12:45:04 -07:00
Brandon Myers 816e62b698
Add domainname to cloudtrail mapping plugin 2018-10-11 11:21:52 -05:00
Brandon Myers 03d18f914c
Reorder exceptions for network related errors in cloudtrail worker 2018-10-09 17:10:58 -04:00
Brandon Myers 6dfd213f17
Remove redundant backslach between brackets 2018-10-05 18:05:46 -04:00
Brandon Myers 82f88cf5aa
Fixup blank lines with whitespace 2018-10-05 17:51:09 -04:00
Brandon Myers 20bc4a6aba
Remove trailing whitespace 2018-10-05 17:47:49 -04:00
Brandon Myers 7689ea0d20
Remove too many blank lines 2018-10-05 17:46:00 -04:00
Brandon Myers c255c94c67
Remove whitespace before parenthesis 2018-10-05 17:37:47 -04:00
Brandon Myers 06f0e78c5a
Remove whitespace after parenthesis 2018-10-05 17:34:36 -04:00
Phrozyn da03c9f821
Fixing indentation error 2018-10-03 17:24:52 -05:00
A Smith 589cf2c0d0
Merge pull request #756 from mozilla/modify_sqs_drop_nondict 
Modify sqs worker to drop non dict messages
 2018-10-03 17:13:55 -04:00
Brandon Myers fed01844d9
Modify sqs worker to drop non dict messages 2018-10-02 14:53:33 -04:00
Michal Purzynski 004047c471 Second part to actually add the ip address 2018-10-02 18:26:20 +02:00
Michal Purzynski b5f3afad0c If cluster_client_ip is present there seems to be no reason to use the sourceipaddress. The cluster_client_ip should overwrite as the 'true' client's IP. This is to enable anomaly detection, like Geo, on traffic going through load balancers. 2018-10-02 18:20:49 +02:00
Phrozyn 62ac957471
Correcting typo 2018-10-01 13:50:40 -05:00
Phrozyn 29ce658a2e
Fixing details.dhost to be hostname 2018-10-01 11:32:38 -05:00
A Smith 8962bcaf1d
Merge pull request #752 from mozilla/fixup_sqs_worker 
Fixup sqs workers to handle network errors
 2018-09-26 13:58:56 -04:00
A Smith 012bd89906
Merge pull request #746 from mozilla/hostname_field_normalization_phaseI 
hostname field normalization phase I
 2018-09-26 13:14:00 -04:00
Brandon Myers ceebae3c6c
Modify mq workers to stop when ctrl-c 2018-09-25 19:59:07 -05:00
Brandon Myers 43d499efb7
Modify sqs workers to handle network connection error 2018-09-25 19:57:39 -05:00
Brandon Myers 144f5b4fe1
Merge pull request #749 from mpurzynski/suricatafixup 
Rename details.alert to details.suricata_alert to avoid conflicts
 2018-09-19 14:28:58 -05:00
Michal Purzynski b04469d0c1 Rename details.alert to details.suricata_alert to avoid conflicts 2018-09-19 12:14:34 -07:00
Brandon Myers 44a1840a2e
Merge pull request #745 from mpurzynski/suricatafixup 
Initial version of the plugin that parses Suricata eve-log alerts and…
 2018-09-19 13:23:37 -05:00
Michal Purzynski 16a5146ae9 Remove unsed code. 2018-09-17 11:43:59 -07:00
Phrozyn fe7e6cb988
moved hostname out of details. 2018-09-13 15:53:38 -05:00
Phrozyn 44a81da8d6
hostname field normalization phase I 2018-09-13 14:04:22 -05:00
Michal Purzynski ba05341f19 Initial version of the plugin that parses Suricata eve-log alerts and matches field names to Bro 2018-09-12 19:25:05 -07:00
Brandon Myers 1150857fd9
Add callerReference to cloudtrail plugin handler 2018-09-06 13:59:27 -05:00
Phrozyn 638a2220bc
changing modification of sourceip to eventsourceipaddress instead of sourceipaddress. 2018-08-19 19:36:39 -05:00
A Smith 371158e5db
Merge pull request #733 from mozilla/properly_kill_bulk_queue 
Modify workers to stop bulk queue on errors
 2018-08-08 14:06:10 -05:00
Brandon Myers 80e3cc78b9
Removed unused sys exit in sqs worker 2018-08-06 13:11:39 -05:00
Brandon Myers a4980a249f
Modify workers to stop bulk queue on errors 2018-08-06 13:09:58 -05:00
Brandon Myers 67cc8be0fe
Add more keys to cloudtrail plugin 2018-08-06 11:14:03 -05:00
Brandon Myers dec8c1ec51
Add parsing for request source in cloudtrail plugin 2018-08-02 12:39:20 -05:00
Jeff Bryner 0e1ef26a90
Add details.requestparameters.instanceType 
Log errors say details.requestparameters.instanceType is sometimes an object:
"instanceType": {"value": "t2.medium"}}
 2018-07-28 12:09:06 -07:00
Brandon Myers 9e05f32acc
Add responseelements lastModified in cloudtrail plugin 2018-07-20 12:08:26 -05:00
Brandon Myers b77e38f8b0
Modify bro plugin to properly handle unicode for smtp 2018-07-19 10:50:45 -05:00
Jeff Bryner 2fe84fad0a
rename details.service to details.finding 
As per: https://github.com/mozilla/guardDuty2MozDef/pull/1/files
 2018-07-11 09:22:59 -07:00
Brandon Myers e4c096a680
Merge pull request #712 from mozilla/GuardDuty-Plugin 
Guard duty plugin
 2018-06-07 18:21:50 -05:00
Jeff Bryner ee14fb2c76
Pull in required fields 
If the sqs message contains, source, summary or processname use them.
 2018-06-07 10:32:59 -07:00
Jeff Bryner edd2f40db5 dot dict import 2018-06-06 12:25:54 -07:00
Jeff Bryner daf5a7db83 guard duty fixup for dates and ip addresses 2018-06-06 12:24:49 -07:00
Brandon Myers 07ed39a39d
Convert value to string for cloudtrail plugin 2018-05-08 18:12:34 -05:00
Brandon Myers 7634112ac6
Lower severity of few logger statements papertrail 2018-05-08 16:10:55 -05:00
A Smith a987b32893
Merge pull request #680 from mozilla/retry_papertrail_error 
Add retry error handling to papertrail worker
 2018-05-08 09:49:10 -07:00
A Smith 6e9d49bd81
Merge pull request #682 from mozilla/add_long_message_plugin 
Add plugin to cut off long message fields
 2018-05-08 09:48:15 -07:00
Brandon Myers 46d6bd1420
Add few more keys to cloudtrail plugin 2018-05-07 21:33:47 -05:00
Brandon Myers 98302918e0
Convert cloudtrail over to dynamic string mapping modification 2018-05-07 21:27:45 -05:00
Brandon Myers 382cd8b50c
Add plugin to cut off long message fields 2018-05-07 16:49:26 -05:00
Brandon Myers 9294d97e3e
Change severity of log line in papertrail worker 2018-05-07 15:44:00 -05:00
Brandon Myers 6dc3944886
Add retry error handling to papertrail worker 2018-05-07 11:48:38 -05:00
A Smith 85c6fdf12b
Merge pull request #677 from mozilla/fixup_sso_feedback 
Fixup worker and alert for sso feedback events
 2018-04-30 15:34:10 -05:00
Brandon Myers 26701ffa15
Fixup alert and worker for SSO feedback events 2018-04-30 12:43:59 -05:00
Brandon Myers d4514e943b
Update ini files to use new virtualenv path 2018-04-20 13:23:36 -05:00
Michal Purzynski 9a85cadd2e Always truncate the details.uri to prevent scanners from crashing us 2018-04-10 14:22:21 -07:00
A Smith 2651326fb2
Merge pull request #646 from mozilla/add_error_handling_papertrail 
Add error handling to papertrail worker
 2018-04-04 10:21:36 -05:00
Michal Purzynski 4699a05b68 Fixup BroFixup by moving the software's log fields away so they don't conflict with a details.version 2018-04-02 10:41:06 -07:00
Brandon Myers e3cd22c585
Add error handling to papertrail worker 2018-03-22 12:39:51 -05:00
Brandon Myers 3445ebdae3
Add handling of securitygroups in cloudtrail plugin 2018-03-15 12:52:07 -05:00
Brandon Myers a98b7136a1
Merge pull request #593 from mpurzynski/master 
A new and better version of brofixup for syslog-ng plus some tiny cle…
 2018-03-08 16:43:56 -06:00
Phrozyn e9a46b5aff
reverting processname edits but leaving regex changes 2018-03-08 10:29:28 -06:00
Tristan Weir a7ce5126c5 Added additional logic to check for field before analysing 2018-03-03 07:51:24 -08:00
A Smith 67da3b7ad2
Merge pull request #622 from mozilla/add_ebs_cloudtrail_mapping 
Add mapping exception for ebsoptimized in cloudtrail plugin
 2018-03-02 14:10:14 -06:00
Phrozyn 75ebf49cdc
removing session_closed regex since it's handled by session_open regex. 2018-03-02 13:41:29 -06:00
Phrozyn e86347b5f3
correcting if statement for details.program. 2018-03-02 12:47:24 -06:00
Brandon Myers c75349e0a3
Add mapping exception for ebsoptimized in cloudtrail plugin 2018-02-26 14:09:58 -06:00
Michal Purzynski 6ca16ae21d Contributors 2018-02-26 10:55:59 -08:00
Michal Purzynski 648d088731 Changes as requested 2018-02-23 16:56:59 -08:00
Phrozyn 1a87bd7764
Updates to parse_sshd.py to account for other fingerprint types. 2018-02-23 18:26:12 -06:00
Brandon Myers 27f928daba
Modify cloudtrail plugin to convert objects to string 2018-02-15 14:45:20 -06:00
Brandon Myers b5e118c0c0
Modify cloudtrail plugin to handle details.responseelements.endpoint 2018-02-15 13:32:35 -06:00
Michal Purzynski 951fcf61c0 A completely new version of the brofixup code with unit tests 2018-02-14 21:01:34 -08:00
Michal Purzynski ea6e080504 Merge remote-tracking branch 'upstream/master' 2018-02-14 21:00:40 -08:00
Brandon Myers 79fd605d3d
Add rule and subnets to cloudtrail plugin 2018-02-14 11:07:40 -06:00
Brandon Myers 55b9f2e840
Improve cloudtrail plugin parsing of string fields 2018-02-13 14:53:43 -06:00
Brandon Myers d16ac47ab8
Update cloudtrail plugin to handle description field type error 2018-02-07 11:43:58 -06:00
Brandon Myers a7058333f3
Add additional safe checks to cloudtrail mq plugin 2018-02-01 13:13:10 -06:00
Brandon Myers 49dc451097
Modify cloudtrail plugin to match on source 2018-02-01 13:02:30 -06:00
Brandon Myers 3cd95c22fe
Change key names to raw_value for details string in messages 2018-01-31 18:10:53 -06:00
Brandon Myers c160030a1b
Convert object type handling for cloudtrail into plugin 2018-01-31 18:07:59 -06:00
Michal Purzynski 927e4d9436 A new and better version of brofixup for syslog-ng plus some tiny cleanups 2018-01-29 14:47:45 -08:00
Brandon Myers eb7ec7ad6a
Modify workers to handle details key as non dict 2018-01-25 12:33:55 -06:00
Brandon Myers 4e4699eb95
Reapply cloudtrail worker improvements 2018-01-18 12:41:41 -06:00
Brandon Myers ec7efb70c3
Add logic to drop event in sns sqs worker 2018-01-12 15:48:16 -06:00
Brandon Myers c18875f65b
Add try except to on_message in cloudtrail worker 2018-01-12 15:05:00 -06:00
Brandon Myers 08762af4b7
Remove unnecessary new line in logger statement 2018-01-12 15:04:34 -06:00
Brandon Myers e5be0a0a3f
Convert sns sqs worker to use logger 2018-01-12 14:51:03 -06:00
Brandon Myers 7833800975
Modify sqs worker to use logger 2018-01-12 14:50:45 -06:00
Brandon Myers 4b248bde1c
Convert papertrail worker to using logger 2018-01-12 14:45:14 -06:00
Brandon Myers df4c12dafd
Convert cloudtrail esworker to using logger 2018-01-12 14:44:55 -06:00
Brandon Myers 38ddb2ee1a
Add logger to mq plugins 2018-01-12 14:44:31 -06:00
Brandon Myers 5835665e55
Log malformed event in eventtask worker 2018-01-11 17:02:33 -06:00
Brandon Myers 7c602afdf9
Switch workers to use lib functions 2018-01-11 16:07:12 -06:00
Brandon Myers c60c7b8c36
Remove extra line after copywrite date 2018-01-04 17:15:35 -06:00
Yash Mehrotra 90d7e3b6d3
Remove free-form 'Contributor:' text from code. Fixes #407 2017-12-23 02:14:53 +05:30
Brandon Myers 6ff09b9de6
Provide temporary patch for cloudtrail worker 2017-12-19 14:14:08 -06:00
Brandon Myers 59d95ff178
Revert "Merge pull request #554 from mozilla/improve_cloudtrail_worker" 
This reverts commit 501819cfb5, reversing
changes made to b09c700cb9.
 2017-12-08 16:09:57 -06:00
Brandon Myers f73cc3364d
Revert "Merge pull request #560 from mozilla/fix_cloudtrail_mapping" 
This reverts commit 804757f242, reversing
changes made to 501819cfb5.
 2017-12-08 16:09:43 -06:00
Brandon Myers ed49aee5ab
Fix missing import statements 2017-11-28 12:54:57 -06:00
Brandon Myers b006036528
Uppercase cloudtrail verb by default 2017-11-28 12:53:31 -06:00
Brandon Myers 4190ef43d6
Remove debugger line in mq worker 2017-11-15 17:25:14 -06:00
Brandon Myers 7c474d72ce
Update cloudtrail esworker fields 2017-11-15 17:16:49 -06:00
Brandon Myers 4278ffa39f
Update description of mq plugin 2017-11-13 22:25:30 -06:00
Brandon Myers f97b0f0c70
Add filterlog firewall mq plugin 2017-11-13 22:21:40 -06:00
Brandon Myers 58fa07d7cf
Add support to eventtask worker for syslog messages 2017-10-30 13:14:45 -05:00
Michal Purzynski d9ff430b21 Use the Bro's src field as sourceipaddress if present 2017-10-26 15:14:14 -07:00
Michal Purzynski aa7097156d Change the type field name to source - ES has problems if there is _type and type 2017-10-14 16:53:42 -07:00
A Smith f7834f79d2 Merge pull request #490 from mpurzynski/normalization_auth 
Normalization auth
 2017-10-12 11:00:17 -05:00
Brandon Myers 8ef7c4fd71
Merge remote-tracking branch 'origin' into add_events_class 2017-10-10 13:15:51 -05:00
Phrozyn 0f6cbd5fde
Merge branch 'naming_convention_changes' of https://github.com/Phrozyn/MozDef into naming_convention_changes 2017-10-10 10:59:42 -05:00
Phrozyn 7cf87ac628
Merge branch 'master' of https://github.com/mozilla/MozDef into naming_convention_changes 2017-10-10 10:59:27 -05:00
Phrozyn b6d5d1b57c
Fixing merge conflict 2017-10-10 10:55:13 -05:00
Phrozyn 1fd7335355
Naming Convention and Logging Changes. 2017-10-04 15:59:49 -05:00
Brandon Myers c4134f1764
Modify mq workers to use save_event method from es client 2017-09-28 14:57:18 -05:00
Brandon Myers badd86a44f Merge pull request #456 from mpurzynski/brofixup 
A first take on the new brofixup plugin.
 2017-09-28 12:02:20 -05:00
Michal Purzynski 435a267922 Last minute changes 2017-09-27 15:48:14 -07:00
Michal Purzynski 8a465bf29a More small fixes, correct unicode handling in SMTP summary 2017-09-27 13:33:08 -07:00
Michal Purzynski a8016907eb Even more refactoring and small changes 2017-09-26 10:25:34 -07:00
Michal Purzynski 991d94308a More unit tests 2017-09-25 17:42:58 -07:00
Michal Purzynski 2e18a286dd Testing never ends 2017-09-22 17:14:29 -04:00
Michal Purzynski c234e19b3f Small fixups 2017-09-21 16:46:25 -04:00
Brandon Myers 6db687cfb5
Modify esworker sns sqs to cast processid to str 2017-09-21 14:57:15 -05:00
Michal Purzynski ede31aad62 Small fixups here and there 2017-09-20 18:02:11 -04:00
Phrozyn bc3b56d151
Corrected some typos and added syslog change to syslog filter 2017-09-05 11:58:05 -05:00
Phrozyn 1a1a892dac
Merge branch 'master' of https://github.com/Phrozyn/MozDef into replace_dots_with_underscores_in_filenames 2017-09-05 10:18:09 -05:00
Gene Wood 6cd241a329 Extract action verb and add it along with readonly to the event 2017-09-01 13:11:28 -07:00
Michal Purzynski fa67e3d5d7 Even more cleanups 2017-08-31 16:40:28 -07:00
Michal Purzynski ccc7aae3c8 Initial commit for the data normalization initiative 2017-08-30 15:55:33 -07:00
Michal Purzynski 74dd2c2374 A first take on the new brofixup plugin. 2017-08-29 15:58:09 -07:00
Phrozyn 6199701f61
updated papertrail with changes from repo. 2017-08-25 13:34:45 -05:00
Phrozyn 4f1007a134
Updated code to reflect naming convention changes. 2017-08-25 12:17:53 -05:00
Phrozyn 2c415b673b
updated dots to underscores 2017-08-25 11:58:31 -05:00
Brandon Myers e396e5f230
Remove unused functions from esworker 2017-08-23 15:33:49 -04:00
Brandon Myers a7934e6f9b
Remove unused functions from mq 2017-08-23 15:22:48 -04:00
Brandon Myers 40fb30172f
Change default mq creds in conf 2017-08-17 18:21:07 -05:00
Brandon Myers 81fa3819cc
Update bot and mq plugin to use GeoIP class 2017-08-08 12:46:54 -05:00
Brandon Myers 4b665d8771
Convert registration term to lowercase fxa plugin 2017-07-17 13:18:48 -05:00
Brandon Myers caaf662ab7
Update fxa mq plugin to use new category 2017-07-17 13:01:44 -05:00
Brandon Myers ad64804e32
Add travisci to project and stabalize tests 2017-07-05 16:37:41 -05:00
Brandon Myers 63b3cf2194
Remove old leftover files 2017-06-15 15:13:03 -05:00
Brandon Myers fe96636655
Improve cloudtrail mq worker 2017-06-15 15:07:46 -05:00
Brandon Myers c632ed8250
Fix mozillaLocation mq plugin 2017-06-15 15:07:46 -05:00
Brandon Myers c6aaa8add8
Remove mozilla mq worker sample conf files 2017-06-15 15:07:45 -05:00
Brandon Myers cd25328625
Remove mozilla specific workers 2017-06-15 15:07:45 -05:00
Brandon Myers e59d2097ed
Remove default rabbitmq config 2017-06-15 15:07:44 -05:00
Brandon Myers b52c506810
Add defaults for sns sqs worker 2017-06-15 15:07:44 -05:00
Brandon Myers 29e3dec9ed
Add alerts to use config files 2017-06-15 15:07:42 -05:00
Brandon Myers bac6c7450a
Remove unncessary parsys file 2017-06-15 15:07:40 -05:00
Brandon Myers 43a722c65d
Fix typo in parsys ini file 2017-06-15 15:07:40 -05:00
Brandon Myers 1c4fc1071c
Remove unused mq workers 2017-06-15 15:07:38 -05:00
Brandon Myers 496311a364
Add parsys mq worker 2017-06-15 15:07:30 -05:00
Brandon Myers 9e734175e7
Add SNS SQS mq worker 2017-06-15 15:07:30 -05:00
Phrozyn ab3714d22a
Adding log drain back into uwsgi ini files. 2017-06-15 15:07:28 -05:00
Phrozyn 06899804fb
Adding contegix-auditd service and dummy conf and ini. 2017-06-15 15:07:25 -05:00
Phrozyn 1b4716ad2c
Moving uwsgi logging to syslog. 2017-06-15 15:07:22 -05:00
Phrozyn 24c2df918f
New contegix worker 2017-06-15 15:07:21 -05:00
Brandon Myers 5180f496e9
Add files for SSO sqs worker 2017-06-15 15:07:19 -05:00
Brandon Myers 7873cc38ea
Add thread to reauth every 30 minutes cloudtrail 2017-06-15 15:07:18 -05:00
Brandon Myers dbb78759ed
Add prefetch option to get_messages 2017-06-15 15:07:18 -05:00
Brandon Myers 1e300f7915
Add exception handling to cloudtrail worker 2017-06-15 15:07:18 -05:00
Brandon Myers 48e008346e
Add bulk to cloudtrail worker 2017-06-15 15:07:18 -05:00
Brandon Myers aa497395a7
Switch cloudtrail from cron to mq worker 2017-06-15 15:07:17 -05:00
Phrozyn 028505cd3b
Adding new mqworkers for fluentd2mozdef from aws infosec services. 2017-06-15 15:06:21 -05:00
Aaron Meihm 39ab8738ea
add configuration to drain mig sqs log queue 2017-06-15 15:06:02 -05:00
Brandon Myers f87c94a088
Unencrypt config files 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:55 -05:00
Phrozyn 97b0d685c6
Fixing mule issue in fxa with moar mules. 2017-06-15 15:05:53 -05:00
Brandon Myers d7a38c83f5
Remove creds from mq directory 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:43 -05:00
Brandon Myers 5fb9fbea7d
Move papertrail disabled to ini script 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:42 -05:00
Brandon Myers fb8806814b
Remove prod versions of esworker conf 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:42 -05:00
Phrozyn 7b903a1b81
Changing filenames to reflect a better workflow from dev to stage to prod. 2017-06-15 15:05:42 -05:00
Phrozyn 8da9fb5c34
Correcting mqwFxa.ini to remove stage from pid path 2017-06-15 15:05:41 -05:00
Brandon Myers c7b1e934b4
Update location of geolitecity data file 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:40 -05:00
Brandon Myers 5d03bc03d7
Remove mules from papertrail 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:33 -05:00
Brandon Myers 577c5cecfa
Fix missing import in fluentdSqsFixup 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:32 -05:00
Brandon Myers 13aa806b1b
Move unittest from mq plugin to own file 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:32 -05:00
Brandon Myers 1fb67e49fb
Remove unittest from fluentdSqsFixup 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:31 -05:00
Phrozyn cf55546506
Omitting the FxaOauthWebserver eventsource. 2017-06-15 15:05:19 -05:00
Brandon Myers d2ea5c3334
Add missing esworker releng conf 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:07 -05:00
Brandon Myers 79b2ee84ca
Add more workers to mqwSyslog 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:07 -05:00
Brandon Myers c37c2fb7d1
Update mq creds in mq alertWorker 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:05:05 -05:00
Phrozyn 66e963fc51
Updating mq/esworker.conf mq creds for mozdef user (was using qa2) 2017-06-15 15:05:01 -05:00
Phrozyn b4ff2e575d
Updating packaged config to include mozdef4. 2017-06-15 15:05:00 -05:00
Brandon Myers ec5d1ad5b7
Keep in sync with qa1 #70 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:04:54 -05:00
Phrozyn dbfc18190f
Fixed mozdefmqwFxaStage to reflect correct pid path. 2017-06-15 15:04:54 -05:00
Phrozyn 9c6d9364de
Fixed mozdefmqwAutoland to reflect correct pid path. 2017-06-15 15:04:53 -05:00
Phrozyn 2089dc225f
Added all prod service files and mq workers. 2017-06-15 15:04:53 -05:00
Phrozyn b86413db27
Updated pid path for all uwsgi instances to run from /var/run/ 2017-06-15 15:04:53 -05:00
Brandon Myers 16abe5adcc
Remove cloudtrail fixup mapping 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:04:52 -05:00
Phrozyn 3e02f27d14
modified esservers to new cluster. 2017-06-15 15:04:45 -05:00
Brandon Myers ee07fe18a3
Modify esservers from localhost to cluster 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:45 -05:00
Brandon Myers 28080dd980
Fix remaining qa references in prod 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:45 -05:00
Brandon Myers 75bb6542ee
Merge prod mq ini files with qa 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:44 -05:00
Brandon Myers ef6e483c7e
First import of existing files from prod 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:44 -05:00
Brandon Myers e9a4a67e5a
Modify .py scripts to use /opt dir 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:41 -05:00
Brandon Myers 007cf86c35
Modify .ini.disabled scripts to use /opt dir 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:41 -05:00
Brandon Myers 50a7cb772a
Modify .ini scripts to use /opt dir 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:40 -05:00
Brandon Myers 81a07bc2d5
Rename mozdefqa1 to localhost in configs 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:40 -05:00
Aaron Meihm a6f7c78597
update vulnerability plugin to handle version 2 messages 2017-06-15 15:03:39 -05:00
Brandon Myers 71692067cc
Add error support plus tests to bulk import 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:38 -05:00
Brandon Myers ea17b5883c
Fix toUTC isoformat problem 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:33 -05:00
Brandon Myers bfba1d3c4c
Add apiVersion mapping fix for cloudtrail 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:23 -05:00
Brandon Myers 6774599a37
Add exception in fxaFixup for fxa-auth-server 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:03:23 -05:00
Brandon Myers 6caaad320d
Remove duplicate definitions of toUTC 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:02:46 -05:00
Brandon Myers e832b313ee
Fix flush_bulk for pyes only 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:02:01 -05:00
Brandon Myers 76174add7d
Update mq directory with search class 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:02:01 -05:00
Brandon Myers 5082d87f68
Update alertWorker config 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:02:00 -05:00
Brandon Myers 49a042107e
Remove mq/safe directory and files 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:01:58 -05:00
Brandon Myers 67b38ae579
Remove mq/mq files and directory 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:01:42 -05:00
Phrozyn e7cef0564f
Adding additional mq ini changes. 2017-06-15 15:00:49 -05:00
Phrozyn edcc26f84e
Modifying thread/Process values to be in alignment with mozdefqa1's resources. Disabled unused workers. 2017-06-15 15:00:49 -05:00
Brandon Myers 375b0290de
Update conf files to use US/Pacific 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:00:48 -05:00
Brandon Myers e5e98c1304
Switch mq directory to US/Pacific 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:00:46 -05:00
Phrozyn dbc7e43e41
unencrypting ini files 2017-06-15 15:00:46 -05:00
Phrozyn ac9925be6d
adding unencrypted mqESmules.ini 
adding unencrypted mqESmulesAWS.ini
 2017-06-15 15:00:45 -05:00
Phrozyn 5c990d90ef
Unencrypting ini files. 2017-06-15 15:00:45 -05:00
Phrozyn 700f0abf5f
Releng Papertrail ini for esworker. 2017-06-15 15:00:44 -05:00
Brandon Myers 99fa7ca655
Remove rra files 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2017-06-15 15:00:42 -05:00
Gene Wood d9911b4a77
adding mozdefmq support for infosec sqs non prod queue 2017-06-15 15:00:42 -05:00
Brandon Myers 1d8c59b93f
Setup codebase for merge of two repos 2017-06-15 14:56:47 -05:00
Brandon Myers 9a2388c398 Update GeoLiteCity.dat location in mq plugin 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2016-10-19 16:35:46 -05:00
Jeff Bryner d9afcb288b Merge pull request #350 from Phrozyn/master 
corrected typo in mq/plugins/fluentdSqsFixup.py
 2016-06-28 20:51:09 -07:00
Phrozyn 58a31fdc3c corrected typo in mq/plugins/fluentdSqsFixup.py 2016-06-28 19:17:37 -05:00
Jeff Bryner a0580d1848 Merge pull request #345 from pwnbus/remove_time_fluentdSqsFixup 
Remove details.time from fluentdSqsFixup
 2016-06-08 17:07:01 -07:00
Brandon Myers f84c3ca4e1 Remove details.time from fluentdSqsFixup 
Signed-off-by: Brandon Myers <bmyers@mozilla.com>
 2016-06-08 18:01:04 -05:00
kang 950b0868eb Sync with rra2json message format 
Add support for RRA versionning
 2016-06-03 11:35:22 -07:00
Jeff Bryner 7fd56b8d93 update geoip cache file location 2016-03-23 14:13:59 -07:00
Jeff Bryner d87569d486 add common/handy options 2016-03-23 12:57:46 -07:00
Aaron Meihm a3d9668888 adds an esworker for processing data from papertrail 2016-03-01 14:57:33 -06:00
Guillaume Destuynder 09f7a038b3 Use details.program as standard field for processname instead of fluentd 2015-10-22 10:54:42 -07:00
Guillaume Destuynder 231c3415b3 Add mq plugin: normalizer for fluentd-SQS messages (AWS). Ensure registration matches your SQS queue tag. 2015-10-22 10:54:15 -07:00
Guillaume Destuynder 334f5466a4 Fix reading of SQS JSON msgs - this works regardless of messages being raw JSON or base64-encoded JSON. 
Since Boto does base64 encode messages while writing to the queue this can happen (also since we use Boto, we were
previously expecting all messages to be base64 encoded, which wouldn't work if your writer wasn't Boto)
 2015-10-20 12:44:03 -07:00
Jeff Bryner f2524fb132 Merge pull request #302 from gdestuynder/master 
Support more validation filters to accomodate different RRA fields.
 2015-10-18 12:29:34 -07:00
Guillaume Destuynder 996a566813 Support more validation filters to accomodate different RRA fields. 
This enhance the validation accuracy ;-)
 2015-10-14 17:21:49 -07:00
Jeff Bryner f259564a78 add sqs-specific worker, closes #294 2015-10-12 14:00:05 -07:00
Jeff Bryner af526d6e4e revert sqs changes due to kombu issues 2015-10-12 13:59:32 -07:00
Guillaume Destuynder ec334de898 Merge branch 'master' of https://github.com/jeffbryner/MozDef 2015-10-09 18:45:30 -07:00