mozilla
/
MozDef
зеркало из https://github.com/mozilla/MozDef.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
5 995 коммитов 4 Ветки 53 Теги 68 MiB
Граф коммитов

364 Коммитов

Автор SHA1 Сообщение Дата
Andrew Krug 1261232367
begin removing OIDC as a requirement. Make SSL optional. 2019-05-08 08:43:36 -07:00
Brandon Myers 854d9356d8
Update kibana docker env to 6.6.2 2019-05-06 14:47:09 -05:00
Brandon Myers 9a44b28be3
Change ES config to listen externally 2019-05-06 12:47:56 -05:00
Brandon Myers fdf5a7cb07
Merge remote-tracking branch 'origin/master' into breaking_es6_changes 2019-04-18 12:54:50 -05:00
Brandon Myers a1bc98b0ec
Merge pull request #1223 from mozilla/create_statefile_index 
Create mozdefstate index with specific mappings in docker environment
 2019-04-18 12:31:59 -05:00
Phrozyn 16bd77711b
adding basic license config, disables pay for features in kibana. 2019-04-17 18:14:40 -05:00
Brandon Myers e1ec513221
Create mozdefstate index with specific mappings in docker environment 2019-04-17 11:08:42 -05:00
Brandon Myers a16a857f3a
Generalize config copy to allow for other worker processes 2019-04-12 19:20:35 -05:00
Brandon Myers 33a770b9c1
Merge pull request #1214 from gene1wood/change-cicd-sequence 
Change CI/CD sequence for v1.38.5
 2019-04-10 11:23:34 -05:00
A Smith 30327e9250
Merge pull request #1179 from mozilla/update_es_docker_containers 
Update Elasticsearch6 docker container
 2019-04-09 17:31:11 -05:00
Gene Wood 3d47a4262a
Clarify meteor build step logging 2019-04-09 15:28:43 -07:00
Phrozyn 2b90d51296
Fixes dashboard error on docker spinup. 2019-04-05 18:07:04 -05:00
Gene Wood dad1af79c5
Revert bugfix for weekly alias always being created 
This reverts part of f7e1a9bacb
because it's causing the error below. Once we figure out the cause we'll
reintroduce this bugfix. This leaves the new get_aliases method in mozdef_util
in place as it's fine (though we're not actually calling it for some reason)

```
Connecting to http://elasticsearch:9200
Traceback (most recent call last):
 File "initial_setup.py", line 61, in <module>
   'events,events-previous,alerts,events-weekly')
 File "/opt/mozdef/envs/python/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
   return func(*args, params=params, **kwargs)
 File "/opt/mozdef/envs/python/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 399, in get_alias
   '_alias', name), params=params)
 File "/opt/mozdef/envs/python/lib/python2.7/site-packages/elasticsearch/transport.py", line 312, in perform_request
   status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
 File "/opt/mozdef/envs/python/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 129, in perform_request
   self._raise_error(response.status, raw_data)
 File "/opt/mozdef/envs/python/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
   raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: TransportError(404, u'index_not_found_exception', u'no such index')
```
 2019-03-29 11:23:18 -07:00
Gene Wood 1669fea708
Merge remote-tracking branch 'upstream/march_swarm' into march_swarm 2019-03-28 10:24:03 -07:00
Gene Wood 3ea1563122
Merge branch 'master' into march_swarm 
# Conflicts:
#	docker/compose/docker-compose-cloudy-mozdef.yml
 2019-03-26 14:52:37 -07:00
Gene Wood cf77a21b82
Limit crond logging 2019-03-26 14:44:44 -07:00
Gene Wood ff603be2e3
Remove need to hard code kibana version in bootstrap 
Instead this calls the kibana API and sets the defaultIndex
 2019-03-26 14:31:42 -07:00
Gene Wood f7e1a9bacb
Fix bug where weekly alias is always created 2019-03-26 14:26:50 -07:00
Gene Wood 49b34d9270
Use kibana_index_name instead of hardcoded value 2019-03-26 14:26:08 -07:00
Gene Wood 199377a095
Fix typo'd example text 2019-03-26 14:19:20 -07:00
Brandon Myers 248e7952f6
Update elasticsearch docker config file for xpack disabled 2019-03-25 12:51:55 -05:00
Brandon Myers 03ce98e965
Update elasticsearch docker container to 6 2019-03-25 12:51:45 -05:00
Gene Wood 1e822e2c0d
Revert #1155 to latest of mozilla.oidc.accessproxy 
Now that mozilla-iam/mozilla.oidc.accessproxy#36 is merged to master
and built and uploaded to dockerhub as tag `latest` we can go back
to using latest.

Note that in #1155 I updated the nginx container but missed the nginxkibana
container which should have also been pinned to 8334f96 dockerhub tag
resulting in only one of the two mozilla.oidc.accessproxy containers we
use to work correctly
 2019-03-22 10:13:31 -07:00
Gene Wood 17d99993a5
Make the docker environment variables available to cron jobs (#1158) 
cron jobs don't have access to the docker environment variables.
This, on instance launch, writes the environment to a file and
configured the mozdef crontab to use this on disk environment
 2019-03-22 09:39:57 -07:00
Gene Wood 810b529692
Pin nginx container to mozillaiam/mozilla.oidc.accessproxy:8334f96 
This gets us mozilla-iam/mozilla.oidc.accessproxy#36 which works
around the bug in zmartzone/lua-resty-openidc#249
 2019-03-21 09:54:53 -07:00
Andrew Krug 78c567226c
add params to cloudinit to set rabbitmq password. 2019-03-20 14:40:55 -07:00
Brandon Myers 3aba690c5c
Update plugin references in cloudy mozdef docker env 2019-03-20 15:26:52 -05:00
Brandon Myers f4c9d4ca04
Rename alert plugins to actions in docker env 2019-03-20 15:25:22 -05:00
Andrew Krug df290da3ec
add make target to cache container artifacts and add latest to mozdef built containers by default 2019-03-20 10:34:52 -07:00
A Smith 14640807a6
Merge pull request #1102 from mozilla/update_rabbitmq_version 
Update rabbitmq version to 3.7
 2019-03-15 16:11:41 -05:00
Brandon Myers f56b227fda
Update rabbitmq version to 3.7 2019-03-14 14:38:48 -05:00
Brandon Myers d93a76b72a
Syslog container wait 30 seconds to boot up 2019-02-28 13:02:34 -06:00
Brandon Myers c9a3d7ae16
Merge pull request #1103 from mozilla/fix_dup_generic_loader 
Fix duplicate alert generic loader
 2019-02-27 18:01:15 -05:00
Brandon Myers e0122099cd
Merge pull request #1109 from mozilla/remove_unused_depends 
Remove unnecessary depends in docker-compose
 2019-02-27 17:58:58 -05:00
Brandon Myers ab43937a66
Consolidate docker run lines for meteor container 2019-02-26 15:02:26 -05:00
Brandon Myers 20c67b91be
Change replicas to 0 for docker environment 2019-02-19 17:01:58 -06:00
Brandon Myers 0a2b5e2d11
Remove event index config for kibana index 2019-02-19 16:47:21 -06:00
Brandon Myers 01c2a43e97
Remove unnecessary depends in docker-compose 2019-02-19 13:18:36 -06:00
Brandon Myers 98cd202be9
Replace flush function with correct refresh ES one 2019-02-14 13:46:17 -06:00
Brandon Myers 8bb00651ba
Modify initial setup script to create index with config 2019-02-06 14:15:37 -06:00
Brandon Myers 7e87f73acc
Merge pull request #1094 from mozilla/sampledata_wait_elasticsearch 
Configure sampledata container to wait for ES first
 2019-02-06 12:59:57 -06:00
Brandon Myers 79fca10df1
Remove cache after yum install commands in docker files 2019-02-05 15:41:14 -06:00
Brandon Myers d67c8f3a87
Configure sampledata container to wait for ES first 2019-02-05 15:10:12 -06:00
Brandon Myers 86a2fd5bad
Merge pull request #1018 from mozilla/update_slack_bot 
Update slack bot
 2019-01-30 11:21:48 -06:00
Brandon Myers 48e71a970a
Merge pull request #1072 from darakian/add-generic-env-run 
Add generic env run
 2019-01-28 17:53:49 -06:00
Jon Moroney a08e85424b Remove volumes and networks from docker/compose/docker-compose-user-env.yml 2019-01-28 15:25:26 -08:00
Jon Moroney 64586ac1ae Update to iheritance model 2019-01-28 14:51:31 -08:00
Jon Moroney c305ece16d Copy default docker-compose but add env file 2019-01-28 10:36:21 -08:00
Brandon Myers 58b3c38edf
Merge pull request #1046 from mozilla/update_es_kibana_version 
Update ES/kibana version to 5.6.14
 2019-01-24 14:42:02 -06:00
Brandon Myers 0fd9cd476d
Modify tester dockerfile to reorder copy and pip commands 2019-01-22 19:43:27 -06:00
Brandon Myers 6ed5fd88df
Update bot docker command to wait for rabbitmq 2019-01-16 15:45:54 -06:00
Brandon Myers 905853675b
Merge remote-tracking branch 'origin/master' into update_slack_bot 2019-01-16 15:41:01 -06:00
Jeff Bryner 937681a001 nix the source of virtualenv 2019-01-15 09:34:10 -08:00
Jeff Bryner 15ccf7c9ca add dev overlays 2019-01-14 16:24:44 -08:00
Jeff Bryner bb7736a3b0 remove old semi dev overlays 2019-01-14 16:24:34 -08:00
Brandon Myers 3106308ea6
Merge pull request #1050 from mozilla/remove_unused_var 
Remove unused python version in dockerfile
 2019-01-14 17:40:19 -06:00
Brandon Myers 9906caf489
Merge pull request #1028 from mozilla/remove_hardcoded_pythonenv 
Move virtualenv code into base container
 2019-01-14 17:39:58 -06:00
Brandon Myers caab44dc8b
Merge pull request #1043 from mozilla/add_sample_dashboards 
Add sample dashboards in docker
 2019-01-14 17:39:51 -06:00
Brandon Myers 1bdbb0cee7
Remove unused python version in dockerfile 2019-01-14 14:45:21 -06:00
Brandon Myers 0a0f839c77
Update ES/kibana version to 5.6.14 2019-01-14 11:45:32 -06:00
Brandon Myers ee9e14459d
Merge pull request #1040 from gene1wood/pin-yum-repo-keys 
Pin yum repo keys
 2019-01-11 13:22:05 -06:00
Brandon Myers 04f159fde4
Add a sample dashboard to bootstrap container 2019-01-11 12:42:35 -06:00
Brandon Myers 7a1d92e2cc
Add logic to create sample visualizations and dashboards 2019-01-11 12:15:56 -06:00
Brandon Myers 18f9b7e9e4
Add logic to wait for .kibana index 2019-01-10 16:23:14 -06:00
Brandon Myers 33c1f8953c
Improve logic to determine if index_patterns are created on setup 2019-01-10 15:50:23 -06:00
Gene Wood 8039098b08
Fix the check of the mongodb key so it checks for the right key 2019-01-10 12:37:48 -08:00
Gene Wood 8ec3436a3b
Pin yum repository keys 
This pins the gpg fingerprint of the yum repositories used both
to prevent the error messages and to make any packages signed by
different keys to cause builds to fail
 2019-01-10 12:28:58 -08:00
Brandon Myers 60f0200e23
Consolidate commands in tester docker container 2019-01-04 14:30:09 -06:00
Brandon Myers d8c99f1a74
Consolidate commands in base docker container 2019-01-04 14:29:25 -06:00
Brandon Myers b0558a1fea
Move virtualenv code into base container 2019-01-04 14:21:56 -06:00
Brandon Myers 9027676942
Update path in kibana dockerfile for initial setup script 2018-12-27 15:00:58 -05:00
Brandon Myers e4accfbaca
Hardcode kibana index doc_id for index mappings 2018-12-27 14:59:59 -05:00
Brandon Myers 9b37692d26
Create index mappings automatically for kibana UI 2018-12-27 13:15:05 -05:00
Brandon Myers 0ceffca1c8
Update mozdef_bot container to support slack bot 2018-12-26 10:03:54 -06:00
Brandon Myers b307220cea
Fixup bot docker-compose configuration 2018-12-26 10:03:25 -06:00
Brandon Myers fc771bd531
Remove unused import statements 2018-12-14 11:34:42 -06:00
andrewkrug 66655a80b6
update to support cloudformation enabling alerts 2018-11-23 12:32:49 -08:00
andrewkrug 5845d59dbb
ensure mozdef always polls the SQS queue we create 2018-11-21 06:55:46 -08:00
andrewkrug 893c44c3fa
add container for generic SQS ingest 2018-11-21 05:44:57 -08:00
Brandon Myers 7016eff491
Modify tester container to copy all python directories for pep8 2018-11-15 13:11:46 -06:00
Brandon Myers 0fb5d2670f
Merge pull request #960 from mozilla/update_es_version 
Update ES version in docker to 5.6.8
 2018-11-14 11:49:10 -06:00
Brandon Myers 8ed5badc0e
Update loginput and restapi configs for tests 2018-11-13 12:24:04 -06:00
Brandon Myers 5e56983cc9
Update ES version in docker to 5.6.8 2018-11-13 11:29:59 -06:00
Jeff Bryner bcc223b127 add a sample data override for one off runs 2018-11-11 16:57:35 -08:00
Jeff Bryner c8758d752d bump version 2018-11-11 15:45:20 -08:00
Brandon Myers 238226342b
Merge pull request #953 from mozilla/remove_unused_config 
Remove references to enableBlockIP config key
 2018-11-09 16:15:28 -06:00
Brandon Myers b27537fd3a
Remove references to enableBlockIP config key 2018-11-09 15:11:21 -06:00
Brandon Myers a0eac2066f
Remove hash from kibanaurl 2018-11-09 12:47:46 -06:00
Jeff Bryner f5662d1a1a remove old settings.js 2018-11-07 10:50:49 -08:00
Jeff Bryner cb9d9a3714 remove deprecated setting 2018-11-06 13:00:53 -08:00
Jeff Bryner c0290344b7 up version for docker compose 2018-11-02 15:14:21 -07:00
Brandon Myers 747b766a07
Merge pull request #910 from mozilla/fix_some_pep8_errors 
Fix some pep8 errors
 2018-10-31 14:25:26 -05:00
Guillaume Destuynder bbb9ab7aeb
use bash instead of curl to check for container being fully-up 
include rabbitmq in the list of such tests
 2018-10-30 16:24:15 -07:00
Brandon Myers 5317839efc
Remove unneeded new line in setup script 2018-10-30 17:42:40 -05:00
Guillaume Destuynder a0557a3865
add a curl command to wait for elasticsearch to be fully up on docker 
compose services which also list elasticsearch as a dependency

This is used instead of wait-for-it to avoid pulling in more deps in
mozdef_base

NOTE: A better way would be to ensure all programs retry when ES is
down, as it would make everything more reliable (and also faster to
boot)
 2018-10-30 14:20:54 -07:00
Guillaume Destuynder bb1175e8d1
upgrade travis docker-compose as per https://docs.travis-ci.com/user/docker/ 2018-10-30 13:53:07 -07:00
Guillaume Destuynder 67c92b592e
port build system to docker compose 3.7 
use dependency-based builds
default to GITHASH=latest instead of the opposite
rename mq_event-task to mq_worker where it wasn't
fix dockerfiles to import from mozdef/* by default
introduce `make BUILD_MODE=pull` to pull images (instead of build)
 2018-10-30 13:01:47 -07:00
Brandon Myers cb096caa7b
Add mq_worker to build docker-compose 2018-10-29 14:26:20 -05:00
Brandon Myers cf3adae759
Merge pull request #875 from mozilla/rename_mq_worker_container 
Rename generic mq_eventtask container to worker
 2018-10-29 11:53:02 -05:00
Brandon Myers 82740d3446
Remove unused supervisor config 2018-10-25 20:19:34 -05:00
Brandon Myers 56b9ef9627
Rename generic mq_eventtask container to worker 2018-10-24 15:45:03 -05:00
Brandon Myers b729f05e70
Merge pull request #868 from gene1wood/add-cloudtrail-docker-service 
Add mq_cloudtrail docker service
 2018-10-24 15:32:41 -05:00
Brandon Myers a5b4970fc3
Merge remote-tracking branch 'origin/master' into fixup_merge_conflicts 2018-10-24 14:08:01 -05:00
Brandon Myers 00b30f9fd5
Move bootstrap initial setup script to container folder 2018-10-24 13:43:01 -05:00
Brandon Myers 281df357ec
Move mozdef_cron config files into container folder 2018-10-24 13:38:11 -05:00
Brandon Myers 96a08daa9b
Move elasticsearch config files into docker container folder 2018-10-24 13:37:02 -05:00
Brandon Myers 23d67d5448
Move kibana settings file inside container folder 2018-10-24 13:32:37 -05:00
Brandon Myers 091511424e
Move rabbitmq config files inside container folder 2018-10-24 13:31:30 -05:00
Brandon Myers 7bd192c408
Merge pull request #769 from mozilla/nginx_docker_proxy_pass_patch 
docker nginx proxy pass fix
 2018-10-24 13:18:00 -05:00
Brandon Myers 663fd76ab2
Merge remote-tracking branch 'origin/infosec_workweek' into virtualenv_path_change 2018-10-24 13:05:30 -05:00
Gene Wood 95b71593ff
Add mq_cloudtrail docker service 2018-10-23 18:08:01 -07:00
andrewkrug 209e292bd8
fix nits 2018-10-23 10:31:27 -07:00
andrewkrug 882259d5cc
fix mozdef base to be first container built 2018-10-22 15:12:14 -07:00
andrewkrug 11fd235672
Merge branch 'infosec_workweek' of https://github.com/mozilla/mozdef into config_system 2018-10-19 13:08:46 -07:00
andrewkrug 1026e85740
fix rabbit integration 2018-10-19 13:08:39 -07:00
Gene Wood 8180017049
Set mongo to use host mount and in turn EFS 2018-10-19 10:06:18 -07:00
andrewkrug af6a19dbe5
remove amqp due to protocol conflict 2018-10-19 09:53:07 -07:00
andrewkrug dbfdab92f2
update docker compose to include envs in mq-eventtask 2018-10-19 08:42:12 -07:00
Guillaume Destuynder 43189ee2b9
no sleeping 90s 2018-10-19 07:14:40 -07:00
Guillaume Destuynder acbdce2bcc
correctly point kibana 2018-10-18 12:32:18 -07:00
Guillaume Destuynder 03a4d26834
link to meteor container 2018-10-18 11:15:51 -07:00
andrewkrug 3429ba23b4
add docker volume for rabbit mq state 2018-10-18 10:56:19 -07:00
andrewkrug c0097d8eb0
remove elasticsearch depends from mq_eventtask 2018-10-18 10:55:01 -07:00
andrewkrug 8413b49f02
add image refs to cloudy mozdef target 2018-10-18 10:53:14 -07:00
andrewkrug aef63a191c
add event task to cloud target 2018-10-18 10:51:03 -07:00
andrewkrug 67bbe6af30
rollback change to compose file 2018-10-18 09:42:45 -07:00
Andrew J Krug 77d9ed6878
Merge pull request #815 from andrewkrug/config_system 
update compose to provide es_url to bootstrap container
 2018-10-18 09:15:22 -07:00
andrewkrug 96b36b0509
update compose to provide es_url to bootstrap container 2018-10-18 09:14:00 -07:00
Guillaume Destuynder 8d6cc002fa
remove stray env file 2018-10-18 09:00:21 -07:00
Guillaume Destuynder 37fdb0aab2
"multiple fixes" 
- makefile whitelist what to upload to s3
- use a kibana reverse proxy
- use an additional env file for the proxy
- expose new ports
- other things?
 2018-10-18 08:58:28 -07:00
Guillaume Destuynder bc0ba6e17b
dont override the docker/compose/cloudy_mozdef.env by making a local 
file instead and touching the current one
 2018-10-18 08:43:48 -07:00
Guillaume Destuynder 0b78dbea58
support new httpsredir=no option for the access proxy 
fix tester pull
force pull new dkr images
 2018-10-18 08:21:27 -07:00
Guillaume Destuynder 0d678526b7
swap nginx for oidc access proxy entierely 
auto pull  cloudmozdef on make
 2018-10-17 16:45:40 -07:00
Guillaume Destuynder 39906a3c62
add accessproxy 2018-10-17 16:21:02 -07:00
andrewkrug e0e93f3710
move configlib back to warehouse 2018-10-17 16:12:02 -07:00
Andrew J Krug e95598934d
Merge pull request #810 from andrewkrug/config_system 
bump configlib over to 2.0.1 from andrewkrug
 2018-10-17 15:44:46 -07:00
andrewkrug 750531b5d5
bump configlib over to 2.0.1 from andrewkrug 2018-10-17 15:42:59 -07:00
Guillaume Destuynder 5b71c82ea4
only use images for cloudy mozdef 2018-10-17 15:39:49 -07:00
Guillaume Destuynder 1e1278ed94
Change all files to be compose 2.2 compliant (its either that or all 3.0 
but certain features are them docker-swarm only)

Change the rebuild/norebuild mechanism to use a parameter instead of
different targets for simplicity
 2018-10-17 14:33:13 -07:00
Andrew J Krug a067c33df6
Merge pull request #808 from andrewkrug/config_system 
add configlib2.0.0 to requirements.txt
 2018-10-17 13:41:07 -07:00
andrewkrug 4ed9ee146b
add configlib2.0.0 to requirements.txt 2018-10-17 13:40:03 -07:00
Guillaume Destuynder c1f62b9e27
use gene's env var for the ES URL 2018-10-17 13:32:35 -07:00
Phrozyn 96e766cc48
Fixing typo in cloudy mozsded yml. 2018-10-17 12:58:27 -07:00
Andrew J Krug 4edd3e9b12
Merge pull request #801 from andrewkrug/config_system 
fix base Dockerfile
 2018-10-17 11:23:55 -07:00
andrewkrug a9098abee9
fix base Dockerfile 2018-10-17 11:22:45 -07:00
Guillaume Destuynder 5fd6df39a7
install git.. 2018-10-17 10:39:38 -07:00
Guillaume Destuynder e503b6f001
fix missing \ 2018-10-17 10:37:04 -07:00
Andrew J Krug d6b5be6512
Merge pull request #798 from andrewkrug/config_system 
temporarily pin to configlib abstraction
 2018-10-17 10:22:25 -07:00
andrewkrug 2da832d9c0
temporarily pin to configlib abstraction 2018-10-17 10:19:36 -07:00
Guillaume Destuynder 80fb5056a0
add run-cloudy-mozdef target and related files 2018-10-17 09:39:19 -07:00