mozilla
/
telemetry-streaming
зеркало из https://github.com/mozilla/telemetry-streaming.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
Spark Streaming ETL jobs for Mozilla Telemetry
103 коммитов 67 Ветки 7 Теги 1,1 MiB
Scala 99.9%
Shell 0.1%
Перейти к файлу
Frank Bertsch 456fcb8250 Add Android Focus eventsToAmplitude config 2018-03-28 11:21:39 -05:00
configs Add Android Focus eventsToAmplitude config 2018-03-28 11:21:39 -05:00
docker Add Integration Tests with Docker 2017-09-18 14:36:48 -05:00
project Bump sbt version to 1.1.1 2018-03-20 16:03:42 +01:00
src Fix extra request attempt in HTTPSink 2018-03-27 15:25:54 -05:00
.gitignore Add .gitignore 2017-01-24 17:34:02 +00:00
.travis.yml Bug 1423340 - Make tests easily runnable from IDE 2018-03-20 23:53:09 +01:00
README.md Bug 1423340 - Make tests easily runnable from IDE 2018-03-20 23:53:09 +01:00
build.sbt Limit Kafka dependency scope 2018-03-20 16:03:42 +01:00
docker_setup.sh Bug 1423340 - Make tests easily runnable from IDE 2018-03-20 23:53:09 +01:00
scalastyle-config.xml Refactor tests to conform with Scalastyle rules, enable Scalastyle in test scope on CI 2018-03-20 14:49:05 +01:00

README.md

Build Status codecov.io

telemetry-streaming

Spark Streaming ETL jobs for Mozilla Telemetry

This service currently contains jobs that aggregate error data on 5 minute intervals. It is responsible for generating the (internal only) error_aggregates and experiment_error_aggregates parquet tables at Mozilla.

Issue Tracking

Please file bugs in the Datasets: Error Aggregates component.

Development

The recommended workflow for running tests is to use your favorite editor for editing the source code and running the tests via sbt. Some common invocations for sbt:

  • sbt test # run the basic set of tests (good enough for most purposes)
  • sbt "testOnly *ErrorAgg*" # run the tests only for packages matching ErrorAgg
  • sbt "testOnly *ErrorAgg* -- -z version" # run the tests only for packages matching ErrorAgg, limited to test cases with "version" in them
  • sbt dockerComposeTest # run the docker compose tests (slow)
  • sbt "dockerComposeTest -tags:DockerComposeTag" # run only tests with DockerComposeTag (while using docker)
  • sbt ci # run all tests

Some tests need Kafka to run. If one prefers to run them via IDE, it's required to run the test cluster:

sbt dockerComposeUp

or via plain docker-compose:

export DOCKER_KAFKA_HOST=$(./docker_setup.sh)
docker-compose -f docker/docker-compose.yml up

It's also good to shut down the cluster afterwards:

sbt dockerComposeStop