Spark bindings for Mozilla Telemetry
Перейти к файлу
kik-kik f579e95fd9
# bug(1854406): Addressing installation problems affecting consumers of this repo. (#277)
* updated setup.py and explitly specifying dependencies versions + removed user_scm_version flag

* Tweaked CI configuration to comment out any potential publishing steps

* trying to fix CI build

* fixing linting error

* Tweaked CI configuration to comment out any potential publishing steps
2023-11-09 11:04:15 +01:00
.circleci # bug(1854406): Addressing installation problems affecting consumers of this repo. (#277) 2023-11-09 11:04:15 +01:00
bin Use a named docker volume for tox cache 2018-09-14 10:13:40 -04:00
docs Final deprecation/archive docs (#275) 2019-12-27 13:08:28 -05:00
moztelemetry # bug(1854406): Addressing installation problems affecting consumers of this repo. (#277) 2023-11-09 11:04:15 +01:00
tests Bug 1429902 - Dataset.dataframe method (#227) 2018-08-22 13:20:43 -07:00
.dockerignore Use a named docker volume for tox cache 2018-09-14 10:13:40 -04:00
.gitignore Bug 1477808 Move from TravisCI to CircleCI 2.0 2018-08-17 14:19:28 -04:00
Dockerfile Copy source into image rather than mount in container 2018-08-17 14:19:28 -04:00
LICENSE add license and how to install 2016-06-03 16:21:06 -07:00
README.md Final deprecation/archive docs (#275) 2019-12-27 13:08:28 -05:00
setup.cfg Update docstring to clarify SparkContext vs. SparkSession (#236) 2018-10-31 10:01:42 -03:00
setup.py # bug(1854406): Addressing installation problems affecting consumers of this repo. (#277) 2023-11-09 11:04:15 +01:00
tox.ini Get test suite working with latest moto 2018-11-06 09:18:57 -05:00

README.md

python_moztelemetry CircleCI Build Status Updates codecov.io

PySpark bindings for Mozilla Telemetry

The underlying data no longer exists, so this repository is archived. See the fx-data-dev mailing list posting for more information.

Documentation

Online documentation is available at https://mozilla.github.io/python_moztelemetry/

Deploying a code change

After having your PR reviewed and merged create a new release on github. A new pypi release will be automatically triggered by Travis.

Installing from pypi

  • To install this package from pypi run:
pip install python_moztelemetry

Updating parse_histograms.py

moztelemetry/parse_histograms.py is a mirror of its counterpart from mozilla-central. To update it to the latest version you can run

bin/update_parser_scripts

Note that this file was formerly called histogram_tools.py and was renamed in Bug 1419761.

Updating message_pb2.py

moztelemetry/heka/message_pb2.py is generated from mozilla-services/heka. To regenerate it, you'll need to install a protobuf package for your system. To avoid installing go-specific extensions, remove the gogo.proto import and the gogoproto options and then run protoc:

git clone https://github.com/mozilla-services/heka
mkdir pythonfiles/
protoc -I heka/message --python_out pythonfiles/ heka/message/message.proto

Testing locally

To test/debug this package locally, you can run exactly the job that CircleCI runs for continuous integration by installing the CircleCI local CLI and invoking:

circleci build --job py36

See [.circleci/config.yml] for the other configured job names (for running tests on different python versions).

The above process takes a few minutes to run every time, so there is also a bin/test script that builds a docker image and python environment (both of which are cached locally) and allows you to run a subset of tests. Here are some sample invocations:

./bin/test tests/ -k test_unpack  # runs only tests with key "test_unpack"
./bin/test tests/heka/            # runs only tests in tests/heka
PYTHON_VERSION=2.7 ./bin/test     # specify a python version

It's also possible to run the tests locally outside of docker by invoking tox directly, but the details of doing so depend on your local development environment and are outside the scope of these docs. Be aware that you will need to have a working installation of Java and libsnappy, likely via your OS's package manager (i.e. brew install snappy on MacOS).

If you're receiving mysterious errors, try removing cached files via:

./bin/clean