mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
Bigquery ETL
4 041 коммит 318 Ветки 1 286 Теги 222 MiB
Python 94.9%
Shell 2.4%
Jinja 1.8%
JavaScript 0.7%
Перейти к файлу
dependabot[bot] e2cc3d1d31
Bump mozilla-metric-config-parser from 2023.5.1 to 2023.6.1 (#3905) 
Bumps [mozilla-metric-config-parser](https://github.com/mozilla/metric-config-parser) from 2023.5.1 to 2023.6.1.
- [Release notes](https://github.com/mozilla/metric-config-parser/releases)
- [Commits](https://github.com/mozilla/metric-config-parser/compare/2023.5.1...2023.6.1)

---
updated-dependencies:
- dependency-name: mozilla-metric-config-parser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		2023-06-06 15:38:12 +00:00
.circleci Fix destination paths when copying stage files in CI (#3895) 2023-06-02 19:06:01 +00:00
.github remove maven from dependabot 2023-05-17 16:33:00 -07:00
.vscode Add Visual Studio Code debug configuration for current Python file. (#3457) 2022-12-13 13:07:27 -08:00
bigquery_etl feat(): added support for `--log-level` to bqetl query command and using logging instead of print() (#3891) 2023-06-06 09:16:41 +00:00
dags Sync currency exchange rates to support VPN wave 6 (DS-2907) (#3897) 2023-06-02 21:38:28 +00:00
docs Deng 1000 docs how to change control (#3903) 2023-06-05 15:53:16 +00:00
script added a debug message and flag for when authenticating to gcloud (#3602) 2023-02-23 18:43:24 +01:00
sql Remove ZetaSQL kludges. (#3898) 2023-06-05 18:03:07 +00:00
sql_generators Remove ZetaSQL kludges. (#3898) 2023-06-05 18:03:07 +00:00
tests GROWTH-41 Add last_seen columns to firefox_android_clients (#3863) 2023-06-05 16:37:59 +00:00
.bigqueryrc Create ~/.bigqueryrc without GCLOUD_SERVICE_KEY (#112) 2019-05-01 13:38:31 -07:00
.dockerignore Use zetasql to get dependencies for dag generation (#1817) 2021-02-18 17:49:46 -05:00
.eslintrc.yml Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) 2020-10-29 17:24:55 -07:00
.flake8 Add and incrementally populate a table for google ads campaign cost metrics (#3468) 2023-01-11 15:58:10 -06:00
.gitignore feat(): Adding ETL for monitoring_airflow datasets (#3204) 2022-10-12 10:57:24 +01:00
.isort.cfg Automatically sort python imports (#1840) 2021-02-24 17:11:52 -05:00
.pre-commit-config.yaml Update pre-commit plugins to match `requirements.txt`. (#3737) 2023-04-26 08:31:48 -07:00
.yamllint.yaml Increase yamllint max line length (#3469) 2022-12-16 12:23:32 -08:00
CODEOWNERS Add change control to the request payload tables (#3833) 2023-05-26 10:37:22 -04:00
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md (#50) 2019-03-30 10:01:54 -07:00
CONTRIBUTING.md Add CODEOWNERS and restricted access datasets documentation (#2524) 2021-12-21 20:25:05 +00:00
Dockerfile Remove ZetaSQL kludges. (#3898) 2023-06-05 18:03:07 +00:00
GRAVEYARD.md Remove asn_aggregates ETL (#2580) 2021-12-15 18:28:29 +00:00
LICENSE Add MPL license (#2353) 2021-09-20 17:59:47 +00:00
M1_MAC_SETUP.md Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
README.md Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
bqetl Update grpcio to a version that provides darwin arm64 builds (#3819) 2023-05-22 12:08:49 -07:00
conftest.py Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
dags.yaml reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) 2023-05-31 11:49:11 +02:00
pytest.ini Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
requirements.in Bump mozilla-metric-config-parser from 2023.5.1 to 2023.6.1 (#3905) 2023-06-06 15:38:12 +00:00
requirements.txt Bump mozilla-metric-config-parser from 2023.5.1 to 2023.6.1 (#3905) 2023-06-06 15:38:12 +00:00
setup.py Move to python 3.10 (#3236) 2022-09-22 15:39:40 -04:00

README.md

CircleCI

BigQuery ETL

This repository contains Mozilla Data Team's:

  • Derived ETL jobs that do not require a custom container
  • User-defined functions (UDFs)
  • Airflow DAGs for scheduled bigquery-etl queries
  • Tools for query & UDF deployment, management and scheduling

For more information, see https://mozilla.github.io/bigquery-etl/

Quick Start

❗ Apple Silicon (M1) user requirement

Enable Rosetta mode for your terminal BEFORE installing below tools using your terminal. See our M1 Mac Setup Guide for more information

Pre-requisites

  • Pyenv (optional) Recommended if you want to install different versions of python, see instructions here. After the installation of pyenv, make sure that your terminal app is configured to run the shell as a login shell.
  • Homebrew (not required, but useful for Mac) - Follow the instructions here to install homebrew on your Mac.
  • Python 3.10+ - (see this guide for instructions if you're on a mac and haven't installed anything other than the default system Python).

GCP CLI tools

  • For Mozilla Employees or Contributors (not in Data Engineering) - Set up GCP command line tools, as described on docs.telemetry.mozilla.org. Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed.
  • For Data Engineering - In addition to setting up the command line tools, you will want to log in to shared-prod if making changes to production systems. Run gcloud auth login --update-adc --project=moz-fx-data-shared-prod (if you have not run it previously).

Installing bqetl

  1. Clone the repository
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
  1. Install the bqetl command line tool
./bqetl bootstrap
  1. Install standard pre-commit hooks
venv/bin/pre-commit install

Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:

cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json

And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl. You may also want to read up on common workflows.