mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
Bigquery ETL
5 334 коммитов 321 ветка 1 298 Теги 223 MiB
Python 94.9%
Shell 2.4%
Jinja 1.8%
JavaScript 0.7%
Перейти к файлу
Chelsey Beck 13d177532e
fixing snapshot schema (#5507) 		2024-05-07 08:36:24 -07:00
.circleci Store debug artifacs when CI jobs downstream of SQL generation fail. (#5489) 2024-05-03 09:58:11 -07:00
.github Set dependabot reviewers to wg team (#4382) 2023-10-05 12:16:43 -07:00
.vscode Add Visual Studio Code debug configuration for current Python file. (#3457) 2022-12-13 13:07:27 -08:00
bigquery_etl Revert "Deploy referenced views to stage (#5497)" (#5510) 2024-05-07 07:53:47 -07:00
docs docs: DENG-809 add instructions for managed backfills (#5377) 2024-04-17 13:13:53 -04:00
script bug 1890351: Move GLAM ETL legacy to glam-prod project 2024-05-06 12:14:08 -04:00
sql fixing snapshot schema (#5507) 2024-05-07 08:36:24 -07:00
sql_generators Merge login funnels into one config file (#5495) 2024-05-06 09:40:51 -07:00
tests feat: add meta_attribution_app field to firefox_android_clients (#5404) 2024-05-07 17:09:06 +02:00
.bigqueryrc
…
.dockerignore
…
.eslintrc.yml
…
.flake8 Add and incrementally populate a table for google ads campaign cost metrics (#3468) 2023-01-11 15:58:10 -06:00
.gitignore feat(): Adding ETL for monitoring_airflow datasets (#3204) 2022-10-12 10:57:24 +01:00
.isort.cfg
…
.pre-commit-config.yaml feat(): Upgrade BQETL Python to 3.11 and pre-commit hooks (#5195) 2024-03-14 14:25:30 +01:00
.yamllint.yaml Increase yamllint max line length (#3469) 2022-12-16 12:23:32 -08:00
CODEOWNERS Fix typo in `CODEOWNERS` path for `search_derived.search_revenue_levers_daily_v1`. (#4837) 2024-01-16 15:05:42 -08:00
CODE_OF_CONDUCT.md
…
CONTRIBUTING.md Add CODEOWNERS and restricted access datasets documentation (#2524) 2021-12-21 20:25:05 +00:00
Dockerfile feat(): Upgrade BQETL Python to 3.11 and pre-commit hooks (#5195) 2024-03-14 14:25:30 +01:00
GRAVEYARD.md Remove asn_aggregates ETL (#2580) 2021-12-15 18:28:29 +00:00
LICENSE Add MPL license (#2353) 2021-09-20 17:59:47 +00:00
README.md feat(): Upgrade BQETL Python to 3.11 and pre-commit hooks (#5195) 2024-03-14 14:25:30 +01:00
bqetl Update grpcio to a version that provides darwin arm64 builds (#3819) 2023-05-22 12:08:49 -07:00
bqetl_project.yaml DENG-2262 Update desktop installs to no longer rely on a GA3 table (#5444) 2024-04-26 11:36:40 -05:00
conftest.py Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
dags.yaml Assign kwindau as owner of bqetl_google_analytics (#5492) 2024-05-03 14:47:41 -05:00
pyproject.toml feat(): Upgrade BQETL Python to 3.11 and pre-commit hooks (#5195) 2024-03-14 14:25:30 +01:00
pytest.ini Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
requirements.in Bump types-python-dateutil from 2.8.19.20240311 to 2.9.0.20240316 (#5221) 2024-03-18 08:52:11 -07:00
requirements.txt Bump pydantic from 1.10.11 to 1.10.13 (#5432) 2024-05-01 08:54:54 -07:00

README.md

CircleCI

BigQuery ETL

This repository contains Mozilla Data Team's:

  • Derived ETL jobs that do not require a custom container
  • User-defined functions (UDFs)
  • Airflow DAGs for scheduled bigquery-etl queries
  • Tools for query & UDF deployment, management and scheduling

For more information, see https://mozilla.github.io/bigquery-etl/

Quick Start

Pre-requisites

  • Pyenv (optional) Recommended if you want to install different versions of python, see instructions here. After the installation of pyenv, make sure that your terminal app is configured to run the shell as a login shell.
  • Homebrew (not required, but useful for Mac) - Follow the instructions here to install homebrew on your Mac.
  • Python 3.11+ - (see this guide for instructions if you're on a mac and haven't installed anything other than the default system Python).

GCP CLI tools

  • For Mozilla Employees (not in Data Engineering) - Set up GCP command line tools, as described on docs.telemetry.mozilla.org. Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed. Run gcloud auth login --update-adc to authenticate against GCP.
  • For Data Engineering - In addition to setting up the command line tools, you will want to log in to shared-prod if making changes to production systems. Run gcloud auth login --update-adc --project=moz-fx-data-shared-prod (if you have not run it previously).

Installing bqetl

  1. Clone the repository
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
  1. Install the bqetl command line tool
./bqetl bootstrap
  1. Install standard pre-commit hooks
venv/bin/pre-commit install

Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:

cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json

And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl. You may also want to read up on common workflows.