mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
Bigquery ETL
6 075 коммитов 318 Ветки 1 286 Теги 222 MiB
Python 94.9%
Shell 2.4%
Jinja 1.8%
JavaScript 0.7%
Перейти к файлу
Mathijs Miermans 5e080716e0
[MC-1458] Add newtab_merino_priors DAG (#6303) 
* [MC-1458] Add newtab_merino_priors DAG

* Extract shared JSON export function

---------

Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>
 		2024-10-14 17:06:12 +00:00
.circleci Add more memory for private-generate-sql task (#6342) 2024-10-11 21:04:14 +00:00
.github Fix PR checklist link (#6340) 2024-10-11 18:44:55 +00:00
.vscode Add Visual Studio Code debug configuration for current Python file. (#3457) 2022-12-13 13:07:27 -08:00
bigquery_etl [MC-1458] Add newtab_merino_priors DAG (#6303) 2024-10-14 17:06:12 +00:00
docs Reference to the shredder mitigation process during backfills. (#6233) 2024-09-20 13:57:19 +00:00
script fix(GLAM): Fix boolean parameter passing to generate_glean_sql (#6155) 2024-09-04 11:41:34 -04:00
sql [MC-1458] Add newtab_merino_priors DAG (#6303) 2024-10-14 17:06:12 +00:00
sql_generators Add partitioning and clustering for event_monitoring materialized views (#6286) 2024-10-02 15:22:12 -07:00
tests fix: only generate Airflow task for BigEye if monitoring enabled in the metadata (#6326) 2024-10-10 21:50:42 +00:00
.bigqueryrc Create ~/.bigqueryrc without GCLOUD_SERVICE_KEY (#112) 2019-05-01 13:38:31 -07:00
.dockerignore Use zetasql to get dependencies for dag generation (#1817) 2021-02-18 17:49:46 -05:00
.eslintrc.yml Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) 2020-10-29 17:24:55 -07:00
.flake8 Add and incrementally populate a table for google ads campaign cost metrics (#3468) 2023-01-11 15:58:10 -06:00
.gitignore Add CLI commands to deploy BigConfig files (#6169) 2024-09-12 20:16:35 +00:00
.isort.cfg Automatically sort python imports (#1840) 2021-02-24 17:11:52 -05:00
.pre-commit-config.yaml fix: Upgrade pre-commit dependencies to match `requirements.in`. (#6271) 2024-09-27 23:53:03 +00:00
.yamllint.yaml Increase yamllint max line length (#3469) 2022-12-16 12:23:32 -08:00
CODEOWNERS Separate DAU metrics into `search_dau_aggregates` table (#6000) 2024-08-13 20:52:38 -04:00
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md (#50) 2019-03-30 10:01:54 -07:00
CONTRIBUTING.md Add CODEOWNERS and restricted access datasets documentation (#2524) 2021-12-21 20:25:05 +00:00
Dockerfile fix(glam): add a more robust way to fetch glean tables (#5787) 2024-06-13 20:38:41 -04:00
GRAVEYARD.md Remove asn_aggregates ETL (#2580) 2021-12-15 18:28:29 +00:00
LICENSE Add MPL license (#2353) 2021-09-20 17:59:47 +00:00
README.md Fix PyPI package publishing in CI (#5669) 2024-05-28 14:03:06 -07:00
bigeye_credentials.ini Add CLI commands to deploy BigConfig files (#6169) 2024-09-12 20:16:35 +00:00
bqetl Add CLI commands to deploy BigConfig files (#6169) 2024-09-12 20:16:35 +00:00
bqetl_project.yaml Skip generating org_mozilla_ios_focus_derived/baseline_clients_last_s… (#6320) 2024-10-09 16:09:03 +00:00
conftest.py Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
dags.yaml [MC-1458] Add newtab_merino_priors DAG (#6303) 2024-10-14 17:06:12 +00:00
pyproject.toml Fix PyPI package publishing in CI (#5669) 2024-05-28 14:03:06 -07:00
pytest.ini Add bigquery-etl-integration-test to testpath (#5976) 2024-07-30 10:16:18 -07:00
requirements.in Bump bigeye-sdk from 0.4.88 to 0.4.90 (#6332) 2024-10-11 16:12:21 +00:00
requirements.txt Bump bigeye-sdk from 0.4.88 to 0.4.90 (#6332) 2024-10-11 16:12:21 +00:00

README.md

CircleCI

BigQuery ETL

This repository contains Mozilla Data Team's:

  • Derived ETL jobs that do not require a custom container
  • User-defined functions (UDFs)
  • Airflow DAGs for scheduled bigquery-etl queries
  • Tools for query & UDF deployment, management and scheduling

For more information, see https://mozilla.github.io/bigquery-etl/

Quick Start

Pre-requisites

  • Pyenv (optional) Recommended if you want to install different versions of python, see instructions here. After the installation of pyenv, make sure that your terminal app is configured to run the shell as a login shell.
  • Homebrew (not required, but useful for Mac) - Follow the instructions here to install homebrew on your Mac.
  • Python 3.11+ - (see this guide for instructions if you're on a mac and haven't installed anything other than the default system Python).

GCP CLI tools

  • For Mozilla Employees (not in Data Engineering) - Set up GCP command line tools, as described on docs.telemetry.mozilla.org. Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed. Run gcloud auth login --update-adc to authenticate against GCP.
  • For Data Engineering - In addition to setting up the command line tools, you will want to log in to shared-prod if making changes to production systems. Run gcloud auth login --update-adc --project=moz-fx-data-shared-prod (if you have not run it previously).

Installing bqetl

  1. Clone the repository
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
  1. Install the bqetl command line tool
./bqetl bootstrap
  1. Install standard pre-commit hooks
venv/bin/pre-commit install

Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:

cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json

And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl. You may also want to read up on common workflows.

Releasing a new version of bqetl

To push a new version of bqetl to PyPI, update the version in pyproject.toml. The version numbers follow the CalVer scheme, with the Micro version numbers starting at 1. For example, for the first package version getting published in March 2024, the version would be 2024.3.1.