Bigquery ETL

Перейти к файлу

kik-kik 8a94713853 feat: update the ETL to be incremental as not all report dates contain all combinations. Also add user facing view for accessing this data		2024-11-20 16:34:30 +00:00
.circleci	Populate empty query.py in stage deploy instead of CI config (#6430 )	2024-10-31 20:50:22 +00:00
.github	Fix PR checklist link (#6340 )	2024-10-11 18:44:55 +00:00
.vscode	Add Visual Studio Code debug configuration for current Python file. (#3457 )	2022-12-13 13:07:27 -08:00
bigquery_etl	Improve checks to account for null replacements (#6508 )	2024-11-15 17:03:02 +00:00
docs	remove backtick (#6486 )	2024-11-14 15:59:51 +00:00
script	Add shell completion script for bqetl (#6372 )	2024-10-21 15:18:16 +00:00
sql	feat: update the ETL to be incremental as not all report dates contain all combinations. Also add user facing view for accessing this data	2024-11-20 16:34:30 +00:00
sql_generators	Remove the scheduler for v4, as rthe tables need to be moved to production via a backfill with shredder mitigation. It will be added after the tables are moved to production. (#6530 )	2024-11-19 19:41:19 +00:00
tests	Support results with no rows in matches_pattern check (#6521 )	2024-11-19 18:13:07 +00:00
.bigqueryrc	Create ~/.bigqueryrc without GCLOUD_SERVICE_KEY (#112 )	2019-05-01 13:38:31 -07:00
.dockerignore	Use zetasql to get dependencies for dag generation (#1817 )	2021-02-18 17:49:46 -05:00
.eslintrc.yml	Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297 ) (#1481 )	2020-10-29 17:24:55 -07:00
.flake8	Add and incrementally populate a table for google ads campaign cost metrics (#3468 )	2023-01-11 15:58:10 -06:00
.gitignore	Add CLI commands to deploy BigConfig files (#6169 )	2024-09-12 20:16:35 +00:00
.isort.cfg	Automatically sort python imports (#1840 )	2021-02-24 17:11:52 -05:00
.pre-commit-config.yaml	fix: Upgrade pre-commit dependencies to match `requirements.in`. (#6271 )	2024-09-27 23:53:03 +00:00
.yamllint.yaml	Increase yamllint max line length (#3469 )	2022-12-16 12:23:32 -08:00
CODEOWNERS	Temporarily add active_users v3 to codeowners until v4 backfill is completed. (#6534 )	2024-11-20 13:14:18 +00:00
CODE_OF_CONDUCT.md	Create CODE_OF_CONDUCT.md (#50 )	2019-03-30 10:01:54 -07:00
CONTRIBUTING.md	Add CODEOWNERS and restricted access datasets documentation (#2524 )	2021-12-21 20:25:05 +00:00
Dockerfile	fix(glam): add a more robust way to fetch glean tables (#5787 )	2024-06-13 20:38:41 -04:00
GRAVEYARD.md	Remove asn_aggregates ETL (#2580 )	2021-12-15 18:28:29 +00:00
LICENSE	Add MPL license (#2353 )	2021-09-20 17:59:47 +00:00
README.md	Fix PyPI package publishing in CI (#5669 )	2024-05-28 14:03:06 -07:00
bigeye_credentials.ini	Add CLI commands to deploy BigConfig files (#6169 )	2024-09-12 20:16:35 +00:00
bqetl	Add CLI commands to deploy BigConfig files (#6169 )	2024-09-12 20:16:35 +00:00
bqetl_project.yaml	Remove row count check for net_thunderbird_android_derived.baseline_clients_last_seen_v1 (#6515 )	2024-11-18 20:13:02 +00:00
conftest.py	Remove java dependency in favor of sqlglot (#3755 )	2023-05-17 14:56:42 -07:00
dags.yaml	feat(DENG-5757): Create new table monitoring_derived.metadata_completeness_v1 (#6397 )	2024-10-25 14:08:37 +00:00
pyproject.toml	Fix PyPI package publishing in CI (#5669 )	2024-05-28 14:03:06 -07:00
pytest.ini	Add bigquery-etl-integration-test to testpath (#5976 )	2024-07-30 10:16:18 -07:00
requirements.in	update dependencies	2024-11-20 16:33:09 +00:00
requirements.txt	update dependencies	2024-11-20 16:33:09 +00:00

README.md

BigQuery ETL

This repository contains Mozilla Data Team's:

Derived ETL jobs that do not require a custom container
User-defined functions (UDFs)
Airflow DAGs for scheduled bigquery-etl queries
Tools for query & UDF deployment, management and scheduling

For more information, see https://mozilla.github.io/bigquery-etl/

Quick Start

Pre-requisites

Pyenv (optional) Recommended if you want to install different versions of python, see instructions here. After the installation of pyenv, make sure that your terminal app is configured to run the shell as a login shell.
Homebrew (not required, but useful for Mac) - Follow the instructions here to install homebrew on your Mac.
Python 3.11+ - (see this guide for instructions if you're on a mac and haven't installed anything other than the default system Python).

GCP CLI tools

For Mozilla Employees (not in Data Engineering) - Set up GCP command line tools, as described on docs.telemetry.mozilla.org. Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed. Run gcloud auth login --update-adc to authenticate against GCP.
For Data Engineering - In addition to setting up the command line tools, you will want to log in to shared-prod if making changes to production systems. Run gcloud auth login --update-adc --project=moz-fx-data-shared-prod (if you have not run it previously).

Installing bqetl

Clone the repository

git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl

Install the bqetl command line tool

./bqetl bootstrap

Install standard pre-commit hooks

venv/bin/pre-commit install

Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:

cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json

And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl. You may also want to read up on common workflows.

Releasing a new version of `bqetl`

To push a new version of bqetl to PyPI, update the version in pyproject.toml. The version numbers follow the CalVer scheme, with the Micro version numbers starting at 1. For example, for the first package version getting published in March 2024, the version would be 2024.3.1.