94f60665d9
Turns out when deploying monitors to views in Bigeye `table_deployments` aren't supported. Instead we'll have to deploy a config similar to this:
```
type: BIGCONFIG_FILE
row_creation_times:
column_selectors:
- name: moz-fx-data-shared-prod.moz-fx-data-shared-prod.telemetry.releases.date
saved_metric_definitions:
metrics:
- saved_metric_id: FRESHNESS_DATA
metric_type:
type: PREDEFINED
predefined_metric: FRESHNESS_DATA
metric_name: Freshness (data)
threshold:
type: AUTO
sensitivity: MEDIUM
upper_bound_only: true
lower_bound_only: false
lookback:
lookback_window:
interval_type: DAYS
interval_value: 2
lookback_type: METRIC_TIME
bucket_size: HOUR
rct_overrides:
- date
- saved_metric_id: VOLUME_DATA
metric_type:
type: PREDEFINED
predefined_metric: VOLUME_DATA
metric_name: Volume (data)
threshold:
type: AUTO
sensitivity: MEDIUM
upper_bound_only: false
lower_bound_only: true
lookback:
lookback_window:
interval_type: DAYS
interval_value: 2
lookback_type: METRIC_TIME
bucket_size: HOUR
rct_overrides:
- date
tag_deployments:
- collection:
name: releases
description: SDK Generated
deployments:
- column_selectors:
- name: moz-fx-data-shared-prod.moz-fx-data-shared-prod.telemetry.releases.*
metrics:
- saved_metric_id: FRESHNESS_DATA
- saved_metric_id: VOLUME_DATA
```
Oddly enough, this was working fine last week...
Disabling monitoring for now to make Airflow happy. I'll look into this further tomorrow.
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| .vscode | ||
| bigquery_etl | ||
| docs | ||
| script | ||
| sql | ||
| sql_generators | ||
| tests | ||
| .bigqueryrc | ||
| .dockerignore | ||
| .eslintrc.yml | ||
| .flake8 | ||
| .gitignore | ||
| .isort.cfg | ||
| .pre-commit-config.yaml | ||
| .yamllint.yaml | ||
| CODEOWNERS | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| GRAVEYARD.md | ||
| LICENSE | ||
| README.md | ||
| bigeye_credentials.ini | ||
| bqetl | ||
| bqetl_project.yaml | ||
| conftest.py | ||
| dags.yaml | ||
| pyproject.toml | ||
| pytest.ini | ||
| requirements.in | ||
| requirements.txt | ||
README.md
BigQuery ETL
This repository contains Mozilla Data Team's:
- Derived ETL jobs that do not require a custom container
- User-defined functions (UDFs)
- Airflow DAGs for scheduled bigquery-etl queries
- Tools for query & UDF deployment, management and scheduling
For more information, see https://mozilla.github.io/bigquery-etl/
Quick Start
Pre-requisites
- Pyenv (optional) Recommended if you want to install different versions of python, see instructions here. After the installation of pyenv, make sure that your terminal app is configured to run the shell as a login shell.
- Homebrew (not required, but useful for Mac) - Follow the instructions here to install homebrew on your Mac.
- Python 3.11+ - (see this guide for instructions if you're on a mac and haven't installed anything other than the default system Python).
GCP CLI tools
- For Mozilla Employees (not in Data Engineering) - Set up GCP command line tools, as described on docs.telemetry.mozilla.org. Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed. Run
gcloud auth login --update-adcto authenticate against GCP. - For Data Engineering - In addition to setting up the command line tools, you will want to log in to
shared-prodif making changes to production systems. Rungcloud auth login --update-adc --project=moz-fx-data-shared-prod(if you have not run it previously).
Installing bqetl
- Clone the repository
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
- Install the
bqetlcommand line tool
./bqetl bootstrap
- Install standard pre-commit hooks
venv/bin/pre-commit install
Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:
cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json
And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl. You may also want to read up on common workflows.
Releasing a new version of bqetl
To push a new version of bqetl to PyPI, update the version in pyproject.toml. The version numbers follow the CalVer scheme, with the Micro version numbers starting at 1. For example, for the first package version getting published in March 2024, the version would be 2024.3.1.