9b5c04a7bb
* Bug 1741487 - Rename url2 and related fields in stable views This removes the following unpopulated fields from Glean views: `metrics.url`, `metrics.text`, `metrics.jwe`, and `metrics.labeled_rate`. If any of these metrics exist in the source table under `2`-suffixed name, it is also aliased to its original name (`url2` to `url` and so on). Suffixed fields are still preserved until view consumers migrate. * Remove redundant comma from generated sql * Ignore missing fields in views if any of them were removed * added a todo comment * Added additional context around why we are excluding some of the non-suffixed fields and why alising to remove suffix 2 from some fields --------- Co-authored-by: Arkadiusz Komarzewski <akomarzewski@mozilla.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net> |
||
|---|---|---|
| .. | ||
| active_users | ||
| country_code_lookup | ||
| derived_view_schemas | ||
| events_daily | ||
| experiment_monitoring | ||
| feature_usage | ||
| glean_usage | ||
| search | ||
| stable_views | ||
| README.md | ||
README.md
SQL generators
sql_generators/ contains scripts for generating SQL queries. Generated SQL query code should not be checked in to main. The scripts for generating SQL queries are executed by CI only and will are followed by generating Airflow DAGs.
Expected structure
The directories in sql_generators/ represent the generated queries and will contain all of the scripts and templates necessary to generate these queries. Each query-specific directory will contain a __init__.py file that contains the query generation logic. Optionally, a templates/ directory can be added which contains the Jinja templates queries are generated from.
Each __init__.py file needs to implement a generate() method that is configured as a click command. The bqetl CLI will automatically add these commands to the ./bqetl query generate command group.
After changes to a schema or adding new tables, the schema is automatically derived from the query and deployed the next day in DAG bqetl_artifact_deployment. Alternatively, it can be manually generated and deployed using ./bqetl generate all and ./bqetl query schema deploy.