b0387fb7de
* Remove referenced_tables usages * Resolve check dependencies when generating DAGs * Add tests for checks automatically resolving dependencies * Incorporate feedback for removing referenced_tables * Use depends_on for empty_checks * More depends_on and fixes |
||
|---|---|---|
| .. | ||
| active_users | ||
| active_users_deletion_requests | ||
| country_code_lookup | ||
| derived_view_schemas | ||
| events_daily | ||
| experiment_monitoring | ||
| feature_usage | ||
| funnels | ||
| glean_usage | ||
| search | ||
| serp_events | ||
| stable_views | ||
| urlbar_events | ||
| use_counters | ||
| README.md | ||
| __init__.py | ||
README.md
SQL generators
sql_generators/ contains scripts for generating SQL queries. Generated SQL query code should not be checked in to main. The scripts for generating SQL queries are executed by CI only and will are followed by generating Airflow DAGs.
Expected structure
The directories in sql_generators/ represent the generated queries and will contain all of the scripts and templates necessary to generate these queries. Each query-specific directory will contain a __init__.py file that contains the query generation logic. Optionally, a templates/ directory can be added which contains the Jinja templates queries are generated from.
Each __init__.py file needs to implement a generate() method that is configured as a click command. The bqetl CLI will automatically add these commands to the ./bqetl query generate command group.
After changes to a schema or adding new tables, the schema is automatically derived from the query and deployed the next day in DAG bqetl_artifact_deployment. Alternatively, it can be manually generated and deployed using ./bqetl generate all and ./bqetl query schema deploy.