* added sql logic for monitoring_airflow
* bqetl_monitoring_airflow added to dags.yaml
* added .probe_cache/ to gitignore
* generated bqetl_monitoring_airflow dag
* added monitoring_airflow_derived to dryrun ignore as it fails to access fivetran data referenced in this dataset
* moved airflow views and sql files to monitoring dataset
* manually triggering fivetran load of airflow metadata as suggested by @ascholtzan in PR#3204
* added schemas as requested by @scholtzan in PR#3204
* added descriptions to airflow_monitoring datasets
* fixed airflow_dag view folder name
* corrected dryrun ignore for monitoring_derived/airflow*
* Ignore .idea/ folder in pycharm
* Make instructions to add field generic with example. Add recommendations to delete field.
* Reformat numbered list for consistency.
* Update instructions based on PR feedback.
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
.vscode's settings.json can get cluttered with installation-specific
settings, which can lead to confusing pull requests. Instead, let's use
the approach outlined here:
https://stackoverflow.com/a/48387809
This provides a set of defaults that people can use, which we can
extend over time.
* Add CI task to push content to generated-sql branch
Fixes#1742
The
[`generated-sql`](https://github.com/mozilla/bigquery-etl/tree/generated-sql)
branch now exists and you can browse the contents. See, for example,
[telemetry.main](https://github.com/mozilla/bigquery-etl/tree/generated-sql/sql/moz-fx-data-shared-prod/telemetry/main)
Follow-ups for which I'll file issues:
- This doesn't currently publish the generated Glean baseline ETL queries
and views; we'll need to update that logic to use probe-scraper metadata
rather than listing tables in BigQuery (due to creds) to integrate it.
- Docs publishing should reference this generated content rather
* Bug 1635906 Add bqetl support for scripts and script for AET lookup
There are some code changes here for DAG generation and for testing.
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* DAG fixups
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
kik-kik
Lucia
William Lachance
Daniel Thorn
Jeff Klukas
Linh Nguyen
Linh Nguyen
Marina Samuel
Anna Scholtz