* Remove referenced_tables usages
* Resolve check dependencies when generating DAGs
* Add tests for checks automatically resolving dependencies
* Incorporate feedback for removing referenced_tables
* Use depends_on for empty_checks
* More depends_on and fixes
* added value_length check and updated some of the ETL checks to use the macro
* added the new check macro to the data checks docs
* implemented lelilia feedback from PR#4672
* simplified the sql logic for the value_length check
* Put assert UDFs in `mozfun` project.
* Tweak syntax in `assert.array_equals()` to avoid SQLGlot parsing error.
https://github.com/tobymao/sqlglot/issues/2348
* Fix SQL syntax error in `assert.struct_equals()` tests.
* Fix UDF dependency file path logic when deploying to stage.
* Change regular expressions in `parse_routine` module to allow quotes around routines' dataset and name.
* DENG-1314 Implement changes to bqetl and create default DAG.
* DENG-1314. Update Documentation.
* DENG-1314. Dummy query to enable generating DAG and run tests.
* DENG-1314. Update tests.
* Update bigquery_etl/cli/query.py
Raise exception when scheduling information is missing.
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* DENG-1314. Update tests.
* DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests.
* Default task and tests update.
* Default task and tests update.
* 3650 - Remove default DAG option, update DAG template comment & tests.
* 3650 - Condition for DAG warning.
* 3650 - Update docs.
* Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update docs/cookbooks/creating_a_derived_dataset.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Small tweaks made to the cli cmds comments / help display for data checks
* added usage docs to data_checks reference docs
* Apply suggestions from code review provided by scholtzan
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added reference doc for data checks and a cookbook for adding new data checks
* Apply suggestions from code review by ascholtz
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Revert "CI fixes for supporting private UDFs in bigquery-etl - DENG-735 (#3631)"
This reverts commit edcfe758f7.
* Added stub UDF for monetized_search
* Add docs for using a private internal UDF
* Fixed typo in airflow_tags.md
As Sean pointed out, the tag provided in the docs was incorrect. This change fixes that.
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Use `mdx_truly_sane_lists` to fix nested list formatting in the docs (by default, mkdocs requires nested lists to be indented by four spaces, which isn't how most nested lists in our docs have been formatted).
* Enable syntax highlighting for code blocks in the docs.
* Add support for GitHub Flavored Markdown task lists in the docs.
Leli
Winnie Chan
Frank Bertsch
Alexander
Anna Scholtz
kik-kik
Sean Rose
Mike Williams
Jan-Erik Rediger
Marlene Hirose
Alekhya
Lucia
Curtis Morales
Daniel Thorn
akkomar