* Put assert UDFs in `mozfun` project.
* Tweak syntax in `assert.array_equals()` to avoid SQLGlot parsing error.
https://github.com/tobymao/sqlglot/issues/2348
* Fix SQL syntax error in `assert.struct_equals()` tests.
* Fix UDF dependency file path logic when deploying to stage.
* Change regular expressions in `parse_routine` module to allow quotes around routines' dataset and name.
* DENG-1314 Implement changes to bqetl and create default DAG.
* DENG-1314. Update Documentation.
* DENG-1314. Dummy query to enable generating DAG and run tests.
* DENG-1314. Update tests.
* Update bigquery_etl/cli/query.py
Raise exception when scheduling information is missing.
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* DENG-1314. Update tests.
* DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests.
* Default task and tests update.
* Default task and tests update.
* 3650 - Remove default DAG option, update DAG template comment & tests.
* 3650 - Condition for DAG warning.
* 3650 - Update docs.
* Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update docs/cookbooks/creating_a_derived_dataset.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Small tweaks made to the cli cmds comments / help display for data checks
* added usage docs to data_checks reference docs
* Apply suggestions from code review provided by scholtzan
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added reference doc for data checks and a cookbook for adding new data checks
* Apply suggestions from code review by ascholtz
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Revert "CI fixes for supporting private UDFs in bigquery-etl - DENG-735 (#3631)"
This reverts commit edcfe758f7.
* Added stub UDF for monetized_search
* Add docs for using a private internal UDF
* Fixed typo in airflow_tags.md
As Sean pointed out, the tag provided in the docs was incorrect. This change fixes that.
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Use `mdx_truly_sane_lists` to fix nested list formatting in the docs (by default, mkdocs requires nested lists to be indented by four spaces, which isn't how most nested lists in our docs have been formatted).
* Enable syntax highlighting for code blocks in the docs.
* Add support for GitHub Flavored Markdown task lists in the docs.
* UDF to retrieve the earliest not null value in an array.
* Test validated.
* Documentation upgrade for running the tests.
* Update sql/mozfun/norm/get_earliest_value/metadata.yaml
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update documentation.
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* DENG-127 Update docs to include that backfills are documented in Bugzilla.
* DENG-127 Add query for backfill validation.
* DENG-127 Add query for backfill validation.
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* Update docs to remove the default instruction to use `mozilla-public-data` as the project to create a new table. Clarify to specify the project.
* Update documentation to create a derived dataset. Replace the use of init files for the use of schemas.
* Small corrections
* Small corrections
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* Ignore .idea/ folder in pycharm
* Make instructions to add field generic with example. Add recommendations to delete field.
* Reformat numbered list for consistency.
* Update instructions based on PR feedback.
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Sean Rose
kik-kik
Anna Scholtz
Alekhya
Mike Williams
Lucia
Curtis Morales
Leli
Winnie Chan
Daniel Thorn
Alexander
akkomar
Frank Bertsch
Rebecca BurWei