* Remove deleted table from skip list
* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl
* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl
* Clean up test
* Larger wildcards to reduce the chance of collision with actual values.
* Formatting
* Update bigquery_etl/metadata/validate_metadata.py
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* Add test for validate_metadata.validate, add profile_id and profile_group_id to id-level_columns file.
* Update tests/cli/test_cli_metadata.py
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
---------
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* Auxiliary functions required to generate the query for a backfill with shredder mitigation.
* Exception handling.
* isort & docstrings.
* Apply flake8 to test file.
* Remove variable assignment to different types.
* Make search case insensitive in function.
* Add test cases for function and update naming in a funcion's parameters for clarity.
* Update bigquery_etl/backfill/shredder_mitigation.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Add test cases for missing parameters or not matching parameters where expected. minimize the calls for get_bigquery_type().
* Encapsulate actions to generate and run custom queries to generate the subsets for shredder mitigation.
* Query template for shredder mitigation.
* Query template for shredder mitigation and formatting.
* Add check for "GROUP BY 1, 2, 3", improve code readibility, remove unnecesary properties in classes.
* Test coverage. Check for "GROUP BY 1, 2, 3", improve readibility, remove unrequired properties in class Subset.
* Increase test coverage. Expand DataType INTEGER required for UNION queries.
* Increase test coverage. Expand DataType INTEGER required for UNION queries.
* Separate INTEFER and NUMERIC types.
* Move util functions and convert method to property, both to resolve a circular import. Adjust tests. Update function return and tests.
* Adding backfill_date to exception message. Formatting.
* Adding backfill_date to exception message. Formatting.
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Auxiliary functions required to generate the query for a backfill with shredder mitigation.
* Exception handling.
* isort & docstrings.
* Apply flake8 to test file.
* Remove variable assignment to different types.
* Make search case insensitive in function.
* Add test cases for function and update naming in a funcion's parameters for clarity.
* Update bigquery_etl/backfill/shredder_mitigation.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Add test cases for missing parameters or not matching parameters where expected. minimize the calls for get_bigquery_type().
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* feat(query-backfill): adding a more flexible approach to overriding scheduling attributes
* Update tests/cli/test_cli_query.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* feat(query.py): adding cleaner override logic per PR comments and also cleaning up comments and rogue print statement
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
* feat(query.py): adding the ability to override scheduling parameters for backfills since backfill can only parse one kind of date parameter
* Update tests/cli/test_cli_query.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* Update bigquery_etl/cli/query.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* feat(query.py): adding back code needed for scheduling_parameters initial value
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
* Update dataset workgroup_access when deprecated: true
* Update deprecation metadata tests
* Add metadata.yaml files in telemetry_derived for tables that are managed through other tooling
* Deprecate telemetry_derived datasets
* Don't try to write existing view files
* Use state_values_v2 for client ad click predictions
* Normalize countries in client_ltv
* Don't get view if unavailable
* Add test for new version of existing table
* Fully qualify tables in view defn
* Skip backfills for queries without metadata.yaml
* Support date_partition_offset
* Fixed exclude, modified exception
* Add test for offset backfill
* Apply suggestions from code review
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Formatting
---------
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* DENG-1314 Implement changes to bqetl and create default DAG.
* DENG-1314. Update Documentation.
* DENG-1314. Dummy query to enable generating DAG and run tests.
* DENG-1314. Update tests.
* Update bigquery_etl/cli/query.py
Raise exception when scheduling information is missing.
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* DENG-1314. Update tests.
* DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests.
* Default task and tests update.
* Default task and tests update.
* 3650 - Remove default DAG option, update DAG template comment & tests.
* 3650 - Condition for DAG warning.
* 3650 - Update docs.
* Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update docs/cookbooks/creating_a_derived_dataset.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added render subcommand to the bqetl check command
* added a dry_run flag to bqqetl check run command
* added a test to make sure run command exists with status code 0
* added test for check render subcommand
* fixing linter checks
* attempting using an alternative way of testing the render command
* fixing render test by testing the _render() directly rather than the render cli wrapper
* removed dead test
* Apply suggestions from code review by ascholtz
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* fixed black and mypy errors
* fixed app_store_funnel_v1 check formatting
* reformatted tests checks
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* initial impl
* Updated based on PR feedback
* Moved check from query to separate command
* Expanded from --partition option to generic --parameter option
* Removed `query check` command (check moved to new command)
* Update bigquery_etl/cli/check.py
remove date param format check
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Removed 'parameter' parameter, everything is passed through ctx.args and then converted to a dict for Jinja rendering. There are no restrictions on ctx.args values.
* Merge error
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* DENG-774 Add change control to active_users_aggregates and test.
* DENG-774 Add test coverage.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Curtis Morales
Lucia
Anna Scholtz
Winnie Chan
Alexander
Braunk
Ben Wu
Frank Bertsch
Alekhya
kik-kik
Glenda Leonard
Daniel Thorn