* chore(deploys): Catch NotFound for missing dataset/project in table deploys
* Update bigquery_etl/deploy.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* update test exception text match
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Add topic selection fields to newtab_visits
* Remove duplicate from schema
* Add new metric to test schema
* Fix event name in test yaml
* Test yaml fix
* Add visit id to test case
---------
Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>
* Add is_sap_monetizable to aggregates table
* Fix tests
* add is_acer_cohort
* Add normalized_default_search_engine column for mobile and desktop
* Fix test cases
* Auxiliary functions required to generate the query for a backfill with shredder mitigation.
* Exception handling.
* isort & docstrings.
* Apply flake8 to test file.
* Remove variable assignment to different types.
* Make search case insensitive in function.
* Add test cases for function and update naming in a funcion's parameters for clarity.
* Update bigquery_etl/backfill/shredder_mitigation.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Add test cases for missing parameters or not matching parameters where expected. minimize the calls for get_bigquery_type().
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Remove percentiles
* Remove tests that test percentiles
* Refresh scripts insert null to new percentiles
* Remove percentile columns from queries and schemas
* Delete more percentile tables
* Formatting
* histogram_cast_struct's keys are strings
* Re-add test after fixing failure cause
* DENG-4201 remove blogs_goals_v1 since now in shared prod
* DENG-4201 remove blogs_sessions_v1 from mktg prod since now in shared prod
* DENG-4201 remove www_site_hits_v1 from mktg prod since now in shared prod
* DENG-4201 remove firefox_whatsnew_summary_v1 from mktg prod since now in shared prod
* DENG-4201 remove www_site_downloads_v1 from mktg prod since now in shared prod
* DENG-4201 remove www_site_events_metrics_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove www_site_landing_page_metrics_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove www_site_page_metrics_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove blogs_daily_summary_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove blogs_landing_page_summary_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove blogs_empty_check_v1
* DENG-4201 remove www_site_empty_check_v1
* DENG-4201 remove www_site_metrics_summary_v1 from mktg prod since now moved to shared prod
* DENG-4201 remove dl with attr v1 and v2 from mktg prod since now in shared prod instead
* DENG-4201 remove tests from mktg prod since tables no longer live in mktg prod
* Add fully-qualified identifiers when formatting queries
* Fully-qualified identifiers for queries in sql/
* Check in only formatted SQL to generated-sql branch
* Add comment
* Fully qualify more tables
* Fully qualify test files
* Formatting improvements around CTEs and unit tests
* Option to skip auto qualifying queries
* RS 959 - Add OS breakouts
* Add mobile OS breakouts
Add mobile OS breakouts
* Add mobile OS breakouts
* Add tests for desktop
* Add mobile tests
* Fix tests
* Fix CI issue
* Limit `derived_view_schemas` SQL generator to actual view directories.
* Fix the `derived_view_schemas` SQL generator to get the view schemas by dry-running their latest SQL and/or from their latest `schema.yaml` file. Getting the schema from the currently deployed view wasn't appropriate because it wouldn't reflect the latest view code.
* Rename `View.view_schema` to `View.schema`.
* Change `View` so its schema dry-runs use the cloud function (CI doesn't have permission to run dry-run queries directly).
* Apply partition column filters in view dry-run queries when possible for speed/efficiency.
* Don't allow missing fields to prevent view schema enrichment.
* Only copy column descriptions during view schema enrichment.
* Only try enriching view schemas from their reference table `schema.yaml` files if those files actually exist.
* Change `main_1pct` view to select directly from the `main_remainder_1pct_v1` table, so the `derived_view_schemas` SQL generator can detect the partition column to use and successfully dry-run the view to determine its schema.
* Formalize the order `bqetl generate all` runs the SQL generators in.
* Have `bqetl generate all` run `derived_view_schemas` last, in case other SQL generators create derived views.
* Fix `Schema._traverse()` to only recurse if both fields are records.
* Fix `format_timedelta` function's parsing of negative timedeltas.
The entire timedelta can be negative.
* Refactor to use a single timedelta regular expression.
* Fix typo in `format_timedelta` function argument.
* updates to mobile_search_aggregates and search_revenue_levers_table
adding search_with_ads_organic to mobile_search_aggregates and search_revenue_levers_table
* Update query.sql
* Fix schema.yaml and test files
* Fix CI issue
Fix CI issue
---------
Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
* feat: update firefox_android_clients_v1 to pull distribution_id only from the baseline ping
* feat: update firefox_android_clients_v1 baseline test schema to include distribution_id
* fix: resolve distribution_id not in baseline error
---------
Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com>
* removing acoustic last engaged timestamp
* removing suppression list model and tests
* removing dependency until frequency is increased
* formatting
* formatting
* adding organic searches with ads to this table
* updating mobile_search_aggregates table with search_with_ads_organic column
* updating the search revenue lever table
- include search_with_ads_organic columns for Bing, Google and DDG
* Fix CI issues
* Fix tests CI failure
* fix tests
* Fix test sql failure
* Update query.sql
reverting back to original code for search_revenue_levers table
---------
Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
* feat(query-backfill): adding a more flexible approach to overriding scheduling attributes
* Update tests/cli/test_cli_query.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* feat(query.py): adding cleaner override logic per PR comments and also cleaning up comments and rogue print statement
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
* Add mobile suggest to event_aggregates and event_aggregates_suggest
* Rename test files
* Update expected values
* Fix tests
* Omit null values from expect.yaml
* updating source table and adding column
* updating tests to add new field
* changing to use updated_at
* formatting
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
Alexander
m-d-bowerman
Alekhya
Katie Windau
Lucia
Winnie Chan
Eduardo Filho
Chelsey Beck
Ben Wu
Anna Scholtz
Sean Rose
Curtis Morales
Preethi Issac
kik-kik
Braunk
Leli