* Generate Bigeye monitoring configs in CI
* Ensure Bigconfig files are loaded exactly once
* Avoid duplicate validation of Bigconfig files
* Authentication using API key to Bigeye
* Remove api-key option for Bigeye and rely on env var instead
* Add default values in template to fix sqlglot parsing error.
* Adding backfill_date to exception message. Formatting.
* Improve getting collumn dtypes by including the schema files. Change custom_query for custom_query_path for readibility.
* DENG_4733. Improve getting column dtypes by including the schema files. Change custom_query for custom_query_path for readibility.
* Formatting.
* Set values back to NULL were corresponds. Improve output information.
* Rename custom_query to custom_query_path to match the expected parameter.
* Missing import
* Larger wildcards to reduce the chance of collision with actual values.
* Desktop_retention_aggregates_v2 changes the data source from clients_first_seen_v2 to v3 as well as adding distribution_id to table
* add owner1 label
* add backfill.yaml script to initiate backfill on this table
* Desktop_retention_clients_v2 changes the data source from clients_first_seen_v2 to v3. There are changes to logic for is_desktop, os, os_version as well as adding profile_group_id and days_active_bits
* add owner1 name to metadata.yaml
* initial commit work in progress
* DSRE-1681 - create new tables for GA3 and GA4 SUMO data, copies of raw
* DSRE-1681 remove clustering from both metadata.yaml files
* Add default values in template to fix sqlglot parsing error.
* Adding backfill_date to exception message. Formatting.
* Improve getting collumn dtypes by including the schema files. Change custom_query for custom_query_path for readibility.
* DENG_4733. Improve getting column dtypes by including the schema files. Change custom_query for custom_query_path for readibility.
* Formatting.
This will generate events_stream and event_monitoring_live tables for ads_backend. We originally skipped this app because there was a delay in its ping table deployment.
* Create derived table containing data inputs into desktop tiles forecasts.
* Create derived table containing data inputs into desktop tiles forecasts.
* Query adjustments and better data descriptions
* Making sure table descriptions are identical.
* Correcting partitioning field
* updated to avoid manual lookback and removed session columns
* Update sql/moz-fx-data-shared-prod/ads_derived/desktop_tiles_forecast_inputs_v1/metadata.yaml
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* Apply suggestions from code review
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* move max date filter
* add dag for job
* column name change _3 to _1to3
* remove monthly run and add the new table to bqetl_ads
* Apply suggestions from code review
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* fix name change in schema
---------
Co-authored-by: Jared Snyder <jsnyder@mozilla.com>
Co-authored-by: Jared Snyder <jaredssnyder@gmail.com>
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* initial add of query.sql and metadata.yaml files
* format sql file to stop Circle CI error
* label count of client_ids, add schema.yaml file
* add dag_name to metadata.yaml
* take out extraneous columns, reorder to match query
* add date partition parameter as first_seen_date
* add period to end of descriptions
* take out explicit dependency
* make query more efficient by setting first_seen_date to equal @submission_date and explicity use Inner join instead of left
* Auxiliary functions required to generate the query for a backfill with shredder mitigation.
* Exception handling.
* isort & docstrings.
* Apply flake8 to test file.
* Remove variable assignment to different types.
* Make search case insensitive in function.
* Add test cases for function and update naming in a funcion's parameters for clarity.
* Update bigquery_etl/backfill/shredder_mitigation.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* Add test cases for missing parameters or not matching parameters where expected. minimize the calls for get_bigquery_type().
* Encapsulate actions to generate and run custom queries to generate the subsets for shredder mitigation.
* Query template for shredder mitigation.
* Query template for shredder mitigation and formatting.
* Add check for "GROUP BY 1, 2, 3", improve code readibility, remove unnecesary properties in classes.
* Test coverage. Check for "GROUP BY 1, 2, 3", improve readibility, remove unrequired properties in class Subset.
* Increase test coverage. Expand DataType INTEGER required for UNION queries.
* Increase test coverage. Expand DataType INTEGER required for UNION queries.
* Separate INTEFER and NUMERIC types.
* Move util functions and convert method to property, both to resolve a circular import. Adjust tests. Update function return and tests.
* Adding backfill_date to exception message. Formatting.
* Adding backfill_date to exception message. Formatting.
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* feat: include is_daily_user filter to ensure only new profiles are included and use submission_date as first_seen_date so that the downstream query date filter works correctly and limits processing
* feat: additional guard clause to include only new_profile true rows where submission_date = first_seen_date
Katie Windau
Anna Scholtz
kik-kik
dependabot[bot]
Lucia
akkomar
Ben Wu
Marlene Hirose
Leli
Eduardo Filho
Arkadiusz Komarzewski
Sergio E. Betancourt
Corban Cloud
Preethi Issac
skahmann3
Abhishek Aggarwal