* Remove hardcoded dataset in baseline clients last seen check
* Remove extra .
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Prepare active_users_aggregates for a backfill with shredder mitigation. Rename columns [first_seen_date, os_version, segment] to cascade upstream changes during the backfill for stable numbers.
* Update CODEOWNERS with the new version.
* Avoid modifying the active_users_aggregates view until the new version is backfilled. Remove the view generation from the new version which conflicts with previous version.
* Generate DAG with version.
* Adjust schema to query.
* Ensure attribution columns are present in mobile's schema.
* Convert NULLS in city to '??' as required by existing data.
* Remove uri_count, active_hours from views. Missing 'AS' added to to query.
* Create iOS_onboarding.toml
Added iOS_onboarding.toml - query for generating the underlying table in the iOS onboarding dashboard.
* Update iOS_onboarding.toml
* feat: remove locale from dimension list, it does not exist in the source table
* Delete sql_generators/funnels/configs/iOS_onboarding.toml
* Create ios_onboarding.toml
updated to lower case file name and removed "locale" since it doesn't exist in firefox_ios_clients
* add a line in bqetl_project.yaml to avoid dry run
---------
Co-authored-by: kik-kik <kignasiak@mozilla.com>
* Switch to baseline beginning Aug 01st 2024
* Modify the search_engine_daily view to handle the cut off
* Fix sql format
* Skip mobile_clients_daily_v2 from dry run
* Remove sql_generator files populating v1
* Add tests for mobile_search_clients_daily_v2
* remove unwanted tests
* Bug 1905938 Support events with no metrics in glean_usage generator
* fix event_error_monitoring
* not
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* fix: add missing project id prefix in the bigconfig templates in the fq_table_name
* feat: update the bigeye collection to use the non-kpi collection
* feat: tweak glean_usage generator to include install_source in derived baseline tables
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/glean_usage/templates/clients_last_seen_joined.query.sql
* feat: update the test definition to include install_source
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* feat: add more granual attribution fields to the mobile_kpi_metric generator
* fix: trailing comma in the EXCEPT union view
* fix: new_profile_clients omitting timestamp field
* feat: for now remove attribution_clients from union
* feat: include is_daily_user filter to ensure only new profiles are included and use submission_date as first_seen_date so that the downstream query date filter works correctly and limits processing
* feat: additional guard clause to include only new_profile true rows where submission_date = first_seen_date
In https://github.com/mozilla/bigquery-etl/pull/2333 we started filtering out overactive clients from desktop events_daily query. Back then I opted for not adding this filter for Glean queries as their event counts were significantly lower than desktop.
We are now having `bqetl_event_rollup.mozilla_vpn_derived__events_daily__v1` failing with `Cannot query rows larger than 100MB limit.` error. We'll fix it by extending the `client_event_count` filter to all queries.
3M threshold seems safe and a good first value to try - I have tested this query with this threshold on `2024-08-18` and got the same number of rows in the output table as currently in production (10598).
I tested `2024-08-19` by running:
```
bqetl generate events_daily --use_cloud_function=False --output_dir=sql_test_events_daily
cat sql_test_events_daily/moz-fx-data-shared-prod/mozilla_vpn_derived/events_daily_v1/query.sql | bq query --project_id=moz-fx-data-shared-prod --parameter=submission_date:DATE:2024-08-19 --use_legacy_sql=false --max_rows=0 --dataset_id=mozdata:tmp --destination_table=akomar_vpn_events_daily_test --replace
```
Ben Wu
Lucia
Anna Scholtz
rzhao
Alekhya
m-d-bowerman
Katie Windau
kik-kik
Sean Rose
akkomar
Alexander
Arkadiusz Komarzewski
ksiegler1