ed8e313189
feat: remove the condition resulting in exception if no metric_type value is defined ( #6552 )
2024-11-22 15:24:56 +00:00
e8821ef26a
Update logic for handling errors when Bigeye partitioning is already set ( #6549 )
2024-11-21 22:30:18 +00:00
fac775de35
Backfill iOS onboarding funnels ( #6548 )
2024-11-21 22:05:48 +00:00
2f655612c8
Disable monitoring for telemetry.releases ( #6547 )
2024-11-21 20:10:41 +00:00
4a8bdc5cd5
Remove Bigeye Airflow dependency ( #6546 )
2024-11-21 19:50:01 +00:00
11d854427c
Deploy materialized views if they do not exist ( #6540 )
2024-11-21 17:31:46 +00:00
60fea351c6
Add preventative check to ensure clients_first_seen_v2 and _v3 are never run out of order ( #6543 )
...
* Add preventative check to ensure this is never run out of order
* Fix table name
2024-11-20 22:46:55 +00:00
65402717f2
Rz modify ios onboarding funnel ( #6503 )
...
* Create iOS_onboarding.toml
Added iOS_onboarding.toml - query for generating the underlying table in the iOS onboarding dashboard.
* Update iOS_onboarding.toml
* feat: remove locale from dimension list, it does not exist in the source table
* Delete sql_generators/funnels/configs/iOS_onboarding.toml
* update ios onboarding funnel
* break ios onboarding funnel into two smaller funnels
* break ios onboarding funnel into two smaller funnels (#6537 )
* Temporarily add active_users v3 to codeowners until v4 backfill is completed. (#6534 )
* chore(DO-1886): Add shredder mitigation to more tables (#6532 )
* Add shredder mitigation to desktop eng & ret tables
* Add shredder mitigation to desktop new profile agg v1
* Add fenix_derived.funnel_retention_week_4_v1 to shredder mitigation
* Add urlbar_events_daily_eng_by_pos_v1 to shredder mitigation
* Add firefox_ios_derived.funnel_retention_week_4_v1 shredder mitigation
* Add desktop funnel installs v1 to shredder mitigation
* Add firefox_nondesktop_exact_mau28_v1
* Add shredder mitigation to urlbar_events_daily_v1
* chore(DO-1886): Add shredder mitigation to more tables (#6535 )
* Add shredder mitigation to searchreport v1
* Add mobile search agg for search report v1 to shredder mitigation
* Add smoot usage fxa compressed v2 to shredder mitigation
* Add smoot usage new profiles comp v2 to shredder mitigation
* change types ofr created_at and job_created_at from STRING to DATE (#6536 )
---------
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com>
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
* Use events_stream table
---------
Co-authored-by: kik-kik <kignasiak@mozilla.com>
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com>
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-11-20 22:37:20 +00:00
19237a99b2
Temporarily change `BigQueryTablePartitionExistenceSensor`s to run in reschedule mode (bug 1932180) ( #6542 )
...
* Temporarily change `BigQueryTablePartitionExistenceSensor`s to run in reschedule mode.
There's a bug in those sensors when they run in deferrable mode where the secondary queries they run to detect partitions fail, which is being fixed in apache/airflow#44225 .
* Fix `test_dag_with_bigquery_table_sensors`.
2024-11-20 21:25:14 +00:00
635e3f47ed
chore(DO-1886): Add shredder_mitigation to mobile engagement and retention tables ( #6538 )
...
* Add retention generator to have shredder mitigation
* Add engagement
2024-11-20 20:31:05 +00:00
7d252ab9b5
Support for deploying queries consisting of only metadata.yaml and schema.yaml ( #6529 )
...
* Support for deploying queries consisting of only metadata.yaml and schema.yaml
* Fix parameter naming for deploy_table; prevent deploys for schemas when views and materialized_views exist
* add test
* Filter out sql and query.py
2024-11-20 19:53:22 +00:00
26ae738d8f
change types ofr created_at and job_created_at from STRING to DATE ( #6536 )
2024-11-20 18:07:48 +00:00
5a4c6c9aed
chore(DO-1886): Add shredder mitigation to more tables ( #6535 )
...
* Add shredder mitigation to searchreport v1
* Add mobile search agg for search report v1 to shredder mitigation
* Add smoot usage fxa compressed v2 to shredder mitigation
* Add smoot usage new profiles comp v2 to shredder mitigation
2024-11-20 17:34:57 +00:00
648e5c724c
chore(DO-1886): Add shredder mitigation to more tables ( #6532 )
...
* Add shredder mitigation to desktop eng & ret tables
* Add shredder mitigation to desktop new profile agg v1
* Add fenix_derived.funnel_retention_week_4_v1 to shredder mitigation
* Add urlbar_events_daily_eng_by_pos_v1 to shredder mitigation
* Add firefox_ios_derived.funnel_retention_week_4_v1 shredder mitigation
* Add desktop funnel installs v1 to shredder mitigation
* Add firefox_nondesktop_exact_mau28_v1
* Add shredder mitigation to urlbar_events_daily_v1
2024-11-20 15:12:09 +00:00
29fc585874
Temporarily add active_users v3 to codeowners until v4 backfill is completed. ( #6534 )
2024-11-20 13:14:18 +00:00
d942a57714
Bump aiohttp from 3.10.2 to 3.10.11 ( #6520 )
...
Bumps [aiohttp](https://github.com/aio-libs/aiohttp ) from 3.10.2 to 3.10.11.
- [Release notes](https://github.com/aio-libs/aiohttp/releases )
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst )
- [Commits](https://github.com/aio-libs/aiohttp/compare/v3.10.2...v3.10.11 )
---
updated-dependencies:
- dependency-name: aiohttp
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-20 00:09:53 +00:00
8aa22355a0
chore(DO-1887): Add shredder mitigation to search aggregate tables ( #6531 )
...
* Add shredder_mitigation = True for search_derived.mobile_search_aggregates_v1
* Adding shredder mitigation to some search_derived aggregate tables
2024-11-19 23:35:47 +00:00
cd9d0287cc
add in os.environ.get to get bearer_token ( #6528 )
...
* add in os.environ.get to get bearer_token
* properly indent key in metadata
2024-11-19 21:05:00 +00:00
02f9d31e95
Remove the scheduler for v4, as rthe tables need to be moved to production via a backfill with shredder mitigation. It will be added after the tables are moved to production. ( #6530 )
2024-11-19 19:41:19 +00:00
42185bab6a
complete backfill ( #6527 )
2024-11-19 18:21:48 +00:00
1e13a6c29a
Support results with no rows in matches_pattern check ( #6521 )
...
* Support results with no rows in matches_pattern check
* remove extra )
2024-11-19 18:13:07 +00:00
6890005c1d
add date partition parameter to scheduling in metadata ( #6524 )
2024-11-19 16:10:55 +00:00
9cccd521b3
Make select explicit because GA keeps adding columns ( #6523 )
2024-11-19 14:23:15 +00:00
34218a1f3b
DSRE 1639 ( #6522 )
...
* Set retention to 25 months on firefox_desktop_derived.pageload_1pct_v1
* Set retention to 25 months on firefox_desktop_derived.pageload_nightly_v1
2024-11-19 13:45:39 +00:00
6687e67ba8
Don't set shredder target for datasets with no deletion requests ( #6519 )
2024-11-19 13:08:57 +00:00
dd2a762441
change schema mode from REPEATED to NULLABLE ( #6518 )
2024-11-18 21:48:06 +00:00
59e49ea36c
Remove hardcoded dataset in baseline clients last seen check ( #6514 )
...
* Remove hardcoded dataset in baseline clients last seen check
* Remove extra .
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-11-18 20:38:11 +00:00
16744aaf20
Set retention to 25 months on firefox_desktop_derived.pageload_1pct_v1 ( #6513 )
2024-11-18 20:25:48 +00:00
92b7098f50
Prepare active_users_aggregates for a backfill with shredder mitigation. ( #6349 )
...
* Prepare active_users_aggregates for a backfill with shredder mitigation. Rename columns [first_seen_date, os_version, segment] to cascade upstream changes during the backfill for stable numbers.
* Update CODEOWNERS with the new version.
* Avoid modifying the active_users_aggregates view until the new version is backfilled. Remove the view generation from the new version which conflicts with previous version.
* Generate DAG with version.
* Adjust schema to query.
* Ensure attribution columns are present in mobile's schema.
* Convert NULLS in city to '??' as required by existing data.
* Remove uri_count, active_hours from views. Missing 'AS' added to to query.
2024-11-18 20:20:22 +00:00
037eccc547
Remove row count check for net_thunderbird_android_derived.baseline_clients_last_seen_v1 ( #6515 )
2024-11-18 20:13:02 +00:00
9fdf7611f3
Deng 6360 add arguments to metadata file ( #6517 )
...
* update logic and metadata to use the Cinder token secret
* add trailing white space in jiinja var
2024-11-18 19:48:20 +00:00
ad51416036
initiate backfill ( #6516 )
2024-11-18 18:44:25 +00:00
b79a7b9242
initial add of scripts for addon_moderations ( #6484 )
...
* initial add of scripts for addon_moderations
* move files over to addon_moderations_derived, update metadata.yaml to partition by date
* update metadata.yaml to type:day and field:date
* change DAG from bqetl_cinder to bqetl_addons
* remove query file, update query.py file"
* take out bexplicit bearer token
* take out clustering
2024-11-15 21:22:05 +00:00
47559f846f
complete backfill ( #6512 )
2024-11-15 20:16:34 +00:00
8cf14f28e7
Set retention to 25 months on firefox_ios_derived.active_users_aggregates_v1 ( #6509 )
2024-11-15 18:54:55 +00:00
aaa1f14ca6
replace metadata file ( #6510 )
2024-11-15 18:48:22 +00:00
b83224393c
Improve checks to account for null replacements ( #6508 )
...
* Backfill info.
* Update tests,
* Update tests,
2024-11-15 17:03:02 +00:00
944d6d55df
initiate backfill ( #6507 )
2024-11-15 16:30:55 +00:00
f40e5415b0
Complete backfill of logical subscription ETLs to populate new discount fields. ( #6506 )
2024-11-15 00:01:07 +00:00
d329ec4222
Fix Christmas date in tentpole dates csv ( #6505 )
2024-11-14 21:02:51 +00:00
44a26dfd1a
Add depends on past = true ( #6501 )
2024-11-14 17:31:21 +00:00
b7ff9f242d
Fix newtab reporting issues caused by the is_list_card filter ( #6500 )
...
* updating filter to address null values
* formatting
2024-11-14 17:01:49 +00:00
2eac9a0fd0
Initiate backfill of logical subscription ETLs to populate new discount fields. ( #6495 )
2024-11-14 16:20:22 +00:00
0ee397021a
remove backtick ( #6486 )
2024-11-14 15:59:51 +00:00
11a9fc0ecc
Update logic to exclude combinations sent previously ( #6498 )
2024-11-14 14:55:31 +00:00
1f91163f6c
feat: adding an additional case for when metric.metric_type is null inside _update_bigconfig ( #6482 )
2024-11-14 10:05:59 +00:00
2bd130526e
Improve Bigeye error reporting ( #6493 )
2024-11-14 00:55:36 +00:00
c5b4f8efc3
Incorporate discounts into the logical subscription ETLs (DENG-974) ( #6474 )
...
* Fix subscription discounts join to exclude discounts which already ended.
* Incorporate discounts into the logical subscription ETLs.
2024-11-14 00:36:51 +00:00
16589156cf
Complete backfill of `subscription_platform_derived.monthly_active_logical_subscriptions_v1`. ( #6492 )
2024-11-13 23:43:58 +00:00
09f1605399
Add more dates to Key Tentpoles data.csv ( #6491 )
...
Add more dates for 2023
2024-11-13 22:31:03 +00:00
kik-kik
Anna Scholtz
Ben Wu
Katie Windau
rzhao
Sean Rose
Marlene Hirose
Lucia
dependabot[bot]
m-d-bowerman
whd
Alekhya
Chelsey Beck
ryboland