26ae738d8f
change types ofr created_at and job_created_at from STRING to DATE ( #6536 )
2024-11-20 18:07:48 +00:00
5a4c6c9aed
chore(DO-1886): Add shredder mitigation to more tables ( #6535 )
...
* Add shredder mitigation to searchreport v1
* Add mobile search agg for search report v1 to shredder mitigation
* Add smoot usage fxa compressed v2 to shredder mitigation
* Add smoot usage new profiles comp v2 to shredder mitigation
2024-11-20 17:34:57 +00:00
648e5c724c
chore(DO-1886): Add shredder mitigation to more tables ( #6532 )
...
* Add shredder mitigation to desktop eng & ret tables
* Add shredder mitigation to desktop new profile agg v1
* Add fenix_derived.funnel_retention_week_4_v1 to shredder mitigation
* Add urlbar_events_daily_eng_by_pos_v1 to shredder mitigation
* Add firefox_ios_derived.funnel_retention_week_4_v1 shredder mitigation
* Add desktop funnel installs v1 to shredder mitigation
* Add firefox_nondesktop_exact_mau28_v1
* Add shredder mitigation to urlbar_events_daily_v1
2024-11-20 15:12:09 +00:00
8aa22355a0
chore(DO-1887): Add shredder mitigation to search aggregate tables ( #6531 )
...
* Add shredder_mitigation = True for search_derived.mobile_search_aggregates_v1
* Adding shredder mitigation to some search_derived aggregate tables
2024-11-19 23:35:47 +00:00
cd9d0287cc
add in os.environ.get to get bearer_token ( #6528 )
...
* add in os.environ.get to get bearer_token
* properly indent key in metadata
2024-11-19 21:05:00 +00:00
42185bab6a
complete backfill ( #6527 )
2024-11-19 18:21:48 +00:00
6890005c1d
add date partition parameter to scheduling in metadata ( #6524 )
2024-11-19 16:10:55 +00:00
9cccd521b3
Make select explicit because GA keeps adding columns ( #6523 )
2024-11-19 14:23:15 +00:00
34218a1f3b
DSRE 1639 ( #6522 )
...
* Set retention to 25 months on firefox_desktop_derived.pageload_1pct_v1
* Set retention to 25 months on firefox_desktop_derived.pageload_nightly_v1
2024-11-19 13:45:39 +00:00
6687e67ba8
Don't set shredder target for datasets with no deletion requests ( #6519 )
2024-11-19 13:08:57 +00:00
dd2a762441
change schema mode from REPEATED to NULLABLE ( #6518 )
2024-11-18 21:48:06 +00:00
16744aaf20
Set retention to 25 months on firefox_desktop_derived.pageload_1pct_v1 ( #6513 )
2024-11-18 20:25:48 +00:00
037eccc547
Remove row count check for net_thunderbird_android_derived.baseline_clients_last_seen_v1 ( #6515 )
2024-11-18 20:13:02 +00:00
9fdf7611f3
Deng 6360 add arguments to metadata file ( #6517 )
...
* update logic and metadata to use the Cinder token secret
* add trailing white space in jiinja var
2024-11-18 19:48:20 +00:00
ad51416036
initiate backfill ( #6516 )
2024-11-18 18:44:25 +00:00
b79a7b9242
initial add of scripts for addon_moderations ( #6484 )
...
* initial add of scripts for addon_moderations
* move files over to addon_moderations_derived, update metadata.yaml to partition by date
* update metadata.yaml to type:day and field:date
* change DAG from bqetl_cinder to bqetl_addons
* remove query file, update query.py file"
* take out bexplicit bearer token
* take out clustering
2024-11-15 21:22:05 +00:00
47559f846f
complete backfill ( #6512 )
2024-11-15 20:16:34 +00:00
8cf14f28e7
Set retention to 25 months on firefox_ios_derived.active_users_aggregates_v1 ( #6509 )
2024-11-15 18:54:55 +00:00
aaa1f14ca6
replace metadata file ( #6510 )
2024-11-15 18:48:22 +00:00
944d6d55df
initiate backfill ( #6507 )
2024-11-15 16:30:55 +00:00
f40e5415b0
Complete backfill of logical subscription ETLs to populate new discount fields. ( #6506 )
2024-11-15 00:01:07 +00:00
d329ec4222
Fix Christmas date in tentpole dates csv ( #6505 )
2024-11-14 21:02:51 +00:00
44a26dfd1a
Add depends on past = true ( #6501 )
2024-11-14 17:31:21 +00:00
b7ff9f242d
Fix newtab reporting issues caused by the is_list_card filter ( #6500 )
...
* updating filter to address null values
* formatting
2024-11-14 17:01:49 +00:00
2eac9a0fd0
Initiate backfill of logical subscription ETLs to populate new discount fields. ( #6495 )
2024-11-14 16:20:22 +00:00
11a9fc0ecc
Update logic to exclude combinations sent previously ( #6498 )
2024-11-14 14:55:31 +00:00
c5b4f8efc3
Incorporate discounts into the logical subscription ETLs (DENG-974) ( #6474 )
...
* Fix subscription discounts join to exclude discounts which already ended.
* Incorporate discounts into the logical subscription ETLs.
2024-11-14 00:36:51 +00:00
16589156cf
Complete backfill of `subscription_platform_derived.monthly_active_logical_subscriptions_v1`. ( #6492 )
2024-11-13 23:43:58 +00:00
09f1605399
Add more dates to Key Tentpoles data.csv ( #6491 )
...
Add more dates for 2023
2024-11-13 22:31:03 +00:00
8721a4cc6b
Revert "Add Bigeye checks for Thunderbird datasets ( #6471 )" ( #6490 )
...
This reverts commit 5aa5f2d44c
2024-11-13 21:25:45 +00:00
eb8718b293
Initiate backfill of `subscription_platform_derived.monthly_active_logical_subscriptions_v1`. ( #6463 )
2024-11-13 20:46:40 +00:00
704141ece3
Revert "Fix bigeye checks to de-dup metric names ( #6485 )" ( #6488 )
...
This reverts commit 1a046fa94a
2024-11-13 20:44:16 +00:00
055a83afc0
Fix the `monthly_active_logical_subscriptions_v1` ETL to correctly calculate `was_active_at_month_end` when a subscription ends exactly at midnight. ( #6489 )
2024-11-13 20:41:47 +00:00
34f7479d55
Fix `stripe_subscriptions_history_v2.valid_to` getting cut short for ended subscriptions with subsequent customer changes. ( #6487 )
2024-11-13 19:57:24 +00:00
1a046fa94a
Fix bigeye checks to de-dup metric names ( #6485 )
2024-11-13 18:23:28 +00:00
0a76f4cc5b
Complete backfill of `subscription_platform_derived.logical_subscription_events_v1`. ( #6476 )
2024-11-13 17:41:58 +00:00
12febd9402
Complete backfill of `subscription_platform_derived.daily_active_logical_subscriptions_v1`. ( #6475 )
2024-11-13 17:41:46 +00:00
f3648ba4dd
Switch from v1 to v2 ( #6481 )
2024-11-13 15:09:18 +00:00
9bad9ae094
Change `subscriptions_changelog_v1` ETL to add all available incremental records in append-only mode. ( #6468 )
2024-11-13 02:16:41 +00:00
5aa5f2d44c
Add Bigeye checks for Thunderbird datasets ( #6471 )
...
* Add Bigeye checks for Thunderbird datasets
* add thunder_desktop_derived bigeye config yaml
add thunder_desktop_derived bigeye config yaml
* Fix CI issue
2024-11-12 23:40:08 +00:00
bf6852e4b1
Create gclid_conversions_v3 & ga_desktop_conversions_v2 ( #6467 )
...
* Initial commit, work in progress
* Not done yet, saving work
* Creating new gclid conversions v3 table
* Fix returned_second_day CTE
* Remove conversion window
* Fix union
* update data type
* Add filters
* Remove 3 views
* Fix the SQL
* Put the parameter back after testing
* Make query scan clients daily less
* Make more efficient
* Make column explicit
* Update checks
* Update schema
* Update ga desktop conversions v1 to use new gclid conv v3
* Create new ga_desktop_conversions_v2
---------
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-11-12 22:44:56 +00:00
a631bc7bfb
Update data.csv for Key Tentpoles ( #6409 )
...
* Update data.csv
Add 2023 Cyber 5 dates
* Update data.csv
Add dates for 2021 and 2022
* Update data.csv
Change Cyber 5 to Cyber Five
---------
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2024-11-12 20:20:06 +00:00
225042f226
DENG-4835 - delete 3 views missed in previous PR ( #6469 )
2024-11-11 16:10:41 +00:00
26d3e9faf0
DENG-4835 - Deleting deprecated views & tables ( #6455 )
...
* DENG-4835 - Delete deprecated views
* DENG-4835 delete deprecated tables
* DENG-4835 delete deprecated tables
* DENG-4835 Remove a deleted table from shredder processing
2024-11-09 18:51:07 +00:00
d0e6d6c1ee
Initiate backfill of `subscription_platform_derived.daily_active_logical_subscriptions_v1`. ( #6462 )
2024-11-08 23:09:41 +00:00
ac0d6e1670
Initiate backfill of `subscription_platform_derived.logical_subscription_events_v1`. ( #6461 )
2024-11-08 23:09:18 +00:00
e0a9f72379
Complete backfill of `subscription_platform_derived.stripe_subscriptions_revised_changelog_v1`. ( #6464 )
2024-11-08 20:12:52 +00:00
68ae66a85d
Update warn checks for net_thunderbird_android_daily_derived ( #6457 )
...
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-11-08 18:09:41 +00:00
b47710ac9f
Initiate backfill of `subscription_platform_derived.stripe_subscriptions_revised_changelog_v1`. ( #6460 )
2024-11-08 00:44:21 +00:00
6b18bcb287
Fix subscription changelog conflicts ( #6458 )
...
* Don't create `adjusted_subscription_end` changelogs that conflict with previous changelogs.
* Don't use unreliable invoice line item `plan_change_date` metadata.
* Guard against similar issues in `stripe_customers_revised_changelog_v1`.
* Don't include customer changes in `stripe_subscriptions_history_v2` after the subscription ends.
* Create `synthetic_plan_change` changelogs from subscription "Remaining time" invoice line items.
* Guard against conflicting changelog records in Stripe history ETLs.
* Add checks to catch any duplicate IDs.
2024-11-07 22:40:15 +00:00
Marlene Hirose
Katie Windau
m-d-bowerman
whd
Ben Wu
Alekhya
Sean Rose
Chelsey Beck
ryboland
Anna Scholtz