8721a4cc6b
Revert "Add Bigeye checks for Thunderbird datasets ( #6471 )" ( #6490 )
...
This reverts commit 5aa5f2d44c
2024-11-13 21:25:45 +00:00
eb8718b293
Initiate backfill of `subscription_platform_derived.monthly_active_logical_subscriptions_v1`. ( #6463 )
2024-11-13 20:46:40 +00:00
704141ece3
Revert "Fix bigeye checks to de-dup metric names ( #6485 )" ( #6488 )
...
This reverts commit 1a046fa94a
2024-11-13 20:44:16 +00:00
055a83afc0
Fix the `monthly_active_logical_subscriptions_v1` ETL to correctly calculate `was_active_at_month_end` when a subscription ends exactly at midnight. ( #6489 )
2024-11-13 20:41:47 +00:00
34f7479d55
Fix `stripe_subscriptions_history_v2.valid_to` getting cut short for ended subscriptions with subsequent customer changes. ( #6487 )
2024-11-13 19:57:24 +00:00
1a046fa94a
Fix bigeye checks to de-dup metric names ( #6485 )
2024-11-13 18:23:28 +00:00
0a76f4cc5b
Complete backfill of `subscription_platform_derived.logical_subscription_events_v1`. ( #6476 )
2024-11-13 17:41:58 +00:00
12febd9402
Complete backfill of `subscription_platform_derived.daily_active_logical_subscriptions_v1`. ( #6475 )
2024-11-13 17:41:46 +00:00
dc2d71e164
Bump setuptools from 70.0.0 to 75.4.0 ( #6479 )
...
Bumps [setuptools](https://github.com/pypa/setuptools ) from 70.0.0 to 75.4.0.
- [Release notes](https://github.com/pypa/setuptools/releases )
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/setuptools/compare/v70.0.0...v75.4.0 )
---
updated-dependencies:
- dependency-name: setuptools
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-11-13 17:05:17 +00:00
0b5fe0c8ac
Bump google-cloud-bigquery from 3.26.0 to 3.27.0 ( #6477 )
...
Bumps [google-cloud-bigquery](https://github.com/googleapis/python-bigquery ) from 3.26.0 to 3.27.0.
- [Release notes](https://github.com/googleapis/python-bigquery/releases )
- [Changelog](https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md )
- [Commits](https://github.com/googleapis/python-bigquery/compare/v3.26.0...v3.27.0 )
---
updated-dependencies:
- dependency-name: google-cloud-bigquery
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-13 16:58:57 +00:00
f3648ba4dd
Switch from v1 to v2 ( #6481 )
2024-11-13 15:09:18 +00:00
9bad9ae094
Change `subscriptions_changelog_v1` ETL to add all available incremental records in append-only mode. ( #6468 )
2024-11-13 02:16:41 +00:00
5aa5f2d44c
Add Bigeye checks for Thunderbird datasets ( #6471 )
...
* Add Bigeye checks for Thunderbird datasets
* add thunder_desktop_derived bigeye config yaml
add thunder_desktop_derived bigeye config yaml
* Fix CI issue
2024-11-12 23:40:08 +00:00
3c13b9af65
Make shredder detect glean derived tables with client_info.client_id ( #6459 )
...
* Make shredder detect glean derived tables with client_info.client_id
* Add skip list
2024-11-12 23:01:47 +00:00
bf6852e4b1
Create gclid_conversions_v3 & ga_desktop_conversions_v2 ( #6467 )
...
* Initial commit, work in progress
* Not done yet, saving work
* Creating new gclid conversions v3 table
* Fix returned_second_day CTE
* Remove conversion window
* Fix union
* update data type
* Add filters
* Remove 3 views
* Fix the SQL
* Put the parameter back after testing
* Make query scan clients daily less
* Make more efficient
* Make column explicit
* Update checks
* Update schema
* Update ga desktop conversions v1 to use new gclid conv v3
* Create new ga_desktop_conversions_v2
---------
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-11-12 22:44:56 +00:00
a631bc7bfb
Update data.csv for Key Tentpoles ( #6409 )
...
* Update data.csv
Add 2023 Cyber 5 dates
* Update data.csv
Add dates for 2021 and 2022
* Update data.csv
Change Cyber 5 to Cyber Five
---------
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2024-11-12 20:20:06 +00:00
c5073d8768
Bump pyarrow from 17.0.0 to 18.0.0 ( #6416 )
...
Bumps [pyarrow](https://github.com/apache/arrow ) from 17.0.0 to 18.0.0.
- [Release notes](https://github.com/apache/arrow/releases )
- [Commits](https://github.com/apache/arrow/compare/go/v17.0.0...apache-arrow-18.0.0 )
---
updated-dependencies:
- dependency-name: pyarrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-12 19:47:29 +00:00
6f3b9cbda0
Bump pytest-flake8 from 1.1.2 to 1.3.0 ( #6470 )
...
Bumps [pytest-flake8](https://github.com/coherent-oss/pytest-flake8 ) from 1.1.2 to 1.3.0.
- [Release notes](https://github.com/coherent-oss/pytest-flake8/releases )
- [Changelog](https://github.com/coherent-oss/pytest-flake8/blob/main/NEWS.rst )
- [Commits](https://github.com/coherent-oss/pytest-flake8/compare/v1.1.2...v1.3.0 )
---
updated-dependencies:
- dependency-name: pytest-flake8
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-11-12 18:02:58 +00:00
225042f226
DENG-4835 - delete 3 views missed in previous PR ( #6469 )
2024-11-11 16:10:41 +00:00
26d3e9faf0
DENG-4835 - Deleting deprecated views & tables ( #6455 )
...
* DENG-4835 - Delete deprecated views
* DENG-4835 delete deprecated tables
* DENG-4835 delete deprecated tables
* DENG-4835 Remove a deleted table from shredder processing
2024-11-09 18:51:07 +00:00
d0e6d6c1ee
Initiate backfill of `subscription_platform_derived.daily_active_logical_subscriptions_v1`. ( #6462 )
2024-11-08 23:09:41 +00:00
ac0d6e1670
Initiate backfill of `subscription_platform_derived.logical_subscription_events_v1`. ( #6461 )
2024-11-08 23:09:18 +00:00
e0a9f72379
Complete backfill of `subscription_platform_derived.stripe_subscriptions_revised_changelog_v1`. ( #6464 )
2024-11-08 20:12:52 +00:00
1f66c24445
Add logging to stage clean and fix --delete-expired ( #6466 )
2024-11-08 18:16:01 +00:00
68ae66a85d
Update warn checks for net_thunderbird_android_daily_derived ( #6457 )
...
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-11-08 18:09:41 +00:00
3d5225bae1
Bump symbolic from 12.12.0 to 12.12.1 ( #6465 )
...
Bumps symbolic from 12.12.0 to 12.12.1.
---
updated-dependencies:
- dependency-name: symbolic
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-08 17:06:24 +00:00
b47710ac9f
Initiate backfill of `subscription_platform_derived.stripe_subscriptions_revised_changelog_v1`. ( #6460 )
2024-11-08 00:44:21 +00:00
6b18bcb287
Fix subscription changelog conflicts ( #6458 )
...
* Don't create `adjusted_subscription_end` changelogs that conflict with previous changelogs.
* Don't use unreliable invoice line item `plan_change_date` metadata.
* Guard against similar issues in `stripe_customers_revised_changelog_v1`.
* Don't include customer changes in `stripe_subscriptions_history_v2` after the subscription ends.
* Create `synthetic_plan_change` changelogs from subscription "Remaining time" invoice line items.
* Guard against conflicting changelog records in Stripe history ETLs.
* Add checks to catch any duplicate IDs.
2024-11-07 22:40:15 +00:00
c5920680f8
[DENG-6279] Deprecate legacy tables in org_mozilla_firefox_derived ( #6448 )
...
* [DENG-6279] Deprecate legacy tables in org_mozilla_firefox_derived
* Deprecate and unschedule org_mozilla_ios_firefox.unified_metrics
2024-11-07 19:08:06 +00:00
87674839eb
Bump google-auth from 2.35.0 to 2.36.0 ( #6456 )
...
Bumps [google-auth](https://github.com/googleapis/google-auth-library-python ) from 2.35.0 to 2.36.0.
- [Release notes](https://github.com/googleapis/google-auth-library-python/releases )
- [Changelog](https://github.com/googleapis/google-auth-library-python/blob/main/CHANGELOG.md )
- [Commits](https://github.com/googleapis/google-auth-library-python/compare/v2.35.0...v2.36.0 )
---
updated-dependencies:
- dependency-name: google-auth
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-07 16:35:51 +00:00
23e69328d5
Deprecate views associated with already deprecated tables ( #6453 )
2024-11-06 19:30:18 +00:00
ed7fc6b151
Update bigconfig.yaml to use tag_deployments ( #6449 )
2024-11-06 17:29:51 +00:00
f666107095
Bump mkdocs-material from 9.5.43 to 9.5.44 ( #6452 )
...
Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material ) from 9.5.43 to 9.5.44.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases )
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG )
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.5.43...9.5.44 )
---
updated-dependencies:
- dependency-name: mkdocs-material
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-06 17:08:21 +00:00
69aba6a473
[DENG-5962] Change shredder state to only use end_date ( #6450 )
2024-11-06 15:19:10 +00:00
221f44ca00
update telemetry for list cards ( #6445 )
...
* update telemetry for list cards
* update test yaml
* fix incorrect bool
---------
Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>
2024-11-05 23:37:50 +00:00
4ad4c6b9d4
Add CLI command to run Bigeye checks ( #6383 )
...
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Reformat
* Add tests
* Address review feedback
* Add flags for deleting Bigeye checks
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Add run command to trigger Bigeye checks
* Use bigquery_bigeye_check to trigger Bigeye runs
* Add unit tests for monitoring run CLI command
* Update DAG tests
* Fix imports
* Address review feedback
* Format custom sql rules file
* Remove bigconfig_custom_rules.sql
* Fix bigeye update for public datasets
2024-11-05 18:02:42 +00:00
6e1e917317
Move event_monitoring_aggregates_v1 to run in on-demand billing ( #6447 )
2024-11-05 17:12:01 +00:00
b114127264
Exclude Fivetran's incorrect Stripe customer discount records from non-prod SubPlat ETLs. ( #6444 )
2024-11-05 16:52:41 +00:00
695ec38dad
Create iOS_onboarding.toml ( #6316 )
...
* Create iOS_onboarding.toml
Added iOS_onboarding.toml - query for generating the underlying table in the iOS onboarding dashboard.
* Update iOS_onboarding.toml
* feat: remove locale from dimension list, it does not exist in the source table
* Delete sql_generators/funnels/configs/iOS_onboarding.toml
* Create ios_onboarding.toml
updated to lower case file name and removed "locale" since it doesn't exist in firefox_ios_clients
* add a line in bqetl_project.yaml to avoid dry run
---------
Co-authored-by: kik-kik <kignasiak@mozilla.com>
2024-11-05 16:47:22 +00:00
265ba349e3
feat: remove check causing a lot of noise that is unactionable from app_store_funnel_v1 ( #6446 )
2024-11-05 14:26:40 +00:00
73f47fc0e3
Exclude Fivetran's incorrect Stripe customer discount records from SubPlat ETLs. ( #6443 )
2024-11-05 02:50:03 +00:00
76351c7cfb
Update the `country_codes_v1.mozilla_vpn_available` column for VPN wave 7 expansion countries (DENG-5882). ( #6442 )
2024-11-05 02:47:30 +00:00
ecacb00e93
Update VPN ETL `in_available_geos` logic for when VPN became available in various countries (DENG-5484) ( #6441 )
...
* Add VPN wave 6 countries to `in_available_geos` logic.
* Add VPN wave 7 countries to `in_available_geos` logic.
2024-11-04 23:44:34 +00:00
a562844c2a
Add submission_date partition to ref table number in bigquery_usage ( #6440 )
2024-11-04 22:06:10 +00:00
68cc748f90
Add reference table number to bigquery_usage view ( #6439 )
2024-11-04 19:28:19 +00:00
a264f0130a
Skip most of the Mozilla Social checks ( #6438 )
2024-11-04 18:15:09 +00:00
d9f0712d07
chore: switch to v2 workgroups for dataplatform access ( #6368 )
2024-11-04 18:12:47 +00:00
209b75ae57
Bigeye - Add support for deploying and removing custom SQL rules ( #6379 )
...
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Reformat
* Add tests
* Address review feedback
* Add flags for deleting Bigeye checks
* Fix formatting
* Fix tests
2024-11-04 17:41:46 +00:00
1e57300de4
Add time partitioning spec to mobile_search_clients_daily_historical_pre202408 ( #6437 )
2024-11-01 21:36:15 +00:00
835081f55d
Add and update VPN VAT rates (DENG-2707) ( #6423 )
...
* Update existing VAT rates.
* Add new VAT rates.
2024-11-01 18:39:47 +00:00
Alekhya
Sean Rose
dependabot[bot]
Katie Windau
Ben Wu
ryboland
Anna Scholtz
m-d-bowerman
rzhao
kik-kik
whd