23e69328d5
Deprecate views associated with already deprecated tables ( #6453 )
2024-11-06 19:30:18 +00:00
ed7fc6b151
Update bigconfig.yaml to use tag_deployments ( #6449 )
2024-11-06 17:29:51 +00:00
f666107095
Bump mkdocs-material from 9.5.43 to 9.5.44 ( #6452 )
...
Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material ) from 9.5.43 to 9.5.44.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases )
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG )
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.5.43...9.5.44 )
---
updated-dependencies:
- dependency-name: mkdocs-material
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-06 17:08:21 +00:00
69aba6a473
[DENG-5962] Change shredder state to only use end_date ( #6450 )
2024-11-06 15:19:10 +00:00
221f44ca00
update telemetry for list cards ( #6445 )
...
* update telemetry for list cards
* update test yaml
* fix incorrect bool
---------
Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>
2024-11-05 23:37:50 +00:00
4ad4c6b9d4
Add CLI command to run Bigeye checks ( #6383 )
...
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Reformat
* Add tests
* Address review feedback
* Add flags for deleting Bigeye checks
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Add run command to trigger Bigeye checks
* Use bigquery_bigeye_check to trigger Bigeye runs
* Add unit tests for monitoring run CLI command
* Update DAG tests
* Fix imports
* Address review feedback
* Format custom sql rules file
* Remove bigconfig_custom_rules.sql
* Fix bigeye update for public datasets
2024-11-05 18:02:42 +00:00
6e1e917317
Move event_monitoring_aggregates_v1 to run in on-demand billing ( #6447 )
2024-11-05 17:12:01 +00:00
b114127264
Exclude Fivetran's incorrect Stripe customer discount records from non-prod SubPlat ETLs. ( #6444 )
2024-11-05 16:52:41 +00:00
695ec38dad
Create iOS_onboarding.toml ( #6316 )
...
* Create iOS_onboarding.toml
Added iOS_onboarding.toml - query for generating the underlying table in the iOS onboarding dashboard.
* Update iOS_onboarding.toml
* feat: remove locale from dimension list, it does not exist in the source table
* Delete sql_generators/funnels/configs/iOS_onboarding.toml
* Create ios_onboarding.toml
updated to lower case file name and removed "locale" since it doesn't exist in firefox_ios_clients
* add a line in bqetl_project.yaml to avoid dry run
---------
Co-authored-by: kik-kik <kignasiak@mozilla.com>
2024-11-05 16:47:22 +00:00
265ba349e3
feat: remove check causing a lot of noise that is unactionable from app_store_funnel_v1 ( #6446 )
2024-11-05 14:26:40 +00:00
73f47fc0e3
Exclude Fivetran's incorrect Stripe customer discount records from SubPlat ETLs. ( #6443 )
2024-11-05 02:50:03 +00:00
76351c7cfb
Update the `country_codes_v1.mozilla_vpn_available` column for VPN wave 7 expansion countries (DENG-5882). ( #6442 )
2024-11-05 02:47:30 +00:00
ecacb00e93
Update VPN ETL `in_available_geos` logic for when VPN became available in various countries (DENG-5484) ( #6441 )
...
* Add VPN wave 6 countries to `in_available_geos` logic.
* Add VPN wave 7 countries to `in_available_geos` logic.
2024-11-04 23:44:34 +00:00
a562844c2a
Add submission_date partition to ref table number in bigquery_usage ( #6440 )
2024-11-04 22:06:10 +00:00
68cc748f90
Add reference table number to bigquery_usage view ( #6439 )
2024-11-04 19:28:19 +00:00
a264f0130a
Skip most of the Mozilla Social checks ( #6438 )
2024-11-04 18:15:09 +00:00
d9f0712d07
chore: switch to v2 workgroups for dataplatform access ( #6368 )
2024-11-04 18:12:47 +00:00
209b75ae57
Bigeye - Add support for deploying and removing custom SQL rules ( #6379 )
...
* Add support for defining custom SQL rules
* Added monitoring rollback command
* Reformat
* Add tests
* Address review feedback
* Add flags for deleting Bigeye checks
* Fix formatting
* Fix tests
2024-11-04 17:41:46 +00:00
1e57300de4
Add time partitioning spec to mobile_search_clients_daily_historical_pre202408 ( #6437 )
2024-11-01 21:36:15 +00:00
835081f55d
Add and update VPN VAT rates (DENG-2707) ( #6423 )
...
* Update existing VAT rates.
* Add new VAT rates.
2024-11-01 18:39:47 +00:00
231c99d013
Specify partition column for Bigeye monitoring for ssl_ratios ( #6436 )
...
`ssl_ratios_v1` is actually a view (pointing to the public `ssl_ratios_v1` table), and Bigeye requires a partition column to be configured explicitly on views. This should fix the failing Airflow deployment task
2024-11-01 16:58:20 +00:00
2a67a3a66f
Bump mkdocs-material from 9.5.42 to 9.5.43 ( #6434 )
...
Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material ) from 9.5.42 to 9.5.43.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases )
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG )
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.5.42...9.5.43 )
---
updated-dependencies:
- dependency-name: mkdocs-material
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-01 15:12:44 +00:00
cb72ce83bc
Bump google-cloud-datacatalog-lineage from 0.3.9 to 0.3.10 ( #6435 )
...
Bumps [google-cloud-datacatalog-lineage](https://github.com/googleapis/google-cloud-python ) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/googleapis/google-cloud-python/releases )
- [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md )
- [Commits](https://github.com/googleapis/google-cloud-python/compare/google-cloud-datacatalog-lineage-v0.3.9...google-cloud-datacatalog-lineage-v0.3.10 )
---
updated-dependencies:
- dependency-name: google-cloud-datacatalog-lineage
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-01 15:12:21 +00:00
c7521153be
CLI command to automate some migration of ETL checks to Bigeye ( #6362 )
...
* Add command to automate migration of ETL checks
* Migrate ETL checks for ssl_ratios
* Update bigconfigs
2024-11-01 12:27:57 +00:00
f8b7cfc11d
Remove duplicate currencies from `mozilla_vpn_derived.exchange_rates_v1` ETL config (DENG-5883). ( #6433 )
2024-10-31 23:59:30 +00:00
059572df0c
Update `mozilla_vpn_derived.exchange_rates_v1` to include VPN wave 7 expansion country currencies (DENG-5883). ( #6432 )
2024-10-31 22:19:08 +00:00
10f15c4fa4
Change Stripe tax transaction exchange rate to BIGNUMERIC type. ( #6428 )
...
Some historical Stripe tax transactions contain exchange rates with 16 digits after the decimal, which exceeds the normal NUMERIC type's maximum scale of 9 digits after the decimal.
2024-10-31 21:05:02 +00:00
70b794b434
Populate empty query.py in stage deploy instead of CI config ( #6430 )
2024-10-31 20:50:22 +00:00
7f72235eb4
Switch to baseline beginning Aug 01st 2024 ( #6425 )
...
* Switch to baseline beginning Aug 01st 2024
* Modify the search_engine_daily view to handle the cut off
* Fix sql format
* Skip mobile_clients_daily_v2 from dry run
* Remove sql_generator files populating v1
* Add tests for mobile_search_clients_daily_v2
* remove unwanted tests
2024-10-31 19:11:36 +00:00
381107db71
Revert #6418 ( #6429 )
2024-10-31 15:41:23 +00:00
759cbe3231
Fix backfill validation tests in CI ( #6398 )
...
* Fix logging in backfill validation
* validate backfill should fail
* validate backfill should still fail
* revert entry, move validate-backfills to after generate-sql
2024-10-30 20:25:41 +00:00
b68db470bd
Sort query references result when parallelism > 0 ( #6427 )
2024-10-30 20:25:31 +00:00
922f5ee70f
Fully-qualify INFORMATION_SCHEMA datasets in ./bqetl view clean ( #6426 )
2024-10-30 20:01:27 +00:00
d486bed1b9
Use INFORMATION_SCHEMA to get existing views for cleaning ( #6424 )
...
* Use INFORMATION_SCHEMA to get existing views for cleaning
* Update _list_managed_views caller
2024-10-30 18:32:01 +00:00
44cb706ff7
remove provider- from engine strings ( #6419 )
2024-10-30 15:59:19 +00:00
26adc67513
Fix materialized views failing stage deploy ( #6421 )
...
* Create test materialized view
* fix copy_sql_to_tmp_dir renaming
* Add materialized views to view dependencies
* remove test files
2024-10-30 15:33:28 +00:00
ca17fa2e91
New view mobile search clients daily ( #6370 )
...
* Add new view for mobile_search_clients_daily
Delete sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/backfill.yaml
Fix format issue
* Update view.sql
* Fix query sql
Fix query.sql format
Fix query sql
* Add medata yaml file
* Fix metadata.yaml file issue
* Delete ]
* Update metadata.yaml
* Update metadata.yaml
* Update metadata.yaml
* Fix schema.yaml
* Fix file
* Fix deploy stage issues
* Fix sql
* Fix sql
* Fix metadata.yaml
---------
Co-authored-by: Alekhya <alekhya@Mac.lan>
2024-10-30 15:33:19 +00:00
2add865249
Speed up schema updates ( #6418 )
...
* Parallelize dependency graph
* Use GCP API to get table schema when not using cloud function
* Reuse GCP credentials
* Update dependency tests
* Remove print
2024-10-30 14:52:05 +00:00
5ce6152d0b
Bump siggen from 2.1.20240726 to 2.2.20241029 ( #6420 )
...
Bumps [siggen](https://github.com/willkg/socorro-siggen ) from 2.1.20240726 to 2.2.20241029.
- [Changelog](https://github.com/willkg/socorro-siggen/blob/main/HISTORY.rst )
- [Commits](https://github.com/willkg/socorro-siggen/compare/v2.1.20240726...v2.2.20241029 )
---
updated-dependencies:
- dependency-name: siggen
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-30 14:49:29 +00:00
a270272b38
Add line to empty query.py in CI copy_debug_sql ( #6417 )
...
* Add line to empty query.py in CI copy_debug_sql
* ooooooop
* remove test file
2024-10-29 20:40:37 +00:00
6c87dfb547
Reuse GCP API credentials for view updates ( #6415 )
...
* Reuse GCP API credentials for view updates
* isort
2024-10-29 15:31:29 +00:00
35c92f086c
Filter overactive clients for `baseline_clients_daily` and `sponsored_tiles_clients_daily` ( #6413 )
...
* Filter overactive clients from baseline_clients_daily
* Filter overactive clients for sponsored_tiles_clients_daily
* Increase sponsored_tiles_clients_daily overactive threshold
* Update sql/moz-fx-data-shared-prod/telemetry_derived/sponsored_tiles_clients_daily_v1/query.sql
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
---------
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-10-28 20:37:25 +00:00
af40ecf372
complete backfill ( #6407 )
2024-10-28 20:03:26 +00:00
7386ab9574
[DENG-5931] Add partitioning to experiment_events_live_v1 ( #6414 )
2024-10-28 19:59:18 +00:00
a37c4c9088
Update to US only and last 28 days only ( #6412 )
2024-10-28 19:06:22 +00:00
3b222a21e9
Parallelize metadata publish ( #6403 )
2024-10-28 19:03:08 +00:00
f8fa9ef3a4
Speed up view deploys by using processing pool ( #6401 )
2024-10-28 19:02:45 +00:00
03e88b1875
Skip updating schemas of tables that are skipped for deploys ( #6410 )
2024-10-28 18:25:54 +00:00
1963d52aa5
DENG-5757 add new summary view of metadata completeness by project/schema ( #6411 )
...
* DENG-5757 add new summary view of metadata completeness by project & dataset
* DENG-5757 add new summary view of metadata completeness by project & dataset
2024-10-28 18:18:30 +00:00
7368691016
Use upstream main commit for CI sql diff ( #6408 )
...
* trivial sql diff
* Use latest upstream main commit for sql diff
* revert sql
2024-10-28 17:57:23 +00:00
Katie Windau
Anna Scholtz
dependabot[bot]
Ben Wu
m-d-bowerman
Sean Rose
rzhao
kik-kik
whd
Alekhya