Граф коммитов

136 Коммитов

Автор SHA1 Сообщение Дата
Ben Wu 037eccc547
Remove row count check for net_thunderbird_android_derived.baseline_clients_last_seen_v1 (#6515) 2024-11-18 20:13:02 +00:00
Anna Scholtz 68ae66a85d
Update warn checks for net_thunderbird_android_daily_derived (#6457)
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-11-08 18:09:41 +00:00
rzhao 695ec38dad
Create iOS_onboarding.toml (#6316)
* Create iOS_onboarding.toml

Added iOS_onboarding.toml - query for generating the underlying table in the iOS onboarding dashboard.

* Update iOS_onboarding.toml

* feat: remove locale from dimension list, it does not exist in the source table

* Delete sql_generators/funnels/configs/iOS_onboarding.toml

* Create ios_onboarding.toml

updated to lower case file name and removed "locale" since it doesn't exist in firefox_ios_clients

* add a line in bqetl_project.yaml to avoid dry run

---------

Co-authored-by: kik-kik <kignasiak@mozilla.com>
2024-11-05 16:47:22 +00:00
Anna Scholtz a264f0130a
Skip most of the Mozilla Social checks (#6438) 2024-11-04 18:15:09 +00:00
Alekhya 7f72235eb4
Switch to baseline beginning Aug 01st 2024 (#6425)
* Switch to baseline beginning Aug 01st 2024

* Modify the search_engine_daily view to handle the cut off

* Fix sql format

* Skip mobile_clients_daily_v2 from dry run

* Remove sql_generator files populating v1

* Add tests for mobile_search_clients_daily_v2

* remove unwanted tests
2024-10-31 19:11:36 +00:00
Anna Scholtz b06c2836ae
Skip list for schema deploys (#6404) 2024-10-25 22:46:57 +00:00
whd cfdc045ff5
chore: skip publishing search terms view that shouldn't actually exist (#6402) 2024-10-25 18:37:51 +00:00
Katie Windau 4e42ac7bcb
feat(DENG-5757): Create new table monitoring_derived.metadata_completeness_v1 (#6397)
* Create new table for tracking improvements to metadata completeness

* shorten metadata description length

* Weekly DAG

* Update start date

* Fix schema.yaml file to remove repeated entry

* Exclude looker tmp tables

* Exclude looker temp tables and skip dry run
2024-10-25 14:08:37 +00:00
Ben Wu d7c8a12d9e
Bug 1905938 Support events with no metrics in glean_usage generator (#6358)
* Bug 1905938 Support events with no metrics in glean_usage generator

* fix event_error_monitoring

* not

* Apply suggestions from code review

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>

---------

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2024-10-16 15:30:28 -07:00
Alekhya deae84e61e
Skip generating org_mozilla_ios_focus_derived/baseline_clients_last_s… (#6320)
* Skip generating org_mozilla_ios_focus_derived/baseline_clients_last_seen_v1/metadata.yaml

* Update bqetl_project.yaml

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-10-09 16:09:03 +00:00
Anna Scholtz a1a12791aa
Update Bigeye warehouse ID (#6297)
* Update Bigeye warehouse ID

* Update test bigeye warehouse ID
2024-10-03 21:12:24 +00:00
kik-kik 4257698cc8
feat(DENG-4602): add BigEye RunMetricOperator to DAG generation (#6285)
* feat: add BigEye RunMetricOperator to DAG generation

* Fix Bigeye DAG generation and move configuration to bqetl_project.yaml

* Reformatting; test fixing

* Install Airflow override requirements

---------

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-10-03 15:13:27 +00:00
Ben Wu a9c904df37
Give reader role to dry run account for datasets in stage (#6288)
* Give reader role to dry run account for datasets in stage

* add sql changes to deploy

* hardcode service accounts

* Revert "add sql changes to deploy"

* Remove func in dryrun

* f
2024-10-02 21:55:19 +00:00
Katie Windau 298e3ce63c
DSRE-1681 Create 2 new SUMO Google Analytics views in shared prod (#6278)
* DSRE-1681 create 2 new views in shared prod

* DSRE-1681 fix formatting on new views

* DSRE-1681 add metadata yaml files for the 2 new views with authorized true

* DSRE-1681 add SUMO access to the 2 new views

* Add better descriptions

* DSRE-1681 fix friendly names

* DSRE-1681 add the _TABLE_SUFFIX filters so it doesn't pull in data from the other tables with the same prefix, i.e. events_intraday_*

* DSRE-1681 Remove table-level access in favor of dataset access

* DSRE-1681 Add new sumo_ga views to dryrun skip

---------

Co-authored-by: Wesley Dawson <whd@mozilla.com>
2024-10-01 03:50:05 +00:00
Ben Wu d9b1c5f2db
Add missing column views to dry run skip (#6266) 2024-09-26 15:42:50 +00:00
Curtis Morales 247729d217
In validation, only read schema files when necessary (#6252)
* Remove deleted table from skip list

* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl

* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl

* Clean up test
2024-09-24 17:03:28 +00:00
Curtis Morales 479182a2e1
Desktop tiles forecast inputs v2 (#6230)
* Bump table to v2

* Delete v1 table

* Make table a standard monthly table

* Update dryrun skip list

* Add date_partition_parameter field
2024-09-23 14:47:58 +00:00
Curtis Morales cebe9ca32b
Fix desktop_tiles_forecast_inputs submission_month parameter and move query to dryrun skip list (#6227)
* Fix desktop_tiles_forecast_inputs submission_month parameter and move query to dryrun skip list

* Update sql/moz-fx-data-shared-prod/ads_derived/desktop_tiles_forecast_inputs_v1/metadata.yaml

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

* Update sql/moz-fx-data-shared-prod/ads_derived/desktop_tiles_forecast_inputs_v1/metadata.yaml

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

---------

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-09-19 20:56:38 +00:00
Arkadiusz Komarzewski c53af44abd
Allow skipping apps in event monitoring generator
Also skip ads_backend because its dataset is restricted.
2024-09-12 17:50:07 +02:00
akkomar 0842f32d8b
Unskip ads_backend in glean_usage generator (#6180)
This will generate events_stream and event_monitoring_live tables for ads_backend. We originally skipped this app because there was a delay in its ping table deployment.
2024-09-12 14:13:46 +02:00
Abhishek Aggarwal b5f4adb19e
DENG-4691: Add view for fakespot cost data in dataservices prod project (#6163)
* DENG-4691: View for fakespot cost data in "dataservices-high-prod" project
    - Only Kuberay created Ray serve app costs are included
2024-09-09 16:42:54 +02:00
Abhishek Aggarwal b57456864f
DENG-4691 Add views for Ray Serve compute costs for fakespot tenant (#6146)
* DENG-4691 Add views for Ray Serve compute costs for fakespot tenant

  - To surface the total compute cost of all Ray Serve workloads of fakespot tenant
     that are deployed via kuberay operator on "dataservices-high-nonprod"
     shared GKE cluster
2024-09-03 15:32:43 +02:00
Ben Wu 119b5f08d1
Create shredder job stats tables and views (#6115) 2024-08-27 18:52:34 +01:00
Anna Scholtz d077ea046e
Limit the projects scanned for routines (#6101) 2024-08-26 15:28:49 -07:00
Curtis Morales 7127641f7c
AD-437 Add PPA measurements w/o advertiser view (#6006)
* Add PPA measurements w/o advertiser view

* Update metadata

* Add new PPA view to dry run skip list

* Should be an authorized view

* Rename table to _limited
2024-08-02 17:40:50 -04:00
Ben Wu 73f0f49df7
[DENG-577] Create monitoring table for stable table expiration (#6009) 2024-08-02 16:53:34 -04:00
Chelsea Troy 88cc1e6509 Skip dry run for these tables 2024-07-25 10:37:17 -05:00
Eduardo Filho b994884098
GLAM purge percentile calculations and prep downstream (#5966)
* Remove percentiles

* Remove tests that test percentiles

* Refresh scripts insert null to new percentiles

* Remove percentile columns from queries and schemas

* Delete more percentile tables

* Formatting

* histogram_cast_struct's keys are strings

* Re-add test after fixing failure cause
2024-07-25 10:44:43 -04:00
Eduardo Filho d0b95438c3
GLAM: Do legacy non-normalized aggregations in the same pass as normalized (#5935)
* GLAM: Do legacy non-normalized aggregations in the same pass as normalized

* Add missing eof space

* Fix tests
2024-07-18 13:16:06 -04:00
Katie Windau eb1dd14e59
DENG-4201 Create downloads_with_attribution_v1 and _v2 in shared prod (to prep for removing from mktg prod) (#5931)
* DENG-4201 create DL w attr v1 and v2 in shared prod

* Update to new paths in shared prod

* DENG-4201 update to v2 version in view.sql

* Update metadata.yaml

* Update metadata.yaml

* Update bqetl_project.yaml
2024-07-17 14:32:06 -05:00
kik-kik 63e507998e
feat: Convert app_store_external to app_store_syndicate and add web_referrer views (#5773)
* feat: add corresponding firefox ios web_referrer tables to be copied over to moz-prod

* feat: convert app_store_external to be a syndicate dateset

* fix: correct typo in the description anx syndication configuration.
2024-07-02 12:05:52 +02:00
akkomar e4a5e3bbcf
DENG-4047 Don't wrap metrics in JSON in accounts events stream tables (#5873) 2024-07-02 11:18:52 +02:00
Curtis Morales 0373f48231
PPA measurements view (#5858)
* PPA measurements view

* Fix CI and updated description
2024-06-28 19:09:33 -04:00
Ben Wu 3da6b38e7f
[Bug 1905444] Add monitor_backend to glean_usage skip list (#5870) 2024-06-28 18:21:31 -04:00
Anna Scholtz 9c6f5a9e58
Remove files from format skip list (#5863) 2024-06-27 15:17:45 -07:00
Anna Scholtz 57bd939905
Fully qualified identifiers in SQL queries (#5764)
* Add fully-qualified identifiers when formatting queries

* Fully-qualified identifiers for queries in sql/

* Check in only formatted SQL to generated-sql branch

* Add comment

* Fully qualify more tables

* Fully qualify test files

* Formatting improvements around CTEs and unit tests

* Option to skip auto qualifying queries
2024-06-27 09:53:33 -07:00
Alekhya efc6aaced3
Add normalized_app_name_os to mobile search aggregate table (#5850)
* Add normalized_app_name_os to mobile search aggregate table

* Fix CI

Fix CI
2024-06-27 10:58:46 -04:00
akkomar 66d1c7aaf6
Bug 1901844 - Skip ads_backend in glean_usage generator (#5832) 2024-06-27 13:05:01 +02:00
Anna Scholtz 992a15e2fe
[Bug 1903788] Update checks for reference_browser_derived.baseline_clients_last_seen_v1 (#5842) 2024-06-26 08:55:37 -07:00
Sean Rose d5422b049f
Move Google Search Console ETLs & views to `moz-fx-data-shared-prod` (DENG-1733) (#5822)
* Move Google Search Console ETLs & views to `moz-fx-data-shared-prod`.

* Temporarily add back Google Search Console views in `moz-fx-data-marketing-prod` to avoid breaking existing reporting.

* Update Google Search Console ETL entries in dry-run skip config.

* Temporarily add pass-through views in `moz-fx-data-marketing-prod` in place of Google Search Console ETL tables to avoid breaking existing reporting.
2024-06-21 11:49:42 -07:00
akkomar fe03b842f2
DENG-2407 Use standard event pings for accounts in event monitoring queries (#5801) 2024-06-19 17:32:45 +02:00
Anna Scholtz aaf75a37de
Skip fenix_derived.android_onboarding_v1 dryruns (#5703) 2024-06-07 08:43:32 -07:00
Anna Scholtz 979fb253cd
[DENG-3417] Get event pings from probe info for monitoring events (#5576)
* Get event pings from probe info for monitoring events

* Update event monitoring aggregate query

* Review feedback
2024-05-16 13:06:31 -07:00
Katie Windau 6108d64d37
DENG-3186 Update shredder config to match updated table name (#5572)
* DENG-3186 update shredder config to match updated table name

* DENG-3186 remove dryrun skip
2024-05-14 13:24:24 -05:00
Katie Windau bab46f14c8
DENG-3186 skip dryrun for desktop engagement clients V1 (#5571) 2024-05-14 10:59:38 -05:00
Ben Wu 74c65529fa
[DENG-3720] Skip relay_backend in glean_usage generator (#5556) 2024-05-10 15:49:13 -04:00
wil stuckey bd53897761
DENG-3606 Add fakespot product views to user facing 'fakespot' dataset (#5522)
* DENG-3606 Add fakespot product views to user facing 'fakespot' dataset

* Format sql

* skip dry run
2024-05-09 15:33:56 -05:00
Katie Windau fc45d45ae3
DENG-2262 Update desktop installs to no longer rely on a GA3 table (#5444)
* DENG-2262 update to no longer rely on GA3 table
2024-04-26 11:36:40 -05:00
Eduardo Filho 974312945e
fix: GLAM checks to prevent skipping days of data (#5366)
* fix: GLAM checks to prevent skipping days of data

* format sql

* Skip dryrun
2024-04-16 11:08:47 -04:00
Anna Scholtz eac0ac80c2
Remove telemetry_derived init.sql files (#5342)
* Remove init.sql files for telemetry_derived queries

* Remove init.sql for events_daily

* Remove init.sql from skip lists

* Remove init.sql references from tooling

* Add schema for baseline_clients_first_seen
2024-04-10 15:36:30 -07:00