Граф коммитов

120 Коммитов

Автор SHA1 Сообщение Дата
Katie Windau ae96d4e981
DENG-4152 - move GA4 tables (#6136)
and views to shared-prod

Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-09-06 10:39:55 -07:00
Sean Rose c4bdac3f32
Remove Google Search Console placeholder views from the `moz-fx-data-marketing-prod` project (DENG-1733). (#6028) 2024-08-09 16:05:15 -07:00
Sean Rose 40ed1d24f5
Move limited historical Google Search Console data into separate views (DENG-4329) (#6027)
* Exclude limited historical Google Search Console data from the existing views.

* Add views for limited historical Google Search Console data.
2024-08-07 14:09:03 -07:00
Marlene Hirose 9de2520821
copy code for metadata, query, schema from marketing-prod to shared-prod (#6026)
* copy code for metadata, query, schema from marketing-prod to shared-prod

* refactor metadata to take out depends_on_table section
2024-08-07 11:56:34 -07:00
Marlene Hirose ea251b9db0
move depends_on_tables_existing to shared_prod.www_site_metrics_summary_v2 from marketing_prod as we will turn off marketing_prod dag in future (#6022) 2024-08-07 09:40:09 -07:00
Katie Windau c037e4fbb4
DENG-4201 Remove GA3 tables from marketing-prod (they are now in shared-prod instead) (#5917)
* DENG-4201 remove blogs_goals_v1 since now in shared prod

* DENG-4201 remove blogs_sessions_v1 from mktg prod since now in shared prod

* DENG-4201 remove www_site_hits_v1 from mktg prod since now in shared prod

* DENG-4201 remove firefox_whatsnew_summary_v1 from mktg prod since now in shared prod

* DENG-4201 remove www_site_downloads_v1 from mktg prod since now in shared prod

* DENG-4201 remove www_site_events_metrics_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove www_site_landing_page_metrics_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove www_site_page_metrics_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove blogs_daily_summary_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove blogs_landing_page_summary_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove blogs_empty_check_v1

* DENG-4201 remove www_site_empty_check_v1

* DENG-4201 remove www_site_metrics_summary_v1 from mktg prod since now moved to shared prod

* DENG-4201 remove dl with attr v1 and v2 from mktg prod since now in shared prod instead

* DENG-4201 remove tests from mktg prod since tables no longer live in mktg prod
2024-07-23 19:01:15 -05:00
Katie Windau 3c393f27dd
Add 1 more column to downloads_v2 (#5861)
* Add new column 'campaign_from_event_params' to schema.yaml

* Updated query
2024-06-27 16:09:57 -05:00
Katie Windau 62414d7dc2
DENG-2492 Deprecate GA3 tables since GA3 feed ending 2024-06-30 (#5848) 2024-06-26 14:39:04 -05:00
Katie Windau 601e3047f2
Add new columns to www_site_hits_v2 & www_site_downloads_v2 (#5839)
* Add new columns to www_site_hits_v2

* Add new cols to www_site_downloads_v2

* Remove comments

* Update column descriptions to clarify which are CTS vs TS
2024-06-25 15:54:20 -05:00
Sean Rose d5422b049f
Move Google Search Console ETLs & views to `moz-fx-data-shared-prod` (DENG-1733) (#5822)
* Move Google Search Console ETLs & views to `moz-fx-data-shared-prod`.

* Temporarily add back Google Search Console views in `moz-fx-data-marketing-prod` to avoid breaking existing reporting.

* Update Google Search Console ETL entries in dry-run skip config.

* Temporarily add pass-through views in `moz-fx-data-marketing-prod` in place of Google Search Console ETL tables to avoid breaking existing reporting.
2024-06-21 11:49:42 -07:00
Sean Rose 6c52d395b8
Minor Google Search Console revisions (DENG-1733) (#5820)
* Add "foxfire" as a Firefox brand keyword for Google Search Console.

* Correct `localized_site` column description.
2024-06-20 16:17:41 -07:00
Sean Rose aa51c96695
More Google Search Console revisions (DENG-1733) (#5587)
* Remove `localized_site_country_code` and `localized_site_country` columns from `search_impressions_by_page` view.

Because they're so sparsely populated that it can be misleading.

* Add `user_region` and `user_subregion` columns to Google Search Console views.

* Change `classify_site_query` UDF to differentiate Firefox, Pocket, and Mozilla brand keywords.

* Use new brand keyword regular expressions from Adria Kyne in `classify_site_query` UDF.
2024-05-15 14:48:46 -07:00
Sean Rose 079fbe2188
Google Search Console revisions (DENG-1733) (#5424)
* Classify queries in Google Search Console data.

* Refactor repeated Google Search Console logic into UDFs.

* Replace `page_path_segment_1` field with `localized_site_code` field, and add related derived fields.

* Rename `country_code` field `user_country_code`.

* Add `user_country` field.

* Include Google Search Console data for MDN.
2024-04-25 14:35:31 -07:00
Katie Windau a1e62a13f3
DENG-3370 update schema for www_site_metrics_summary_v1 to include 3 new columns (#5355) 2024-04-10 07:40:08 -05:00
Katie Windau 9f735c7600
DENG-3367 add partnership udf logic to views (#5353)
* DENG-3355 add funnel_derived_clients UDF to clients_first_seen_28_days_later view

* DENG-3355 add distribution model to clients first seen 28 days later

* DENG-3355 add partner org to clients first seen 28 days later

* DENG-3355 add the UDFs to clients_first_seen

* DENG-3355 remove fully qualified project name from the UDFs

* DENG-3355 add fully qualified project name to UDFs

* DENG-3355 add partnership logic to GA4 site metrics summary

* DENG-3355 add stub implementation of funnel_derived_ga_metrics UDF

* DENG-3355 add stub implementation test for distribution_model_ga_metrics

* DENG-3355 add stub tests for partner_org_ga_metrics UDF

* DENG-3355 add stub test for funnel_derived_clients UDF

* DENG-3355 add stub implementation test for distribution_model_clients

* DENG-3355 add stub implementation test for partner_org_clients

* DENG-3355 fix formatting of stub funnel_derived_ga_metrics

* DENG-3355 fix formatting of metadata yaml files to include newlines

* DENG-3355 fix format of partner org clients metadata.yaml

* DENG-3355 add comment to partner org clients udf.sql

* Update sql/moz-fx-data-shared-prod/udf/distribution_model_ga_metrics/udf.sql

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

* DENG-3367 remove unnecessary alias in www_site_metrics_summary view

---------

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-04-09 16:15:58 -05:00
Leli c57ef86a4f
Deng 3287 move acoustic data from marketing to shared prod (#5316)
* DENG-3151 create acoustic_external dataset in BigQuery

* DENG-3287 move data from marketing to shared-prod

* acoustic_derived dataset
2024-04-03 16:26:44 +02:00
Anna Scholtz 459c22e625
Remove init.sql files that can simply be replaced by schema.yaml (#5237) 2024-03-26 15:09:16 -07:00
Leli 009feaf699
fix: DENG-3161 add date_partition_parameter: null to suppression list DAG (#5258) 2024-03-21 18:25:31 +01:00
Leli 7057f83985
DENG-3161 create full suppression list Accoustic (#5246)
Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-03-20 17:25:55 +01:00
Katie Windau 9ed40453f1
DENG-3162 - limit 2 views to only go back to 10/1/23 (#5245)
* DENG-3162 - limit 2 views to only go back to 10/1/23

* DENG-3162 - add comments explaining the new filter
2024-03-20 10:25:33 -05:00
kik-kik c4ff6ae410
fix: incorrect type for event_timestamp raw_recipient_raw_v1 (#5234) 2024-03-19 15:42:05 +01:00
kik-kik 07266ec236
feat(): Acoustic configuration updated, init.sql files removed and user facing views defined (#5122)
* Acoustic configuration updated, init.sql files removed and user facing views defined

* added missing dataset_metadata for acoustic

* change access group to new braze wg

* fix workgroup

---------

Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>
2024-03-15 12:42:58 +01:00
Katie Windau dd7355f8da
DENG-2837 - add 3 new data checks (#5161) 2024-03-04 10:54:28 -06:00
Katie Windau 0c4652f0f2
DENG-2938 Update product download logic to limit to Firefox Desktop downloads only in www_site_landing_page_metrics_v2 (#5158)
* DENG-2938 - update schema column descriptions for product download columns

* DENG-2938 - update product download logic to be firefox desktop only

* DENG-2938 add backticks around date
2024-03-01 14:55:23 -06:00
Katie Windau c3e4c5984d
DENG-2940 - add 2 new columns to www_site_hits_v2 (#5157) 2024-03-01 13:18:31 -06:00
Katie Windau b63c4e7b11
DENG-2924 - update product download logic (#5155)
* DENG-2924 - update product download logic

* DENG-2924 fix typo

* DENG-2919 switch to if per Kik's request

* DENG-2924 update alias name
2024-03-01 11:32:10 -06:00
Katie Windau 997788b4ec
DENG-2919 - add platform filter (#5145) 2024-02-29 16:40:34 -06:00
Katie Windau bf7fd41c2e
DENG-2919 Change logic for measuring product downloads (#5142)
* DENG-2919 - update www_site_metrics_summary_v2 logic to count only Desktop Firefox downloads towards downloads

* DENG-2919 wrap reserved col names with backticks
2024-02-29 13:45:13 -06:00
Winnie Chan 6c501a620c
Deng-2845: Remove default deprecated false (#5118)
* Removed deprecated field

* Removed deprecated false in metadata yaml

* Fixed test
2024-02-27 10:33:50 -08:00
Anna Scholtz e5a6c18070
Revert "DENG-2837 add data checks to new ga4 tables (#5102)" (#5117)
This reverts commit 74ddf001e9.
2024-02-26 11:53:53 -08:00
Katie Windau 74ddf001e9
DENG-2837 add data checks to new ga4 tables (#5102)
* DENG-2837 Add 2 new QA checks

* DENG-2837 add spaces to end of checks

* DENG-2837 add where clause to blogs_goals_v2 check

* DENG-2837 - add QA check for www_site_hits_v2

* DENG-2837 adding more checks and adding extra empty line

* DENG-2837 add not null check to ga_sessions_v2

* DENG-2837 add pattern check for visit identifier on blogs_goals_v2
2024-02-26 13:13:04 -06:00
Katie Windau a78e30c1f2
DENG-2784 add ga4 sensors (#5082)
* DENG-2784 - add wait task for blogs partitions to 4 blogs tables

* DENG-2784 - add sensors for events table for production property

* DENG-2784 - add sensor for production property for the 3 tables in mozilla org derived

* DENG-2784 fix backticks

* DENG-2784 - add spaces to jinja, also update gclid conv to ga4 dag

* DENG-2784 - leave Marlene's DAG where it is

* DENG-2784 - switch to depends_on

* DENG-2784 add dag names to depends_on

* DENG-2784 - remove some of the unneeded dependencies and change name from partition to table

* DENG-2784 switch from prod to wmo to be more clear

* DENG-2784 switch from prod to wmo to be more clear
2024-02-23 13:45:28 -06:00
Katie Windau 24eb991b08
DENG-2492 Create new GA4 derived table: blogs_landing_page_summary_v2 (#5073)
* DENG-2492 Initial commit for new table blogs_landing_page_summary_v2

* DENG-2492 add cleaned landing page logic

* DENG-2492 fix typo in query

* DENG-2492 add more explicit table references and rename stage table

* DENG-2492 include table aliases in group by

* DENG-2492 update the schema column descriptions

* DENG-2492 wrap date with backticks
2024-02-21 10:27:57 -06:00
Katie Windau cb0a34b53b
DENG-2492 update schema.yaml (#5078) 2024-02-21 09:15:33 -06:00
Katie Windau 3e4f72e378
DENG-2492 update column metadata (#5074) 2024-02-20 16:08:13 -06:00
Katie Windau b39d80f37c
DENG-2492 Create new GA4 derived table: blogs_daily_summary_v2 (#5070)
* DENG-2492 initial commit, work in progress

* DENG-2492 work in progress

* DENG-2492 work in progress

* DENG-2492 initial commit

* DENG-2492 update query

* DENG-2492 remove trailing space from metadata.yaml

* DENG-2492 add column descriptions to schema.yaml
2024-02-20 11:38:34 -06:00
Katie Windau efaf721743
DENG-2492 Create new GA4 derived table: blogs_goals_v2 (#5068)
* DENG-2492 - initial commit new blogs_goals_v2 GA4 table

* DENG-2492 add visit_identifier for clustering

* DENG-2492 rename column from share to social_share

* DENG-2492 update query.sql
2024-02-20 09:08:46 -06:00
Sean Rose b3625348bf
Correct `partition_id` expressions for Google Search Console table partition sensors. (#5058)
`macros.ds_add()` returns a string (not a datetime), and `ds_nodash` expects a datetime input.
2024-02-15 16:06:20 -08:00
Sean Rose 25e4c224a0
Support waiting for tables/partitions to exist before running ETLs (#5039)
* Implement `depends_on_tables_existing` and `depends_on_table_partitions_existing` scheduling metadata.

* Refactor repeated timedelta validation logic into `validate_timedelta_string` utility function.

* Replace Google Search Console empty-check ETLs with `depends_on_table_partitions`.
2024-02-15 14:38:52 -08:00
Katie Windau 28fbb554be
DENG-2492 - remove unnecessary filter (#5047) 2024-02-14 15:48:16 -06:00
Katie Windau d7ae3df07f
DENG-2492 update product download types (#5033) 2024-02-13 14:24:27 -06:00
Katie Windau a4512d9661
DENG-2492 update product dl types (#5032) 2024-02-13 13:45:28 -06:00
Katie Windau 85c29e9c46
DENG-2492 - update product download event types going forward (#5031)
* DENG-2492 - update product download event types going forward

* DENG-2492 re-order and simplify
2024-02-13 12:39:09 -06:00
Katie Windau b605cd9e26
DENG-2492 blogs sessions v2 (#5019)
* DENG-2492 filter to blog.mozilla.org only since new ID contains other domains also
2024-02-12 13:32:09 -08:00
Katie Windau ee8de94705
DENG-2492 Create new GA4 derived table: blogs_sessions_v2 (#5018)
* DENG-2492 initial commit for new table blogs_sessions_v2

* DENG-2492 wrap keywords with backticks
2024-02-12 10:54:41 -06:00
Katie Windau cfbf296f32
DENG-2492 create new GA4 derived table firefox_whatsnew_summary_v2 (#5006)
* DENG-2492 create new GA4 derived table firefox_whatsnew_summary_v2

* DENG-2492 shorten column descriptions

* DENG-2492 clear up visits column desc
2024-02-09 14:21:58 -06:00
Katie Windau 55eb0c8299
DENG-2492 fix offsets for page levels, they were all off by 1 (#5005) 2024-02-09 12:25:21 -06:00
Sean Rose 802da71a2c
Add ETLs and views for Google Search Console data (DENG-1733) (#4892)
* Add ETLs for historical Google Search Console data synced by Fivetran.

* Fix formatting of `CASE` subclauses like `WHEN` inside Jinja blocks.

* Add ETLs for current Google Search Console data exported directly to BigQuery.

* Add views for Google Search Console data.
2024-02-07 12:53:32 -08:00
Katie Windau 3084f7ed88
DENG-2492 - create new GA4 www_site_page_metrics_v2 (#4952)
* DENG-2492 - create new GA4 www_site_page_metrics_v2

* DENG-2492 fix query formatting
2024-02-02 12:37:54 -06:00
Katie Windau 3560ac46da
DENG-2492 - add new column engagement time and update hit time column… (#4947)
* DENG-2492 - add new column engagement time and update hit time column definition

* DENG-2492 - fix formatting for updated SQL

* DENG-2492 update hit_time calculation
2024-02-01 14:57:53 -06:00