* feat: move mobile_engagement SQL logic into mobile_kpi_support_metrics generator
* feat: remove mobile_engagement queries from telemetry and telemetry_erived
* feat: apply formatting to engagement_clients view and add checks template
* feat: add engagement view template and add engagement templates to the generation logic
* bug: fix engagement query errors
* feat: sneak in removing of retention views inside telemetry (mobile kpi generator will overwrite these anyways)
* feat: remove bqetl_mobile_engagement_model DAG from DAG list as no longer needed.
* feat: fix an engagement_v1 query template and removed unecessary comment from the metadata template
* fix: update engagement templates to ensure valid query is rendered
* initial commit for desktop_retention_clients view
* initial commit for desktop_retention_clients view - after formatting
* initial add of query script for telemetry_derived.retention_v1 table
* change view to table as I need to use the submission_date query parameter
* move view to query for desktop_retention_clients
* reformat file and add COALESCE on submission_date for new_profiles
* take out app_name from retention_v1 query. Change isp_name to isp, take out mozfun UDF on normalized_os_version in CTE and update main query from retention_clients
* run formatting on retention_v1/query.sql
* move files from retention to desktop_retention, retention_clietns to desktop_retention_clients
* add newline to end of schema.yaml file
* reinstate retention_v1 deprecated folder/metadata.yaml file
* refactor desktop_retention_clients_v1/desktop_retention metadata.yaml - add in clustering, take out extra space indent
* refactor metadata.yaml, add schema.yaml, change query.sql to pull from telemetry_derived.desktop_retention_clients_v1 not view
* add in metric_date for desktop_retention_v1 partition, take out require partition date filtering for both retention and retention_clients
* add desktop_retention_model to dags.yaml
* remove retention_clients_v1
* change column name is_new_profile to new_profile_metric_date
* take out 'app_name' from group by
* add telemetry_derived.desktop_retention_clients_v1 to shredder config
* DENG-3186 new desktop engagement model
* DENG-3186 add distribution ID to eng client V1 query
* DENG-3186 - add distribution ID to the schema.yaml for eng client
* DENG-3186 add distribution ID to the agg table
* DENG-3186 adding locale to both client & agg tables
* DENG-3186 add app_version to client & agg tables
* DENG-3168 move lifecycle stage from engagement agg table to engagement agg view
* DENG-3186 - add new desktop_engagement_client_v1 table to shredder
* DENG-3186 update formatting of config.py
* DENG-3186 switch to get country, os, os version, and build # from clients last seen and remove join to clients daily
* DENG-3471 suppression list from campaign monitor
* Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* after seans coder review
---------
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* DENG-3471 suppression list from campaign monitor
* Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
---------
Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
* DENG-3008 removing misspelled file
* DENG-3008 adding suppressions model
* adding products model
* updating dataset
* adding newsletter model
* adding waitlist model
* removing create statement and adding modified timestamp
* updating name and description for newsletters
* updating description
* adding users model
* adding braze dag
* changing subscribed to boolean
* updating type to date
* updating to full refresh
* updating email id to external id for standardization
* updating format
* updating nullable fields
* updating format
* updating nullable fields
* adding subscription map model
* adding subscriptions model
* adding user profiles model
* updating format
* formatting
* removing last modified date in favor of only timestamp
* updating table name
* feat: add fenix_derived test tables to the data-observability-dev project
* feat: add event_monitoring_live_v1 to data-observability-dev
* feat: add new temp DAG for running copy to data-observability-dev tasks
* fix: using fully qualified source table name in fenix_derived.events_daily_v1 inside data-observabiolity project
* fix: correct event queries used to copy data into data-observability-dev project
* Add ETLs for historical Google Search Console data synced by Fivetran.
* Fix formatting of `CASE` subclauses like `WHEN` inside Jinja blocks.
* Add ETLs for current Google Search Console data exported directly to BigQuery.
* Add views for Google Search Console data.
* Update bqetl_google_analytics_derived_ga4 yaml configs
* Switch to using countifs for more readability
* reformat and switch to countifs for readability
* Initial draft
* adding new GA4 dag
* fixing source for browser
* work in progress
* change group by to be explicit
* adding non_fx_sessions
* adding non-fx-sessions
* adding campaign to query
* adding new column to group by
* added ad_content column
* Adding downloads column
* Adding non fx downloads
* reformat the SQL
* Add owner to new DAG
* Updata data types in the schema file
* Add missing comma
* Update query.sql
* Update start date and reduce from tier1 -> tier2
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* update source and medium columns to use last touch attribution instead of first touch attribution
* update group by
---------
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* DENG-2262 - add new DAG bqetl_desktop_installs_v1
* Initial commit for DENG-2262
* DENG-2262 - reformatted query.sql
* DENG-2262 add empty new line to end of view.sql
* update
* This is the modified codes following suggestions in https://github.com/mozilla/bigquery-etl/pull/4467. This pull request aims to update codes that create four mobile feature usage tables
* updated codes following comments from kik in 72d6d71910
* update feature usage table codes following suggestions
* added new data to mobile feature usage table (logins data for iOS) and modified to add unnest for nested values
* update codes based on suggestion from https://github.com/mozilla/bigquery-etl/pull/4648/files
* update metadata.yaml and dags.yaml according to suggestions
* update metadata.yaml and dags.yaml according to suggestions
* remove clustering and references in all the metadata files, add LEFT JOIN to all the UNNEST, modify SQL files to folllow the incremental submission date format, and rename dau as events(metrics)_ping_client_count
* updated schema to reflect the name change for dau
* update according to comments on dec 15
* update the distinct client count name
* fix the yaml errors
* fix no new line error fordags.yaml
---------
Co-authored-by: Ruoxi Zhao <rzhao@rzhao-37509.local>
Co-authored-by: Ruoxi Zhao <rzhao@rzhao-37509.lan>
* Add support for assigning Airflow tasks to task groups
* Generate separate Airflow tasks for glean_usage
* Remove Airflow dependencies from old glean_usage tasks
* Update scheduler of aggregates to run after upstreams.
* Update dags for new scheduler of analytics_aggregates
* Update dag bqetl_search
* Remove DAG.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* feat: urlbar events final release
* feat: new result types
* feat: add interaction and group
* fix: date
* fix: use BQ builtin for UUIDs
* Add the view_v2'
* Add new table to the DAG
* fix CI error
fix ci error
* remove teon brooks
* Incorporate feedback by Curtis
Incorporate feedback from Curtis
---------
Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
* New job to calculate newtab visits -> activity stream sessions
* Removing newline chars at end of file
* Removing newline chars at end of file
* Removing newline chars at end of file
* Addressing comment suggestions
* Format
* Add bqetl_ads DAG
* Add ACL to nt_visits_to_sessions_conversion_factors_daily_v1
* Add metadata files
* Add view to dry_run skip list
* Oops, fix the view
---------
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* Add review checker derived datasets
* Add bqetl_review_checker dag
Fix
* Fix CI validate dag step
* Incorporate feedback from Alex
* Fix CI
* change client last seen to clients first seen
change client last seen to clients first seen
* fix dag
kik-kik
Chelsey Beck
Eduardo Filho
Sean Rose
Katie Windau
Marlene Hirose
Leli
Anna Scholtz
Winnie Chan
ksiegler1
Alexander
rzhao
Daniel Thorn
Lucia
Frank Bertsch
Rebecca BurWei
akkomar
Sergio E. Betancourt
Alekhya