mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
bigquery-etl/bqetl_project.yaml

421 строка
29 KiB
YAML
Исходник Обычный вид История

Move routine config to bqetl_project.yaml (#4038)
2023-07-11 20:52:48 +03:00
default:
project: moz-fx-data-shared-prod
sql_dir: sql/
test_project: bigquery-etl-integration-test
Move view, schema and remaining configs to bqetl_project.yaml (#4051) * Move view configs to bqetl_project.yaml * Move schema config to bqetl_project.yaml * Move docs config to bqetl_project.yaml * Replace remaining configs
2023-07-11 23:10:57 +03:00
non_user_facing_dataset_suffixes: # suffixes of datasets with non-user-facing views
- _derived
- _external
- _bi
- _restricted
- glam_etl
user_facing_project: mozdata
backfill_project: moz-fx-data-backfill
additional_projects: # additional valid projects managed outside of bigquery-etl
- moz-fx-data-pioneer-nonprod
- moz-fx-data-pioneer-prod
public_project: mozilla-public-data
Move routine config to bqetl_project.yaml (#4038)
2023-07-11 20:52:48 +03:00
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
dry_run:
function: https://us-central1-moz-fx-data-shared-prod.cloudfunctions.net/bigquery-etl-dryrun
skip:
# Access Denied
- sql/moz-fx-data-shared-prod/account_ecosystem_derived/ecosystem_client_id_lookup_v1/query.sql
- sql/moz-fx-data-shared-prod/account_ecosystem_derived/desktop_clients_daily_v1/query.sql
- sql/moz-fx-data-shared-prod/account_ecosystem_restricted/ecosystem_client_id_deletion_v1/query.sql
- sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/query.sql
- sql/moz-fx-data-shared-prod/activity_stream/impression_stats_flat/view.sql
- sql/moz-fx-data-shared-prod/activity_stream/tile_id_types/view.sql
Copy App Store tables from Fivetran (#4030) * Copy App Store tables from Fivetran * Move dryrun to config * Generate DAG * Dont dryrun views * Add schemas
2023-07-07 21:24:03 +03:00
- sql/moz-fx-data-shared-prod/app_store_external/**/*.sql
- sql/moz-fx-data-shared-prod/app_store/**/*.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/monitoring_derived/deletion_request_volume_v1/query.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/schema_error_counts_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/schema_error_counts_v2/query.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/suggest_impression_rate_v1/init.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/suggest_impression_rate_v1/query.sql
- sql/moz-fx-data-shared-prod/monitoring/schema_error_counts_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/structured_error_counts_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring/structured_error_counts/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/telemetry_missing_columns_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring/telemetry_missing_columns_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/telemetry_missing_columns_v2/view.sql
- sql/moz-fx-data-shared-prod/monitoring/telemetry_missing_columns_v2/view.sql
- sql/moz-fx-data-shared-prod/monitoring*/suggest*_rate*_live*/*.sql
- sql/moz-fx-data-shared-prod/monitoring*/topsites*_rate*_live*/*.sql
- sql/moz-fx-data-shared-prod/pocket/pocket_reach_mau/view.sql
- sql/moz-fx-data-shared-prod/telemetry/buildhub2/view.sql
FXA-6721 Setup import of accounts table from FxA stage CloudSQL (#4327) * FXA-6721 Setup import of accounts table from FxA stage CloudSQL * Fix typo
2023-09-22 16:27:07 +03:00
- sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/init.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_oauth_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/query.sql
feat(DENG-1576): introducing fxa_log_device_command_events_v2 to include GCP based logs (#4308) * introducing fxa_log_device_command_events_v2 to pull relevant logs from GCP log table * updated the bqetl_fxa_events DAG * correcting the source table * added fxa_log_device_command_events_v2 to dry run skip list due to the source table permissions issue and added date filters to incidcate tiemframes for which events are included in both tables * Update sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v2/query.sql Co-authored-by: akkomar <akkomar@users.noreply.github.com> * made changes as suggested by srose in PR#4308 --------- Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-09-21 13:06:14 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v2/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_last_seen_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/init.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/query.sql
DENG-722 Handle events from FxA services migrated to new GCP environment (#4288) Co-authored-by: kik-kik <kignasiak@mozilla.com> Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2023-09-13 21:49:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_gcp_stderr_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_gcp_stdout_events_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_auth_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_content_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_stdout_events_v1/query.sql
# feat(): updated fxa nonprod queries updated to be in line with production queries (#4297) * updated fxa nonprod/staging queries to be in line with what production queries look like * Apply suggestions from code review provided by srose Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com> * tweaks made as suggested by srose in PR#4297 --------- Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2023-09-21 17:54:17 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_gcp_stdout_events_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_gcp_stderr_events_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/init.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/query.sql
feat(DENG-1696): docker fxa admin server sanitized updating after gcp migration (#4400) * added v2 of docker_fxa_admin_server_events * updated the view to include the fields needed * added schema for firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1 * Update sql/moz-fx-data-shared-prod/firefox_accounts/docker_fxa_admin_server_sanitized/view.sql Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com> * added docker_fxa_admin_server_sanitized_v2 to dry_run skip list --------- Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2023-10-10 22:23:49 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v2/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/init.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/query.sql
feat(DENG-1578): docker_fxa_customs_sanitized_v2 query added and view updated (#4315) * added docker_fxa_customs_sanitized_v2 query to pull from the new fxa log table and updated the view to union v1 and v2 * updated schema file for docker_fxa_customs_sanitized_v2 * tweaks made as sugested by srose in PR#4315 * added docker_fxa_customs_sanitized_v2 to skip list due to permissions * once again scheduling fxa.docker_fxa_customs_sanitized_v1 as AWS events appear to still be arriving * added schema for docker_fxa_customs_sanitized_v1 * updated date filter to represent when we stopped receiving relevant events and descheduled v1
2023-09-29 11:51:36 +03:00
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v2/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/fivetran_costs_derived/destinations_v1/query.sql
- sql/moz-fx-data-shared-prod/fivetran_costs_derived/incremental_mar_v1/query.sql
- sql/moz-fx-data-shared-prod/fivetran_costs_derived/monthly_costs_v1/query.sql
- sql/**/apple_ads_external*/**/query.sql
- sql/moz-fx-data-shared-prod/regrets_reporter/regrets_reporter_update/view.sql
- sql/moz-fx-data-shared-prod/revenue_derived/client_ltv_v1/query.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_decoded_all/view.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_decoded_structured/view.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_decoded_stub_installer/view.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_decoded_telemetry/view.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_error_structured/view.sql
- sql/moz-fx-data-shared-prod/monitoring/payload_bytes_error_all/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/shredder_progress/view.sql
- sql/moz-fx-data-shared-prod/monitoring/shredder_progress/view.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/telemetry_distinct_docids_v1/query.sql
- sql/moz-fx-data-shared-prod/revenue_derived/client_ltv_normalized/query.sql
- sql/moz-fx-data-shared-prod/revenue_derived/client_ltv_normalized_v1/query.sql
Remove stray commas from skip list (#4026)
2023-07-06 22:12:52 +03:00
- sql/moz-fx-data-shared-prod/stripe_derived/**/*.sql
- sql/moz-fx-data-shared-prod/stripe_external/**/*.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-cjms-nonprod-9a36/cjms_bigquery/flows_live/view.sql
- sql/moz-fx-cjms-nonprod-9a36/cjms_bigquery/flows_v1/query.sql
- sql/moz-fx-cjms-nonprod-9a36/cjms_bigquery/refunds_v1/query.sql
- sql/moz-fx-cjms-nonprod-9a36/cjms_bigquery/subscriptions_v1/query.sql
- sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql
- sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/refunds_v1/query.sql
- sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/apple_subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/google_subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/nonprod_apple_subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/nonprod_google_subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/nonprod_stripe_subscriptions_history_v1/query.sql
Include customer data in `stripe_subscriptions_history_v2` (DENG-974) (#4132) * Add `stripe_customers_revised_changelog_v1` ETL. * Add `stripe_customers_history_v1` ETL. * Replace `stripe_external.pre_fivetran_customers` with `stripe_external.pre_fivetran_customer`. It's almost exactly the same, except it also has a `created` column. * Rename `stripe_external.pre_fivetran_customer` to `pre_fivetran_customers_v1`. * Include customers deleted before the initial Fivetran Stripe sync in `stripe_customers_revised_changelog_v1`. * Include customer data in `stripe_subscriptions_history_v2`.
2023-08-01 00:02:06 +03:00
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_customers_revised_changelog_v1/query.sql
Add Stripe-based logical subscriptions ETLs (DENG-977) (#4166) * Add `subscription_platform_derived.services_v1` ETL. * Add `subscription_platform_derived.subplat_flow_events_v1` ETL. * Add `subscription_platform_derived.subplat_attribution_impressions_v1` ETL. * Add `mozilla_vpn_derived.users_attribution_v1` table. * Add `subscription_platform_derived.stripe_logical_subscriptions_history_v1` ETL. * Add `subscription_platform_derived.logical_subscriptions_history_v1` ETL. * Add `subscription_platform_derived.daily_active_logical_subscriptions_v1` ETL. * Add `subscription_platform_derived.monthly_active_logical_subscriptions_v1` ETL. * Add `subscription_platform_derived.logical_subscription_events_v1` ETL. * Add `subscription_platform.logical_subscriptions` view. * Add `subscription_platform.daily_active_logical_subscriptions` view. * Add `subscription_platform.monthly_active_logical_subscriptions` view. * Add `subscription_platform.logical_subscription_events` view.
2023-08-19 00:07:05 +03:00
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_logical_subscriptions_history_v1/query.sql
Add `stripe_products_v1` and `stripe_plans_v1` ETLs (DENG-1448). (#4234)
2023-09-05 18:47:56 +03:00
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_plans_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_products_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_subscriptions_revised_changelog_v1/query.sql
- sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_subscriptions_history_v1/query.sql
- sql/moz-fx-data-shared-prod/stripe/itemized_payout_reconciliation/view.sql
Skip sql/moz-fx-data-shared-prod/stripe/itemized_tax_transactions/view.sql validation (#4070)
2023-07-12 23:02:07 +03:00
- sql/moz-fx-data-shared-prod/stripe/itemized_tax_transactions/view.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/add_device_events_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/devices_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/fxa_attribution_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/funnel_product_page_to_subscribed_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/guardian_apple_events_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/login_flows_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/protected_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/site_metrics_empty_check_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/site_metrics_summary_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/subscriptions_v1/query.sql
Add Stripe-based logical subscriptions ETLs (DENG-977) (#4166) * Add `subscription_platform_derived.services_v1` ETL. * Add `subscription_platform_derived.subplat_flow_events_v1` ETL. * Add `subscription_platform_derived.subplat_attribution_impressions_v1` ETL. * Add `mozilla_vpn_derived.users_attribution_v1` table. * Add `subscription_platform_derived.stripe_logical_subscriptions_history_v1` ETL. * Add `subscription_platform_derived.logical_subscriptions_history_v1` ETL. * Add `subscription_platform_derived.daily_active_logical_subscriptions_v1` ETL. * Add `subscription_platform_derived.monthly_active_logical_subscriptions_v1` ETL. * Add `subscription_platform_derived.logical_subscription_events_v1` ETL. * Add `subscription_platform.logical_subscriptions` view. * Add `subscription_platform.daily_active_logical_subscriptions` view. * Add `subscription_platform.monthly_active_logical_subscriptions` view. * Add `subscription_platform.logical_subscription_events` view.
2023-08-19 00:07:05 +03:00
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/users_attribution_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/users_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/devices_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/subscriptions_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/users_v1/query.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/telemetry_missing_columns_v3/query.sql
Bug 1842216 exclude main_remainder_v4 from telemetry_missing_columns_v3 (#4031) * Bug 1842216 exclude main_remainder_v4 from telemetry_missing_columns_v3 * exclude view from dryrun * add remainder for saved_session and first_shutdown * Update sql/moz-fx-data-shared-prod/monitoring_derived/telemetry_missing_columns_v3/query.sql Co-authored-by: Daniel Thorn <dthorn@mozilla.com> --------- Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2023-07-11 20:08:28 +03:00
- sql/moz-fx-data-shared-prod/monitoring/telemetry_missing_columns/view.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-experiments/monitoring/query_cost_v1/query.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/subscriptions_v1/init.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/users_v1/init.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/protected_v1/init.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_derived/add_device_events_v1/init.sql
- sql/moz-fx-data-shared-prod/mozilla_vpn_external/devices_v1/init.sql
- sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/query.sql
Add has_adblocker_addon to search_clients_daily (#3558) * feat: adblocker addons field * Update sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v8/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * fix: use private table * fix: where clause for private table * Reference static addons table * Switch to new monetization_blocking_addons table * Drop project name in reference to monetization_blocking_addons * Don't dry-run search_clients_daily * Add has_adblocker_addons to search_clients_daily_v8 tests --------- Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2023-07-19 19:04:38 +03:00
- sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v8/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/search_terms*/**/*.sql
- sql/moz-fx-data-bq-performance/release_criteria/dashboard_health_v1/query.sql
- sql/moz-fx-data-bq-performance/release_criteria/rc_flattened_test_data_v1/query.sql
- sql/moz-fx-data-bq-performance/release_criteria/release_criteria_summary_v1/query.sql
- sql/moz-fx-data-bq-performance/release_criteria/stale_tests_v1/query.sql
- sql/moz-fx-data-bq-performance/release_criteria/release_criteria_v1/query.sql
- sql/moz-fx-data-shared-prod/contextual_services/**/*.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/**/*.sql
- sql/moz-fx-data-shared-prod/**/topsites_impression/view.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_v1/init.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/adm_forecasting_v1/query.sql
- sql/moz-fx-data-shared-prod/regrets_reporter/regrets_reporter_summary/view.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/request_payload_suggest_v2/query.sql
- sql/moz-fx-data-shared-prod/contextual_services_derived/request_payload_tiles_v2/query.sql
[RS-688] Suggest forecasting v1 (#3719) * Add DE for mobile * Format query * Update sql/moz-fx-data-shared-prod/contextual_services_derived/adm_forecasting_v1/query.sql Co-authored-by: skahmann3 <16420065+skahmann3@users.noreply.github.com> * Fix two typos * Suggest has no new markets * Fix merge error and reformat * Update & separate suggest forecasting from Tiles * Less filters Don't need all the filters in the back half of this query since they're already employed in the source table * Update to focus on funnel metrics - I manually renamed the files from "suggest_forecasting" to "suggest_revenue_levers_daily" so hopefully that won't break anything. - Since this is being used for funnel analysis, there are multiple ratios built using two other variables in the table. We could drop the ratios if storage costs are a concern, I've kept them since I prefer the convenience of having the ratios ready to go. * Removed ratios Think better to define ratios in Looker (since then can more easily change the definitions) than to set in BQ which is harder to change/ may require backfill. Note for DE: schema command doesn't work for me, so may need to be run manually * add schema.yaml and scheduling to metadata.yaml * correct the view name * add suggest revenue to skip dry run * correct dag generation * Add changes to bqetl_search.py dag --------- Co-authored-by: Curtis Morales <cmorales@mozilla.com> Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com> Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2023-07-13 00:24:47 +03:00
- sql/moz-fx-data-shared-prod/contextual_services_derived/suggest_revenue_levers_daily_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/regrets_reporter_derived/regrets_reporter_summary_v1/*.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/**/*.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/**/*.sql
- sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report/view.sql
- sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update/view.sql
- sql/moz-fx-data-marketing-prod/acoustic/**/*.sql
- sql/moz-fx-data-marketing-prod/adjust_derived/**/*.sql
- sql/moz-fx-data-shared-prod/monitoring_derived/airflow_*/*.sql
- sql/moz-fx-data-shared-prod/google_ads_derived/**/*.sql
- sql/moz-fx-data-shared-prod/**/client_deduplication*/*.sql
- sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql
- sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v2/query.sql
- sql/moz-fx-data-shared-prod/fenix_external/installs_by_country_v1/query.sql
- sql/moz-fx-data-shared-prod/fenix/installs_by_country/view.sql
- sql/moz-fx-data-shared-prod/firefox_desktop/top_sites/view.sql
- sql/moz-fx-data-shared-prod/firefox_desktop/quick_suggest/view.sql
# Materialized views
- sql/moz-fx-data-shared-prod/telemetry_derived/experiment_search_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/experiment_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/experiment_search_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/experiment_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/experiment_search_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/experiment_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta_derived/experiment_search_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta_derived/experiment_events_live_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/experiment_enrollment_cumulative_population_estimate_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/experiment_enrollment_cumulative_population_estimate/view.sql
# Already exists (and lacks an "OR REPLACE" clause)
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/clients_first_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/clients_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_fenix_derived/clients_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/org_mozilla_vrbrowser_derived/clients_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/core_clients_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry/fxa_users_last_seen_raw_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/core_clients_first_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/fxa_users_services_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/messaging_system_derived/cfr_users_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/messaging_system_derived/onboarding_users_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/messaging_system_derived/snippets_users_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/messaging_system_derived/whats_new_panel_users_last_seen_v1/init.sql
# Reference table not found
add adjust view to bqetl_project to skip (#4046)
2023-07-11 16:01:51 +03:00
- sql/moz-fx-data-shared-prod/adjust/adjust_kpi_deliverables/view.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/monitoring_derived/structured_detailed_error_counts_v1/view.sql
- sql/moz-fx-data-shared-prod/monitoring/structured_detailed_error_counts/view.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/migrated_clients_v1/query.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox_derived/incline_executive_v1/query.sql
- sql/moz-fx-data-shared-prod/org_mozilla_firefox/migrated_clients/view.sql
- sql/moz-fx-data-shared-prod/telemetry/sponsored_tiles_ad_request_fill/view.sql
# No matching signature for function IF
- sql/moz-fx-data-shared-prod/static/fxa_amplitude_export_users_last_seen/query.sql
# Duplicate UDF
- sql/moz-fx-data-shared-prod/static/fxa_amplitude_export_users_daily/query.sql
# Syntax error
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v1/init.sql
# HTTP Error 408: Request Time-out
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v1/query.sql
DS-2947 clients_first_seen_v2 (#3962) * DENG-850 Add test setup. * DS-2947. Create new dataset and tests for Firefox Desktop Clients. * DS-2947. Update dataset name to clients_first_seen_v2. * DS-2947. Dataset name to clients_first_seen_v2. * DS-2947. Updating tests. * DS-2947. Schema for clients_first_seen_v2. * DS-2947. Tests update. * Tests update * Restore test files * DS-2947. Get data from main and new profile ping. Get first dltoken and dlsource available. Update tests. * DS-2947. Use main ping's submission_timestamp_min to find the earliest ping. * Remove app_display_versin as it is normalised in app_version. Update fields on a 7 day window. Retrieve data from the ping with the earliest NOT NULL value to remove NULLS when main ping is not available. * Remove app_display_versin as it is normalised in app_version. Update fields on a 7 day window. Retrieve data from the ping with the earliest NOT NULL value to remove NULLS when main ping is not available. * Update schemas, remove duplicated columns from query and init. Adapt existing unitest and add unitest for 7-day window updates. Include scheduler in DAG bqetl_analytics_tables. * Update to enable initialize from query.sql. Remove init.sql. * Update DAGs dependencies. * DAG bqetl_main_summary updated. * Query and tests update to join with sample_id. * Refactor metadata fields in query and tests. * Schema and descriptions updated. Remove filter to query the existing table. Remove the DATETIME, the first_seen_date is equivalent. * Column required to be explicit in the query to match the schema. * Test fix. * Tests tmp changes. * Remove 7-day window update and update tests. * Add second_seen_date to the query * DS-3037 Add second_seen_date and tests. * DS-3037 Add is_init to calculate second_seen_date. * remove files in analysis dataset * DS-3037 Add is_init to calculate second_seen_date. Formatting. * DS-2986 Add initialize script. Change submission_timestamp_min to submission_date due to NULL values in that field. * DENG-1314. Update metadata reported pings and tests in the query. * DS-3054. Update bqetl initialize command and query to support parallel run. * DS-3054. Update query to use submission_timestamp_min from main ping where available for precision in source ping for first_seen_date, add source ping of second_seen_date, get only first_seen date from new_profile and shutdown ping due to 16% clients with more than one new_profile ping. Add capability to run in parallel in bqetl. Update tests. * DS-3054. Remove initialize.py. * Reset unrelated formatting changes from this branch to match the main branch. * Correct jira template. * DS-2947. Update naming for attribution dltoken and dlsource. * DS-2947. Update column names and tests and clarity for the initialization command. * DS-2986. Create table with schema and metadata in command initialize. * Document what is the result expected from each subquery. * Documentation update. * DS-3145. Include user agent and the source ping. Add query documentation. * DS-3145. Update tests. * DS-3146 Update logic to get attributes only from the ping that reports the first_seen_date, include locale, update the source for app_build_id and collect second_seen_date only from main ping. * DS-3146 Update logic to get attributes only from the ping that reports the first_seen_date, include locale, update the source for app_build_id and collect second_seen_date only from main ping. * DS-3054. Updates and save initialization. * DS-3054. Table name required for DAG generation. * DS-2947_implement_bigquery_changes_in_another_PR. * DS-2947 Naming * Add clients_first_seen_v2 to skip dry-run. --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-09-25 17:39:04 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v2/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/latest_versions/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/italy_covid19_outage_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_nightly_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_nightly_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_1pct_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_1pct_v1/query.sql
# Query parameter not found
- sql/moz-fx-data-shared-prod/telemetry_derived/experiments_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_scalar_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_boolean_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_content_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_gpu_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_parent_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_histogram_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_aggregates_v2/query.sql
Glam non norm agg (#3873) * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * fix schema files * fix clients_histogram_probe_counts schema * remove another init.sql * fix dryrun ignore order * fix table name * change dryrun ignore order to try avoiding fenix for being on path * another change in dryrun * Move glam queries from dryrun to bqetl_project.yaml to ignore * add tbl deps on tests
2023-07-20 00:01:58 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_probe_counts_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_bucket_counts_v1/query.sql
Glam non norm agg (#3873) * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * fix schema files * fix clients_histogram_probe_counts schema * remove another init.sql * fix dryrun ignore order * fix table name * change dryrun ignore order to try avoiding fenix for being on path * another change in dryrun * Move glam queries from dryrun to bqetl_project.yaml to ignore * add tbl deps on tests
2023-07-20 00:01:58 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_non_norm_histogram_bucket_counts_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/glam_client_probe_counts_extract_v1/query.sql
Bug 1844886: Add non-norm cols to glam scalar tbls (#4111) * Add non-norm cols to scalar tbls * Add missing schema to scalar_percentiles_v1 * Add missing schema to scalar_percentiles_v1
2023-07-25 18:13:20 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/client_probe_counts/view.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/glam_sample_counts_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/glam_user_counts_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/scalar_percentiles_v1/query.sql
Glam non norm agg (#3873) * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * fix schema files * fix clients_histogram_probe_counts schema * remove another init.sql * fix dryrun ignore order * fix table name * change dryrun ignore order to try avoiding fenix for being on path * another change in dryrun * Move glam queries from dryrun to bqetl_project.yaml to ignore * add tbl deps on tests
2023-07-20 00:01:58 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/histogram_percentiles_v1/query.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_scalar_probe_counts_v1/query.sql
Bug 1844886: Add non-norm cols to glam scalar tbls (#4111) * Add non-norm cols to scalar tbls * Add missing schema to scalar_percentiles_v1 * Add missing schema to scalar_percentiles_v1
2023-07-25 18:13:20 +03:00
- sql/moz-fx-data-shared-prod/telemetry/client_probe_counts/view.sql
Add ConfigLoader and move dry run skip to bqetl_project.yaml (#4000) * Add ConfigLoader and move dry run skip to bqetl_project.yaml * format tests
2023-07-06 20:42:29 +03:00
- sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_sql_run_check_v1/query.sql
# Dataset sql/glam-fenix-dev:glam_etl was not found
- sql/glam-fenix-dev/glam_etl/**/*.sql
# Query templates
- sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/fenix_metrics.template.sql
- sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_daily_v1/mobile_search_clients_daily.template.sql
# Temporary table does not exist
# TODO: remove this in a follow up PR
- sql/moz-fx-data-shared-prod/*/baseline_clients_first_seen_v1/*.sql
# Query too complex
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/event_types_history_v1/query.sql
- sql/moz-fx-data-shared-prod/firefox_accounts_derived/event_types_history_v1/init.sql
# Tests
- sql/moz-fx-data-test-project/test/simple_view/view.sql
Move format skip files to bqetl_project.yaml (#4033)
2023-07-10 20:10:47 +03:00
format:
skip:
- bigquery_etl/glam/templates/*.sql
- sql_generators/events_daily/templates/**/*.sql
- sql_generators/glean_usage/templates/*.sql
- sql_generators/search/templates/*.sql
- sql_generators/experiment_monitoring/templates/**/*.sql
- sql_generators/feature_usage/templates/*.sql
- sql/moz-fx-data-shared-prod/telemetry/fenix_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/fennec_ios_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/fire_tv_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/first_shutdown_summary/view.sql
- sql/moz-fx-data-shared-prod/telemetry/first_shutdown_summary_v4/view.sql
- sql/moz-fx-data-shared-prod/telemetry/focus_android_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/lockwise_mobile_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/rocket_android_events_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/simpleprophet_forecasts/view.sql
- sql/moz-fx-data-shared-prod/telemetry/smoot_usage_day_0/view.sql
- sql/moz-fx-data-shared-prod/telemetry/smoot_usage_day_13/view.sql
- sql/moz-fx-data-shared-prod/telemetry/sync/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_anonymous_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_anonymous_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_core_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_core_parquet_v3/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_downgrade_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_downgrade_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_focus_event_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_focus_event_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_heartbeat_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_heartbeat_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_ip_privacy_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_ip_privacy_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_mobile_event_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_mobile_event_parquet_v2/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_new_profile_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_new_profile_parquet_v2/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_shield_study_parquet/view.sql
- sql/moz-fx-data-shared-prod/telemetry/telemetry_shield_study_parquet_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry/windows_10_aggregate/view.sql
- sql/moz-fx-data-shared-prod/telemetry/windows_10_build_distribution/view.sql
- sql/moz-fx-data-shared-prod/telemetry/windows_10_patch_adoption/view.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_boolean_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_histogram_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_scalar_aggregates_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_scalar_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_aggregates_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_probe_counts_v1/query.sql
Glam non norm agg (#3873) * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch) * glam: Non normalized aggregations to legacy histograms * glam: add non-normalized aggs to probe counts extract * glam: add init.sql to relevant tbls for non-norm aggs * glam: ignore dryrun histogram_percentiles * glam: add description and eol to init * fix schema files * fix clients_histogram_probe_counts schema * remove another init.sql * fix dryrun ignore order * fix table name * change dryrun ignore order to try avoiding fenix for being on path * another change in dryrun * Move glam queries from dryrun to bqetl_project.yaml to ignore * add tbl deps on tests
2023-07-20 00:01:58 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_histogram_bucket_counts_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_non_norm_histogram_bucket_counts_v1/query.sql
Move format skip files to bqetl_project.yaml (#4033)
2023-07-10 20:10:47 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/clients_scalar_probe_counts_v1/query.sql
Bug 1844886: Add non-norm cols to glam scalar tbls (#4111) * Add non-norm cols to scalar tbls * Add missing schema to scalar_percentiles_v1 * Add missing schema to scalar_percentiles_v1
2023-07-25 18:13:20 +03:00
- sql/moz-fx-data-shared-prod/telemetry/client_probe_counts/view.sql
Move format skip files to bqetl_project.yaml (#4033)
2023-07-10 20:10:47 +03:00
- sql/moz-fx-data-shared-prod/telemetry_derived/core_clients_daily_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/core_clients_last_seen_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/core_clients_last_seen_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/core_live/view.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/devtools_events_amplitude_v1/view.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/error_aggregates/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/event_events_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/event_events_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/events_live/view.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/experiment_enrollment_aggregates_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/experiment_enrollment_aggregates_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/experiments_daily_active_clients_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/firefox_desktop_exact_mau28_by_client_count_dimensions_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/fxa_users_services_daily_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/fxa_users_services_last_seen_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/glam_user_counts_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/latest_versions/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_events_v1/init.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/main_events_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/scalar_percentiles_v1/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/smoot_usage_desktop_v2/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/smoot_usage_fxa_v2/query.sql
- sql/moz-fx-data-shared-prod/telemetry_derived/smoot_usage_new_profiles_v2/query.sql
- sql/moz-fx-data-shared-prod/udf_js/jackknife_percentile_ci/udf.sql
- sql/moz-fx-data-shared-prod/udf_js/bootstrap_percentile_ci/udf.sql
- sql/moz-fx-data-shared-prod/udf_legacy/contains.sql
- sql/moz-fx-data-shared-prod/udf_legacy/date_format.sql
- sql/moz-fx-data-shared-prod/udf_legacy/date_trunc.sql
- sql/moz-fx-data-shared-prod/udf_legacy/to_iso8601.sql
- stored_procedures/safe_crc32_uuid.sql
Move routine config to bqetl_project.yaml (#4038)
2023-07-11 20:52:48 +03:00
routine:
dependency_dir: udf_js_lib/
publish:
gcs_bucket: moz-fx-data-prod-bigquery-etl
gcs_path: ""
skip:
- sql/moz-fx-data-shared-prod/udf/main_summary_scalars/udf.sql
assert_udf_dir: tests/assert
example_dir: examples
project: mozfun
Move view, schema and remaining configs to bqetl_project.yaml (#4051) * Move view configs to bqetl_project.yaml * Move schema config to bqetl_project.yaml * Move docs config to bqetl_project.yaml * Replace remaining configs
2023-07-11 23:10:57 +03:00
view:
validation:
skip:
# tests
- sql/moz-fx-data-test-project/test/simple_view/view.sql
# Access Denied
- sql/moz-fx-data-shared-prod/telemetry/experiment_enrollment_cumulative_population_estimate/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/events/view.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/deletion_request/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/regret_details/view.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/video_data/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/deletion_request/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql
- sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update/view.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/video_index/view.sql
- sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql
- sql/moz-fx-data-shared-prod/regrets_reporter_ucs/main_events/view.sql
- sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql
- sql/moz-fx-data-shared-prod/**/client_deduplication/view.sql
Skip sql/moz-fx-data-shared-prod/stripe/itemized_tax_transactions/view.sql validation (#4070)
2023-07-12 23:02:07 +03:00
- sql/moz-fx-data-shared-prod/stripe/itemized_tax_transactions/view.sql
Move view, schema and remaining configs to bqetl_project.yaml (#4051) * Move view configs to bqetl_project.yaml * Move schema config to bqetl_project.yaml * Move docs config to bqetl_project.yaml * Replace remaining configs
2023-07-11 23:10:57 +03:00
publish:
skip:
Fix paths for skipped views (#4063)
2023-07-12 19:53:59 +03:00
- sql/moz-fx-data-shared-prod/activity_stream/tile_id_types/view.sql
- sql/moz-fx-data-shared-prod/pocket/pocket_reach_mau/view.sql
- sql/moz-fx-data-shared-prod/telemetry/buildhub2/view.sql
Move view, schema and remaining configs to bqetl_project.yaml (#4051) * Move view configs to bqetl_project.yaml * Move schema config to bqetl_project.yaml * Move docs config to bqetl_project.yaml * Replace remaining configs
2023-07-11 23:10:57 +03:00
# Dataset glam-fenix-dev:glam_etl was not found
# TODO: this should be removed if views are to be automatically deployed
- sql/glam-fenix-dev/glam_etl/**/view.sql
# tests
- sql/moz-fx-data-test-project/test/simple_view/view.sql
schema:
mozilla_pipeline_schemas_uri: https://github.com/mozilla-services/mozilla-pipeline-schemas
skip_prefixes:
- pioneer
- rally
docs:
docs_dir: docs
default_projects:
- mozfun
- moz-fx-data-shared-prod
source_url: https://github.com/mozilla/bigquery-etl/blob/generated-sql
edit_url: https://github.com/mozilla/bigquery-etl/edit/generated-sql
public_data:
bucket: mozilla-public-data-http
api_version: v1
endpoint: https://public-data.telemetry.mozilla.org/
review_link: https://bugzilla.mozilla.org/show_bug.cgi?id=
render:
skip:
# uses {%s} which results in unknown tag exception
- sql/mozfun/hist/string_to_json/udf.sql
[Bug 1821784] Be explicit about skipping generation for certain Glean artifacts (#4069) * Fix skipping certain Glean artifacts * Explicitly skip specific glean_usage generated artifacts * Review feedback
2023-07-14 21:54:00 +03:00
generate:
glean_usage:
skip_existing: # Skip automatically updating the following artifacts
- sql/moz-fx-data-shared-prod/mozilla_vpn/deletion_request/**
- sql/moz-fx-data-shared-prod/mozilla_vpn/events/**
- sql/moz-fx-data-shared-prod/mozilla_vpn/main/**
Add client_deduplication to view generation skip list (#4098)
2023-07-19 19:42:07 +03:00
- sql/moz-fx-data-shared-prod/fenix/client_deduplication/**