mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
bigquery-etl/dags.yaml

1677 строки
42 KiB
YAML
Исходник Обычный вид История

Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
---
Add version to error_aggregates query
2020-06-03 20:49:11 +03:00
bqetl_error_aggregates:
schedule_interval: 3h
default_args:
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
owner: wkahngreene@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
email:
[
"telemetry-alerts@mozilla.com",
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
"wkahngreene@mozilla.com",
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
]
start_date: "2019-11-01"
Add version to error_aggregates query
2020-06-03 20:49:11 +03:00
retries: 1
retry_delay: 20m
depends_on_past: false
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
KPI dashboard generated Airflow DAG
2020-05-22 03:01:42 +03:00
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
bqetl_ssl_ratios:
More task re-scheduling
2020-07-10 20:33:34 +03:00
schedule_interval: 0 2 * * *
Add Markdown descriptions to DAGs
2021-01-30 02:04:42 +03:00
description: The DAG schedules SSL ratios queries.
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
default_args:
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
owner: chutten@mozilla.com
start_date: "2019-07-20"
email: ["telemetry-alerts@mozilla.com", "chutten@mozilla.com"]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
bqetl_amo_stats:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
# yamllint disable rule:line-length
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Add-on download and install statistics to power the
[addons.mozilla.org](https://addons.mozilla.org) (AMO) stats pages.
See the [post on the Add-Ons Blog](https://blog.mozilla.org/addons/2020/06/10/improvements-to-statistics-processing-on-amo/).
# yamllint enable rule:line-length
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
default_args:
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
owner: kik@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-06-01"
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
email: ["telemetry-alerts@mozilla.com", "kik@mozilla.com"]
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
bqetl_core DAG
2020-06-05 21:36:53 +03:00
bqetl_core:
More task re-scheduling
2020-07-10 20:33:34 +03:00
schedule_interval: 0 2 * * *
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
description:
Tables derived from the legacy telemetry `core` ping sent by various
mobile applications.
bqetl_core DAG
2020-06-05 21:36:53 +03:00
default_args:
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
bqetl_core DAG
2020-06-05 21:36:53 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_core DAG
2020-06-05 21:36:53 +03:00
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
bqetl_nondesktop:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 3 * * *
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
default_args:
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
owner: "ascholtz@mozilla.com"
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
email: [
"telemetry-alerts@mozilla.com",
]
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
bqetl_mobile_search:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
email:
- "telemetry-alerts@mozilla.com"
- "anicholson@mozilla.com"
- "akomar@mozilla.com"
search_revenue_levers_daily_v1 table (#3534) * add new data table search_revenue_levers_daily * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Restore deleted tasks * Change DOU to DAU * Add Bing cohort and tidy query * Format * Add schema file for search_revenue_levers_daily * Add back check for distribution_id and fix tagged_follow_on * Update dags * One more generated task * Be explicit in group by and order * Fix comment * Delete commented out line * Format * Add acer_cohort table to moz-fx-data-shared-prod * Rename query to init.sql * Update acer cohort query * Delete no longer needed comment * Drop references to mozdata * Update sql/moz-fx-data-shared-prod/search_derived/acer_cohort_v1/init.sql Co-authored-by: akkomar <akkomar@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/search_derived/acer_cohort_v1/metadata.yaml Co-authored-by: akkomar <akkomar@users.noreply.github.com> * Update schema * Remove clustering from table --------- Co-authored-by: Xuan Luo <xluo@mozilla.com> Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-02-01 22:29:49 +03:00
- "cmorales@mozilla.com"
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
bqetl_fxa_events:
schedule_interval: 30 1 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Copies data from a Firefox Accounts (FxA) project. Those source tables
are populated via Cloud Logging (Stackdriver). We hash various fields
as part of the import.
The DAG also provides daily aggregations on top of the raw log data,
which eventually power high-level reporting about FxA usage.
Bug 1707571 - Ignore isverified in fxa_auth_events (#1975) * Bug 1707571 Ignore isverified in fxa_auth_events This is hopefully a temporary workaround while we understand the incompatible schema change in the source table. Very similar to the fix in https://github.com/mozilla/bigquery-etl/pull/1683 * Add docs about this failure mode.
2021-04-26 17:50:00 +03:00
Tasks here have occasionally failed due to incompatible schema changes
in the tables populated by Cloud Logging.
See https://github.com/mozilla/bigquery-etl/issues/1684 for an example
mitigation.
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
default_args:
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
owner: kik@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-03-01"
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
email: ["telemetry-alerts@mozilla.com", "kik@mozilla.com"]
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
retries: 1
retry_delay: 10m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
FXA-6721 Setup import of accounts table from FxA stage CloudSQL (#4327) * FXA-6721 Setup import of accounts table from FxA stage CloudSQL * Fix typo
2023-09-22 16:27:07 +03:00
bqetl_accounts_backend_external:
schedule_interval: 30 1 * * *
description: |
Copies data from Firefox Accounts (FxA) CloudSQL databases.
This DAG is under active development.
default_args:
owner: akomar@mozilla.com
start_date: "2023-09-19"
FXA-6721 Setup import of accounts table from FxA production CloudSQL (#4423)
2023-10-25 10:50:25 +03:00
email: ["akomar@mozilla.com", "telemetry-alerts@mozilla.com"]
FXA-6721 Setup import of accounts table from FxA stage CloudSQL (#4327) * FXA-6721 Setup import of accounts table from FxA stage CloudSQL * Fix typo
2023-09-22 16:27:07 +03:00
retries: 1
retry_delay: 10m
tags:
- impact/tier_3
- repo/bigquery-etl
DENG-2095 Create users_services_daily table for accounts_backend (#4906)
2024-02-19 11:07:32 +03:00
bqetl_accounts_derived:
schedule_interval: 30 2 * * *
description: |
Derived tables for analyzing data from Mozilla Accounts (`accounts_backend` and
`accounts_frontend` Glean applications).
This DAG is under active development.
default_args:
owner: akomar@mozilla.com
start_date: "2024-01-01"
email: ["akomar@mozilla.com", "ksiegler@mozilla.com"]
retries: 1
retry_delay: 10m
tags:
- impact/tier_3
- repo/bigquery-etl
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
bqetl_subplat:
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
schedule_interval: 45 1 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
Daily imports for Subscription Platform data from Stripe and the Mozilla VPN
operational DB as well as derived tables based on that data.
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
Depends on `bqetl_fxa_events`, so is scheduled to run a bit after that.
Bug 1768419 - Add note to subplat dag docs for airflow triage (#2959) * Bug 1768419 - Add note to subplat dag docs for airflow triage * enriched the bqetl_subplat DAG description with a bit more detail regarding expected failures Co-authored-by: kik-kik <kignasiak01@gmail.com>
2022-06-01 16:54:19 +03:00
Stripe data retrieved by stripe_external__itemized_payout_reconciliation__v5
task has highly viariable availability timing, so it is possible for it to
fail with the following type of error:
Update `bqetl_subplat` DAG description with new expected Stripe report error message. (#5642)
2024-05-23 18:42:21 +03:00
`Report 'frr_...' did not succeed, status was 'pending'`
Bug 1768419 - Add note to subplat dag docs for airflow triage (#2959) * Bug 1768419 - Add note to subplat dag docs for airflow triage * enriched the bqetl_subplat DAG description with a bit more detail regarding expected failures Co-authored-by: kik-kik <kignasiak01@gmail.com>
2022-06-01 16:54:19 +03:00
In such cases the failure is expected, the task will continue to retry every
30 minutes until the data becomes available. If failure observed looks
different then it should be reported using the Airflow triage process.
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
start_date: "2021-07-20"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
bqetl_mozilla_vpn_site_metrics:
wait for GA data (#1973) Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2021-04-27 22:19:23 +03:00
schedule_interval: 0 15 * * *
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
description: |
Daily extracts from the Google Analytics tables for Mozilla VPN as well as
derived tables based on that data.
Depends on Google Analytics exports, which have highly variable timing, so
wait for GA data (#1973) Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2021-04-27 22:19:23 +03:00
queries depend on site_metrics_empty_check_v1, which retries every 30
minutes to wait for data to be available.
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
start_date: "2021-04-22"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
bqetl_gud:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
description: Optimized tables that power the
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
[Mozilla Growth and Usage Dashboard](https://gud.telemetry.mozilla.org).
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
default_args:
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
bqetl_messaging_system:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Daily aggregations on top of pings sent for the `messaging_system`
namespace by desktop Firefox.
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
default_args:
owner: najiang@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "najiang@mozilla.com"]
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
bqetl_activity_stream:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Daily aggregations on top of pings sent for the `activity_stream`
namespace by desktop Firefox. These are largely related to activity
on the newtab page and engagement with Pocket content.
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
default_args:
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
owner: anicholson@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
email: ["telemetry-alerts@mozilla.com", "anicholson@mozilla.com"]
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
bqetl_search:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
email:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
- "telemetry-alerts@mozilla.com"
- "anicholson@mozilla.com"
- "akomar@mozilla.com"
search_revenue_levers_daily_v1 table (#3534) * add new data table search_revenue_levers_daily * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Update sql/moz-fx-data-shared-prod/search_derived/search_revenue_levers_daily_v1/query.sql Co-authored-by: Curtis Morales <cmorales@mozilla.com> * Restore deleted tasks * Change DOU to DAU * Add Bing cohort and tidy query * Format * Add schema file for search_revenue_levers_daily * Add back check for distribution_id and fix tagged_follow_on * Update dags * One more generated task * Be explicit in group by and order * Fix comment * Delete commented out line * Format * Add acer_cohort table to moz-fx-data-shared-prod * Rename query to init.sql * Update acer cohort query * Delete no longer needed comment * Drop references to mozdata * Update sql/moz-fx-data-shared-prod/search_derived/acer_cohort_v1/init.sql Co-authored-by: akkomar <akkomar@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/search_derived/acer_cohort_v1/metadata.yaml Co-authored-by: akkomar <akkomar@users.noreply.github.com> * Update schema * Remove clustering from table --------- Co-authored-by: Xuan Luo <xluo@mozilla.com> Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-02-01 22:29:49 +03:00
- "cmorales@mozilla.com"
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
bqetl_addons:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 4 * * *
description: |
Daily rollups of addon data from `main` pings.
Depends on `bqetl_search`, so is scheduled after that DAG.
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
default_args:
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
owner: kik@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
email:
- "telemetry-alerts@mozilla.com"
- "kik@mozilla.com"
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
DAGs for client queries
2020-06-23 19:22:41 +03:00
bqetl_devtools:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Summarizes usage of the Dev Tools component of desktop Firefox.
DAGs for client queries
2020-06-23 19:22:41 +03:00
default_args:
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
DAGs for client queries
2020-06-23 19:22:41 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
DAGs for client queries
2020-06-23 19:22:41 +03:00
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
bqetl_main_summary:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
General-purpose derived tables for analyzing usage of desktop Firefox.
This is one of our highest-impact DAGs and should be handled carefully.
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
default_args:
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
email:
[
"telemetry-alerts@mozilla.com",
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
"ascholtz@mozilla.com",
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
]
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
Refactoring
2020-06-25 22:44:37 +03:00
bqetl_experiments_daily:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add Markdown descriptions to DAGs
2021-01-30 02:04:42 +03:00
The DAG schedules queries that query experimentation related
metrics (enrollments, search, ...) from stable tables to finalize
numbers of experiment monitoring datasets for a specific date.
DAG for daily experiments clients
2020-06-25 00:42:55 +03:00
default_args:
Experiment enrollment monitoring queries (#1656) * Experiment enrollment aggregates hourly * Experiment enrollments recents query * Add execution_delay support for tasks * Experiment enrollment aggregates base query * Schedule experiment enrollment cumulative population estimate and active population * Experiment enrollment monitoring queries as views * Script for exporting experiment monitoring data to GCS * Export experiment monitoring data script aggregating data of longer running experiments * Parallelize experiment monitoring data export * init.sql for experiment enrollment monitoring queries * Use Airflow ds_format macro for hourly destination table * Use Airflow macros for experiments monitoring hourly execution delay * experiment_enrollment_cumulative_population_estimate as query instead of view * Fix referenced tables in enrollment_aggregates_hourly metadata and add comment * Simplify cumulative population estimate query
2021-01-14 00:53:32 +03:00
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
Experiment enrollment monitoring queries (#1656) * Experiment enrollment aggregates hourly * Experiment enrollments recents query * Add execution_delay support for tasks * Experiment enrollment aggregates base query * Schedule experiment enrollment cumulative population estimate and active population * Experiment enrollment monitoring queries as views * Script for exporting experiment monitoring data to GCS * Export experiment monitoring data script aggregating data of longer running experiments * Parallelize experiment monitoring data export * init.sql for experiment enrollment monitoring queries * Use Airflow ds_format macro for hourly destination table * Use Airflow macros for experiments monitoring hourly execution delay * experiment_enrollment_cumulative_population_estimate as query instead of view * Fix referenced tables in enrollment_aggregates_hourly metadata and add comment * Simplify cumulative population estimate query
2021-01-14 00:53:32 +03:00
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
DAG for daily experiments clients
2020-06-25 00:42:55 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
# DAG for exporting query data marked as public to GCS
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
# queries should not be explicitly assigned to this DAG (done automatically)
Use consistent naming for DAG publishing JSON
2020-06-01 22:40:03 +03:00
bqetl_public_data_json:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 5 * * *
description: |
Daily exports of query data marked as public to GCS.
Depends on the results of several upstream DAGs, the latest of which
runs at 04:00 UTC.
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
default_args:
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-04-14"
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
bqetl_internet_outages:
updated schedule datetime to run later due to query running with missing data (#2757) Co-authored-by: kik <=>
2022-02-28 15:01:20 +03:00
schedule_interval: 0 7 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Update DAGs with Markdown description
2021-01-30 02:25:05 +03:00
DAG for building the internet outages datasets.
See [bug 1640204](https://bugzilla.mozilla.org/show_bug.cgi?id=1640204).
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
default_args:
owner: aplacitelli@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-01-01"
Updates to owners of queries and DAGs (#2289) * Updates to owners of queries and DAGs This is intended to generate some discussion about who new owners should be and to get that codified. * Make ascholtz owner of experiments daily active clients * Add klukas as addons owner + generate dags
2021-08-31 21:38:22 +03:00
email: ["aplacitelli@mozilla.com"]
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Add query for monitoring deletion request volume (#1106)
2020-07-13 21:11:06 +03:00
bqetl_deletion_request_volume:
schedule_interval: 0 1 * * *
default_args:
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
owner: akomar@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-06-29"
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
email: ["telemetry-alerts@mozilla.com", "akomar@mozilla.com"]
Add query for monitoring deletion request volume (#1106)
2020-07-13 21:11:06 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create DAG for events rollup (#1301) * Create DAG for events rollup * Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-11 01:50:45 +03:00
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
bqetl_fenix_event_rollup:
schedule_interval: 0 2 * * *
Create DAG for events rollup (#1301) * Create DAG for events rollup * Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-11 01:50:45 +03:00
default_args:
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
owner: wlachance@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-09-09"
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
email: ["wlachance@mozilla.com"]
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
bqetl_org_mozilla_fenix_derived:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 2 * * *
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
default_args:
depends_on_past: false
email:
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
- amiyaguchi@mozilla.com
- telemetry-alerts@mozilla.com
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
email_on_failure: true
email_on_retry: true
owner: amiyaguchi@mozilla.com
retries: 2
retry_delay: 30m
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-10-18"
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
Bug 1803425 - Create clients_yearly table for Fenix clients (#3383) * Create clients_yearly table for Fenix clients This table contains a year of history for Fenix. It utilizes similar logic to search.search_clients_last_seen, but the naming specifies the year-long byte field. Currently only looks at Fenix data, which migrated ~2020-09-01; so full data is only available beginning 2021-09-01. * Add schema for baseline_clients_daily * Move to referenced tables * Remove duplicates from baseline_clients_daily * Add dependency on baseline_clients_daily
2022-12-01 19:21:54 +03:00
bqetl_org_mozilla_firefox_derived:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
- frank@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
owner: frank@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2022-11-30"
tags:
- impact/tier_1
Bug 1810236 - Generate additional deletion requests for focus (#3604)
2023-02-23 01:51:06 +03:00
bqetl_org_mozilla_focus_derived:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
- akomar@mozilla.com
Bug 1810236 - Generate additional deletion requests for focus (#3604)
2023-02-23 01:51:06 +03:00
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
owner: akomar@mozilla.com
Bug 1810236 - Generate additional deletion requests for focus (#3604)
2023-02-23 01:51:06 +03:00
retries: 2
retry_delay: 30m
start_date: "2023-02-22"
tags:
- impact/tier_1
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
bqetl_google_analytics_derived:
Change GA dag to run later (#1515)
2020-11-04 22:47:32 +03:00
schedule_interval: 0 23 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Daily aggregations of data exported from Google Analytics.
The GA export runs at 15:00 UTC, so there's an effective 2-day delay
for user activity to appear in these tables.
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
default_args:
Assign kwindau as owner of bqetl_google_analytics (#5492) Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com>
2024-05-03 22:47:41 +03:00
owner: kwindau@mozilla.com
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
email:
Assign kwindau as owner of bqetl_google_analytics (#5492) Co-authored-by: Katie Windau <153020235+kwindau@users.noreply.github.com>
2024-05-03 22:47:41 +03:00
- kwindau@mozilla.com
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
- telemetry-alerts@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-10-31"
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
bqetl_monitoring:
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Update DAGs with Markdown description
2021-01-30 02:25:05 +03:00
This DAG schedules queries and scripts for populating datasets
used for monitoring of the data platform.
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
default_args:
owner: ascholtz@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
email: ["ascholtz@mozilla.com"]
start_date: "2018-10-30"
Add event_types query generation
2020-11-12 17:10:07 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add event_types query generation
2020-11-12 17:10:07 +03:00
feat(): Adding ETL for monitoring_airflow datasets (#3204) * added sql logic for monitoring_airflow * bqetl_monitoring_airflow added to dags.yaml * added .probe_cache/ to gitignore * generated bqetl_monitoring_airflow dag * added monitoring_airflow_derived to dryrun ignore as it fails to access fivetran data referenced in this dataset * moved airflow views and sql files to monitoring dataset * manually triggering fivetran load of airflow metadata as suggested by @ascholtzan in PR#3204 * added schemas as requested by @scholtzan in PR#3204 * added descriptions to airflow_monitoring datasets * fixed airflow_dag view folder name * corrected dryrun ignore for monitoring_derived/airflow*
2022-10-12 12:57:24 +03:00
bqetl_monitoring_airflow:
DENG-797 change airflow DAG schedule (#4267)
2023-09-08 18:53:07 +03:00
schedule_interval: 0 10 * * *
feat(): Adding ETL for monitoring_airflow datasets (#3204) * added sql logic for monitoring_airflow * bqetl_monitoring_airflow added to dags.yaml * added .probe_cache/ to gitignore * generated bqetl_monitoring_airflow dag * added monitoring_airflow_derived to dryrun ignore as it fails to access fivetran data referenced in this dataset * moved airflow views and sql files to monitoring dataset * manually triggering fivetran load of airflow metadata as suggested by @ascholtzan in PR#3204 * added schemas as requested by @scholtzan in PR#3204 * added descriptions to airflow_monitoring datasets * fixed airflow_dag view folder name * corrected dryrun ignore for monitoring_derived/airflow*
2022-10-12 12:57:24 +03:00
description: |
This DAG schedules queries and scripts for populating datasets
used for monitoring of Airflow DAGs.
default_args:
chore(): updating DAG owner for kignasiak to kik to keep it consistent (#4178) * updating DAG owner for kignasiak to kik to keep it consistent * regenerated dags
2023-08-14 17:34:12 +03:00
owner: kik@mozilla.com
email: ["kik@mozilla.com"]
feat(): Adding ETL for monitoring_airflow datasets (#3204) * added sql logic for monitoring_airflow * bqetl_monitoring_airflow added to dags.yaml * added .probe_cache/ to gitignore * generated bqetl_monitoring_airflow dag * added monitoring_airflow_derived to dryrun ignore as it fails to access fivetran data referenced in this dataset * moved airflow views and sql files to monitoring dataset * manually triggering fivetran load of airflow metadata as suggested by @ascholtzan in PR#3204 * added schemas as requested by @scholtzan in PR#3204 * added descriptions to airflow_monitoring datasets * fixed airflow_dag view folder name * corrected dryrun ignore for monitoring_derived/airflow*
2022-10-12 12:57:24 +03:00
start_date: "2022-09-01"
retries: 2
retry_delay: 30m
tags:
- impact/tier_2
Add event_types query generation
2020-11-12 17:10:07 +03:00
bqetl_event_rollup:
schedule_interval: 0 3 * * *
Bug 1805722 - Deprecate bqetl_event_rollup (#3542)
2023-01-30 20:35:12 +03:00
description: |
Desktop tables (`telemetry_derived.events_daily_v1` and upstream) are deprecated and paused
(have their scheduling metadata commented out) per https://bugzilla.mozilla.org/show_bug.cgi?id=1805722#c10
Add event_types query generation
2020-11-12 17:10:07 +03:00
default_args:
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
owner: wlachance@mozilla.com
Add event_types query generation
2020-11-12 17:10:07 +03:00
start_date: "2020-11-03"
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
email: ["wlachance@mozilla.com"]
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add queries and views for fetch marketing spend data (#1575)
2020-11-30 23:34:55 +03:00
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
bqetl_iprospect:
schedule_interval: 0 4 * * *
description: |
This DAG imports iProspect data from moz-fx-data-marketing-prod-iprospect.
depends_on_past: false
default_args:
owner: ascholtz@mozilla.com
Update notifications for bqetl_iprospect
2021-11-03 22:32:58 +03:00
email:
[
"ascholtz@mozilla.com",
"echo@mozilla.com",
"shong@mozilla.com"
]
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
start_date: "2021-04-19"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
bqetl_search_dashboard:
default_args:
depends_on_past: false
email:
Enforce more yaml lint rules (#1878)
2021-03-10 01:25:01 +03:00
- telemetry-alerts@mozilla.com
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
- akomar@mozilla.com
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
email_on_failure: true
email_on_retry: true
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
owner: akomar@mozilla.com
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
retries: 2
retry_delay: 30m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-12-14"
Update scheduler of aggregates to run after upstreams. (#4503) * Update scheduler of aggregates to run after upstreams. * Update dags for new scheduler of analytics_aggregates * Update dag bqetl_search * Remove DAG. --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-11-06 19:55:12 +03:00
schedule_interval: 30 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
Adding accessibility clients query for platform accessibility clients (#1612) Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-12-11 23:05:50 +03:00
bqetl_desktop_platform:
schedule_interval: 0 3 * * *
default_args:
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
owner: ascholtz@mozilla.com
Updated accessibility clients query and start date. (#1623)
2020-12-15 20:41:30 +03:00
start_date: "2018-11-01"
Adding accessibility clients query for platform accessibility clients (#1612) Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-12-11 23:05:50 +03:00
email:
[
"telemetry-alerts@mozilla.com",
Reassign DAG and query ownerships to ascholtz and anicholson (#3867)
2023-05-30 20:05:40 +03:00
"ascholtz@mozilla.com",
Adding accessibility clients query for platform accessibility clients (#1612) Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-12-11 23:05:50 +03:00
"yzenevich@mozilla.com",
]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
bqetl_internal_tooling:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
This DAG schedules queries for populating queries related to Mozilla's
internal developer tooling (e.g. mozregression).
default_args:
depends_on_past: false
email:
Enforce more yaml lint rules (#1878)
2021-03-10 01:25:01 +03:00
- wlachance@mozilla.com
- telemetry-alerts@mozilla.com
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
email_on_failure: true
email_on_retry: true
end_date: null
owner: wlachance@mozilla.com
retries: 2
retry_delay: 30m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-06-01"
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Bug 1647328 - Schedule performance release criteria queries (#1596)
2021-03-10 22:27:32 +03:00
bqetl_release_criteria:
schedule_interval: daily
default_args:
owner: perf-pmo@mozilla.com
start_date: "2020-12-03"
email:
- telemetry-alerts@mozilla.com
Remove dthorn as dag owner (#4695)
2023-12-15 21:19:42 +03:00
- esmyth@mozilla.com
Bug 1647328 - Schedule performance release criteria queries (#1596)
2021-03-10 22:27:32 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
bqetl_pocket:
default_args:
depends_on_past: false
email:
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
- kik@mozilla.com
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
owner: kik@mozilla.com
Retry `bqetl_pocket` DAG tasks for 10 hours rather than 1 hour. (#3635) Because the files from Pocket may not always be available on time (e.g. bug 1818043).
2023-03-04 02:00:19 +03:00
# Retry more than normal because the files from Pocket may not always be available on time.
retries: 10
retry_delay: 60m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2021-03-10"
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
description: |
Import of data from Pocket's Snowflake warehouse.
Originally created for [Bug 1695336](
https://bugzilla.mozilla.org/show_bug.cgi?id=1695336).
Bug 1815242 - Update bqetl_pocket triage notes (#4733)
2023-12-20 20:52:54 +03:00
*Triage notes*
As long as the most recent DAG run is successful this job can be considered healthy.
In such case, past DAG failures can be ignored.
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
schedule_interval: 0 12 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Create adMarketplace Daily Engagements table (#1891)
2021-03-17 21:03:54 +03:00
Schedule queries
2021-03-16 18:49:17 +03:00
bqetl_desktop_funnel:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Schedule queries
2021-03-16 18:49:17 +03:00
This DAG schedules desktop funnel queries used to power the
[Numbers that Matter dashboard](https://protosaur.dev/numbers-that-matter/)
Add view for desktop funnel activation
2021-03-19 22:20:05 +03:00
schedule_interval: 0 4 * * *
Schedule queries
2021-03-16 18:49:17 +03:00
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-01-01"
email:
[
"telemetry-alerts@mozilla.com",
"ascholtz@mozilla.com",
]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
bqetl_firefox_ios:
default_args:
depends_on_past: false
email:
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
- kik@mozilla.com
fixing broken test for firefox_ios_derived.baseline_clients_yearly_v1 (#4645)
2023-12-04 18:48:56 +03:00
- frank@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
reassigning DAG ownership to kik as per "adopt a DAG" doc (#3862) * reassigning DAG owners as per "adopt a DAG" doc * removed jeff from bqetl_addons queries
2023-05-31 12:49:11 +03:00
owner: kik@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
retries: 2
retry_delay: 30m
start_date: "2021-03-18"
description: Schedule daily ios firefox ETL
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Schedule release data import
2021-04-14 22:54:28 +03:00
bqetl_releases:
default_args:
depends_on_past: false
email:
- ascholtz@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: ascholtz@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2021-04-14"
description: |
Schedule release data import from https://product-details.mozilla.org/1.0
Add metadata for public release dataset views
2021-04-15 19:28:14 +03:00
For more context, see
https://wiki.mozilla.org/Release_Management/Product_details
Schedule release data import
2021-04-14 22:54:28 +03:00
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
bqetl_ctxsvc_derived:
default_args:
depends_on_past: false
email:
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
- ctroy@mozilla.com
- wstuckey@mozilla.com
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
owner: ctroy@mozilla.com
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
retries: 2
retry_delay: 30m
start_date: '2021-05-01'
description: Contextual services derived tables
schedule_interval: 0 3 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add command to import subscriber report from apple (#2116)
2021-06-15 01:16:38 +03:00
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
bqetl_search_terms_daily:
default_args:
depends_on_past: false
email:
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
- ctroy@mozilla.com
- wstuckey@mozilla.com
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
- rburwei@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
owner: ctroy@mozilla.com
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
retries: 2
retry_delay: 30m
start_date: '2021-09-20'
description: |
Derived tables on top of search terms data.
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
Note that the tasks for populating `suggest_impression_sanitized_v*` are
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
particularly important because the source unsanitized dataset has only
a 2-day retention period, so errors fairly quickly become unrecoverable
and can impact reporting to partners. If this task errors out, it could
indicate trouble with an upstream task that runs in a restricted project
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
outside of Airflow. Contact `ctroy`, `wstuckey`, `whd`, and `jbuck`.
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
schedule_interval: 0 3 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
- impact/tier_1
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
Schedule experimenter_experiments_import
2021-07-22 01:40:18 +03:00
bqetl_experimenter_experiments_import:
schedule_interval: "*/10 * * * *"
description: |
Imports experiments from the Experimenter V4 and V6 API.
Imported experiment data is used for experiment monitoring in
[Grafana](https://grafana.telemetry.mozilla.org/d/XspgvdxZz/experiment-enrollment).
default_args:
owner: ascholtz@mozilla.com
start_date: "2020-10-09"
retries: 0
email:
- ascholtz@mozilla.com
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Move feature_usage_v2 to separate DAG
2021-08-03 19:18:50 +03:00
bqetl_feature_usage:
schedule_interval: 0 5 * * *
description: |
Daily aggregation of browser features usages from `main` pings,
`event` pings and addon data.
Depends on `bqetl_addons` and `bqetl_main_summary`, so is scheduled after.
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-01-01"
email:
- "telemetry-alerts@mozilla.com"
- "ascholtz@mozilla.com"
- "loines@mozilla.com"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
bqetl_urlbar:
schedule_interval: 0 3 * * *
description: |
Daily aggregation of metrics related to urlbar usage.
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
start_date: "2021-08-01"
email:
- "telemetry-alerts@mozilla.com"
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
- "anicholson@mozilla.com"
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
- "akomar@mozilla.com"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Schedule query for unified metrics
2021-10-13 01:59:24 +03:00
bqetl_unified:
schedule_interval: 0 3 * * *
description: |
Schedule queries that unify metrics across all products.
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-10-12"
email:
- "telemetry-alerts@mozilla.com"
- "ascholtz@mozilla.com"
- "loines@mozilla.com"
Create derived dataset and DAG for the aggregation of active users. (#2952) * Create derived dataset and DAG for the aggregation of active users. * Change APROX_DISTINCT_COUNT for COUNT(DISTINCT) * Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity. * Improve descriptions * Add clustering by channel * DAG update * Clustering based on users most common filtering * Add notification for analytics DAG * agg_active_users to query from telemetry_derived. * Add query to create agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Add date to init.sql * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/mozfun/bytes/zero_right/metadata.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Remove CAST from init.sql for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql * Update project in ini.sql to `moz-fx-data-shared-prod`. * Update name of aggregation and DAGS for consistency * Add active_users_aggregates_v1 to align with current naming convention * Update to active_users_aggregates * Update query for active_users_aggregates * Add uri_count and active_hours query for active_users_aggregates * Format query, update DAG to remove agg_active_users_v1 * Generate DAG to correct CI error. Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 17:23:50 +03:00
- "lvargas@mozilla.com"
Schedule query for unified metrics
2021-10-13 01:59:24 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add regrets reporter summary table (fixes #2572) (#2574) * added regrets_reporter_summary query * regrets_reporter_summary init added * submission date modified to be dynamic @submission_date * added labels * Apply suggestions from code review by @wlach Fixing SQL logic Co-authored-by: Will Lachance <wlachance@mozilla.com> * Apply suggestions from code review by @jklukas Co-authored-by: Jeff Klukas <jklukas@mozilla.com> * added metadata.yaml for regrets_reporter_summary view * added regrets_reporter_summary to dry_run exclusion * regrets_reporter_summary dag updated Co-authored-by: = <=> Co-authored-by: Will Lachance <wlachance@mozilla.com> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-12-16 21:14:51 +03:00
bqetl_regrets_reporter_summary:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
chore(): updating DAG owner for kignasiak to kik to keep it consistent (#4178) * updating DAG owner for kignasiak to kik to keep it consistent * regenerated dags
2023-08-14 17:34:12 +03:00
- kik@mozilla.com
Add regrets reporter summary table (fixes #2572) (#2574) * added regrets_reporter_summary query * regrets_reporter_summary init added * submission date modified to be dynamic @submission_date * added labels * Apply suggestions from code review by @wlach Fixing SQL logic Co-authored-by: Will Lachance <wlachance@mozilla.com> * Apply suggestions from code review by @jklukas Co-authored-by: Jeff Klukas <jklukas@mozilla.com> * added metadata.yaml for regrets_reporter_summary view * added regrets_reporter_summary to dry_run exclusion * regrets_reporter_summary dag updated Co-authored-by: = <=> Co-authored-by: Will Lachance <wlachance@mozilla.com> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-12-16 21:14:51 +03:00
email_on_failure: true
email_on_retry: true
end_date: null
chore(): updating DAG owner for kignasiak to kik to keep it consistent (#4178) * updating DAG owner for kignasiak to kik to keep it consistent * regenerated dags
2023-08-14 17:34:12 +03:00
owner: kik@mozilla.com
Add regrets reporter summary table (fixes #2572) (#2574) * added regrets_reporter_summary query * regrets_reporter_summary init added * submission date modified to be dynamic @submission_date * added labels * Apply suggestions from code review by @wlach Fixing SQL logic Co-authored-by: Will Lachance <wlachance@mozilla.com> * Apply suggestions from code review by @jklukas Co-authored-by: Jeff Klukas <jklukas@mozilla.com> * added metadata.yaml for regrets_reporter_summary view * added regrets_reporter_summary to dry_run exclusion * regrets_reporter_summary dag updated Co-authored-by: = <=> Co-authored-by: Will Lachance <wlachance@mozilla.com> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-12-16 21:14:51 +03:00
retries: 2
retry_delay: 30m
start_date: '2021-12-12'
description: Measure usage of the regrets reporter addon
schedule_interval: 0 4 * * *
tags:
- impact/tier_1
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
bqetl_cjms_nonprod:
DVPN-89 - Catch all flows at end of day for cjms nonprod (#2873)
2022-04-07 00:31:21 +03:00
schedule_interval: 0 * * * *
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
description: |
Hourly ETL for cjms nonprod.
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
start_date: "2022-03-24"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
retries: 2
retry_delay: 5m
tags:
- impact/tier_3
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
bqetl_acoustic_contact_export:
pushing bqetl acoustic dags by 1 hour to see if that resolves sensor issues (#3002)
2022-06-02 16:18:41 +03:00
schedule_interval: 0 9 * * *
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
description: |
Processing data loaded by
fivetran_acoustic_contact_export
DAG to clean up the data loaded from Acoustic.
default_args:
Deng 3287 move acoustic data from marketing to shared prod (#5316) * DENG-3151 create acoustic_external dataset in BigQuery * DENG-3287 move data from marketing to shared-prod * acoustic_derived dataset
2024-04-03 17:26:44 +03:00
owner: leli@mozilla.com
fix: DENG-3287 change start date for DAGs acoustic (#5335)
2024-04-05 17:39:52 +03:00
start_date: "2024-04-03"
Deng 3287 move acoustic data from marketing to shared prod (#5316) * DENG-3151 create acoustic_external dataset in BigQuery * DENG-3287 move data from marketing to shared-prod * acoustic_derived dataset
2024-04-03 17:26:44 +03:00
email: ["telemetry-alerts@mozilla.com", "leli@mozilla.com"]
bug(1844385): bqetl_acoustic_contact_export contacts_raw_v1 failing due to invalid schema update error. (#4142) * ensuring double_opt_in field is of type INTEGER and reduced the number of retries for this job * rerun dag creation * change retry count in metadata and rerun dag creation * changed the number of retries for acoustic DAGs to 1 --------- Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>
2023-07-31 19:12:09 +03:00
retries: 1
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
retry_delay: 5m
tags:
- impact/tier_3
bqetl_acoustic_raw_recipient_export:
pushing bqetl acoustic dags by 1 hour to see if that resolves sensor issues (#3002)
2022-06-02 16:18:41 +03:00
schedule_interval: 0 9 * * *
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
description: |
Processing data loaded by
fivetran_acoustic_raw_recipient_export
DAG to clean up the data loaded from Acoustic.
default_args:
Deng 3287 move acoustic data from marketing to shared prod (#5316) * DENG-3151 create acoustic_external dataset in BigQuery * DENG-3287 move data from marketing to shared-prod * acoustic_derived dataset
2024-04-03 17:26:44 +03:00
owner: leli@mozilla.com
fix: DENG-3287 change start date for DAGs acoustic (#5335)
2024-04-05 17:39:52 +03:00
start_date: "2024-04-03"
Deng 3287 move acoustic data from marketing to shared prod (#5316) * DENG-3151 create acoustic_external dataset in BigQuery * DENG-3287 move data from marketing to shared-prod * acoustic_derived dataset
2024-04-03 17:26:44 +03:00
email: ["telemetry-alerts@mozilla.com", "leli@mozilla.com"]
bug(1844385): bqetl_acoustic_contact_export contacts_raw_v1 failing due to invalid schema update error. (#4142) * ensuring double_opt_in field is of type INTEGER and reduced the number of retries for this job * rerun dag creation * change retry count in metadata and rerun dag creation * changed the number of retries for acoustic DAGs to 1 --------- Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>
2023-07-31 19:12:09 +03:00
retries: 1
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
retry_delay: 5m
tags:
- impact/tier_3
Create derived dataset and DAG for the aggregation of active users. (#2952) * Create derived dataset and DAG for the aggregation of active users. * Change APROX_DISTINCT_COUNT for COUNT(DISTINCT) * Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity. * Improve descriptions * Add clustering by channel * DAG update * Clustering based on users most common filtering * Add notification for analytics DAG * agg_active_users to query from telemetry_derived. * Add query to create agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Add date to init.sql * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/mozfun/bytes/zero_right/metadata.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Remove CAST from init.sql for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql * Update project in ini.sql to `moz-fx-data-shared-prod`. * Update name of aggregation and DAGS for consistency * Add active_users_aggregates_v1 to align with current naming convention * Update to active_users_aggregates * Update query for active_users_aggregates * Add uri_count and active_hours query for active_users_aggregates * Format query, update DAG to remove agg_active_users_v1 * Generate DAG to correct CI error. Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 17:23:50 +03:00
bqetl_analytics_aggregations:
default_args:
depends_on_past: false
email:
- "telemetry-alerts@mozilla.com"
- "lvargas@mozilla.com"
- "gkaberere@mozilla.com"
email_on_failure: true
email_on_retry: true
end_date: null
owner: lvargas@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-05-12'
description: Scheduler to populate the aggregations required for
analytics engineering and reports optimization.
It provides data to build growth, search and usage metrics, as well
as acquisition and retention KPIs, in a model that facilitates
reporting in Looker.
Update scheduler of aggregates to run after upstreams. (#4503) * Update scheduler of aggregates to run after upstreams. * Update dags for new scheduler of analytics_aggregates * Update dag bqetl_search * Remove DAG. --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-11-06 19:55:12 +03:00
schedule_interval: 15 4 * * *
Create derived dataset and DAG for the aggregation of active users. (#2952) * Create derived dataset and DAG for the aggregation of active users. * Change APROX_DISTINCT_COUNT for COUNT(DISTINCT) * Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity. * Improve descriptions * Add clustering by channel * DAG update * Clustering based on users most common filtering * Add notification for analytics DAG * agg_active_users to query from telemetry_derived. * Add query to create agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Add date to init.sql * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/mozfun/bytes/zero_right/metadata.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Remove CAST from init.sql for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql * Update project in ini.sql to `moz-fx-data-shared-prod`. * Update name of aggregation and DAGS for consistency * Add active_users_aggregates_v1 to align with current naming convention * Update to active_users_aggregates * Update query for active_users_aggregates * Add uri_count and active_hours query for active_users_aggregates * Format query, update DAG to remove agg_active_users_v1 * Generate DAG to correct CI error. Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 17:23:50 +03:00
tags:
- impact/tier_1
Decision support metrics for fog initial commit (#2984) * decision support metrics for fog initial commit * reverting overwrite * init.sql to initialize table * changing to main_1pct for the time being * init sql correction * missing as keyword * updating dag order * trying main 1pct * removed init.sql, added schema.yaml * update dag to reflect different table wait * re-generated DAG * fix tag * fix tags in dag * no clustering
2022-06-02 00:00:29 +03:00
bqetl_fog_decision_support:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- pmcmanis@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: pmcmanis@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-05-25'
description: This DAG schedules queries for calculating FOG decision
support metrics
schedule_interval: 0 4 * * *
tags:
- impact/tier_3
- repo/bigquery-etl
RS-306 - ETL for newtab interactions (#3073)
2022-07-18 23:32:03 +03:00
bqetl_newtab:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- anicholson@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: anicholson@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-07-01'
description: Schedules newtab related queries.
schedule_interval: daily
tags:
- impact/tier_1
Add desktop mobile monthly search agg (#3152) * add desktop+mobile monthly search aggregates * apply feedback * generate dag via dags.yaml * update dag to reference empty tables * correct yamllint issues
2022-08-18 14:47:22 +03:00
bqetl_desktop_mobile_search_monthly:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- akommasani@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: akommasani@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2019-01-01'
description: Generate mnthly client data from daily search table
schedule_interval: "0 5 2 * *"
tags:
- impact/tier_1
- repo/bigquery-etl
[DENG-358] New dataset for collecting domain metadata and new top_domains table (#3282) * New dataset for collecting domain metadata and new top_domains table * f lint * Add dag, schema and update to monthly schedule * lint * Add required tag and partition column * format fix * Explicit column names in the final query * Fix schema issues
2022-10-20 01:13:43 +03:00
bqetl_domain_meta:
default_args:
depends_on_past: false
email:
- wstuckey@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: wstuckey@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-10-13'
description: Domain metadata
schedule_interval: monthly
tags:
Add support for `table_partition_template` in dag task generation (#3710) * Update domain metadata dag. * Remove from triage with tags * Remove telemetry-alerts email * Add date formatting for monthly partition id * Add support for `table_partition_format` in dag generation * Don't add partition format if there's already a destination table * use the correct name * Add partition templates for all time partitioning types * lint fixes * more docs * update all dags to include `table_partition_template` parameter. * don't set if we have a partition offset * don't add the parameter for the default 'day' partitioning scheme
2023-04-12 18:28:19 +03:00
- impact/tier_3
- triage/no_triage
[DENG-358] New dataset for collecting domain metadata and new top_domains table (#3282) * New dataset for collecting domain metadata and new top_domains table * f lint * Add dag, schema and update to monthly schedule * lint * Add required tag and partition column * format fix * Explicit column names in the final query * Fix schema issues
2022-10-20 01:13:43 +03:00
- repo/bigquery-etl
ADPRODUCTS-367: sponsored_tiles_clients_daily (#3203) * sponsored_tiles_clients_daily Client-level code for desktop + mobile. Issue with UNION ALL for desktop and mobile tables as the experiments structure is different between the devices * Delete .pre-commit-config.yaml * fix: experiments * initial commit for sponsored tiles clients daily * Add submission_date * Clean up dag imports * Revert import change, add dag to main summary * Nan's edits Co-authored-by: Rebecca BurWei <rburwei@mozilla.com> Co-authored-by: Perry McManis <pmcmanis@mozilla.com> Co-authored-by: Wil Stuckey <wstuckey@mozilla.com> Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2022-10-31 17:50:11 +03:00
bqetl_sponsored_tiles_clients_daily:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- skahmann@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: skahmann@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-09-13'
description: daily run of sponsored tiles related fields
schedule_interval: 0 4 * * *
tags:
- impact/tier_3
- repo/bigquery-etl
Activation metric for mobile new profiles (#3347) * Fenix and Firefox iOS new profile activation metric * Update DAGs * parenthesis for readability of between statements Co-authored-by: Curtis Morales <cmorales@mozilla.com> * remove redundant filters for incremental build Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2022-12-01 03:08:17 +03:00
bqetl_mobile_activation:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- vsabino@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: vsabino@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2021-01-01'
description: Queries related to the mobile activation metric used by Marketing
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
Create table, view, DAG for Firefox Android Clients. (#3379) * Create table, view, DAG for Firefox Android Clients. * Normalization of adjust_network and install_source in the view, typo. Move dataset from telemetry_derived to fenix_derived. * DENG-178_ Update logic to use fenix.first_session instead of org_mozilla_firefox, consider pings received after client's first seen date. Use first_run_date as datetime for comparison with metrics datetimes. Remove condition of metrics ping without adjust data to differentiate when a ping is not received and when is empty. Health check date types. Update clustering as adjust data is more likely to be null. Find deterministic values for fisr_session ping data. Remove uniqueness validation of clients in different metrics pings. Create UDF for the logic of finding the first adjust (value,datetime) pair. * DENG-178_ Refactor query for readibility and consistency with existing datasets. * Update clustering and description in metadata. Collect only `first seen` clients & metrics ping data for channel release. Update LEAST for COALESCE to avoid NULL returned. Collect core dimensions from `baseline_clients_first_seen` for data completeness when first_session ping is not reported. * Delete schema file. * Reducing duplicated logic, readability. Adding sample_id to the table and the clustering. * Compare to find first value also for first_seen_date,submission_date, first_run_date, first_reported_country, first_reported_isp and channel. * DAG partition parameter NULL. FUll outer join in init to collect ping's data for not yet first_seen clients. Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-12-13 23:21:10 +03:00
bqetl_analytics_tables:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- lvargas@mozilla.com
- gkaberere@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: lvargas@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-12-01'
description: Scheduled queries for analytics tables.
engineering.
schedule_interval: 0 2 * * *
tags:
- impact/tier_1
- repo/bigquery-etl
Add Campaign Conversions by Date table (#3484) * Create campaign_conversions_by_date table * Update query for dupes; dont dryrun * Remove ORDER BY; update description * Update DAG
2023-01-05 18:15:16 +03:00
bqetl_fivetran_google_ads:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- frank@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: frank@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-01-01'
description: Queries for Google Ads data
coming from Fivetran. Fivetran
updates these tables every hour.
schedule_interval: 0 2 * * *
tags:
- impact/tier_2
- repo/bigquery-etl
Add and incrementally populate a table for google ads campaign cost metrics (#3468) * Add and incrementally populate a table for google ads campaign cost metrics * Register dag in dags.yaml * make two strings match that apparently have to match * Consistentify another thing in the dag * Reformat a sql file * Update sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/metadata.yaml Add update dependency on upstream table Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Small adjustments apropos of review * Attempt to fix verify-dags-up-to-date * Evidently there are too many blank lines in the metadata file * Stop dry running an access denied table Also raise the line length limit on the linter, which otherwise prevented the needed change * represent micros more accurately * Update dags.yaml to include Frank as maintainer Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * rename ad_clicks * Update sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Add conversions to columns of output table * Group stats table by campaign id and date * Add revenue generating ad clicks and disambiguate that number from marketing ad clicks * SO IT TURNS OUT, campaigns with the same ID can have different names over time. This associates the appropriate names with each id so we can sum up metrics by campaign rather than by NAME of campaign. * ./bqetl format /Users/chelseatroy/mozilla/bigquery-etl/sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/query.sql * Document the fenix campaign identifier * ./bqetl format * ./bqetl dag generate bqetl_campaign_cost_breakdowns * ./bqetl dag generate bqetl_org_mozilla_firefox_derived Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2023-01-12 00:58:10 +03:00
bqetl_campaign_cost_breakdowns:
default_args:
depends_on_past: false
email:
- ctroy@mozilla.com
- frank@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: ctroy@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2021-09-20'
description: |
Derived tables on top of fenix installation and DOU metrics,
as well as Google ads campaign data.
schedule_interval: 0 3 * * *
tags:
- impact/tier_2
- repo/bigquery-etl
Add fivetran_costs (#3509) * Add fivetran_costs * Add fivetran_costs - adding schemas to all tables and adressing other suggestions * Add fivetran_costs - renaming tables, adding to the schema * Add fivetran_costs - adding fivetran-dev * Add fivetran_costs - adding tests * Add fivetran_costs - adding tests * Add fivetran_costs - adding tests * Add fivetran_costs - adding tests * implementing suggestions * rerun dag creation
2023-01-31 18:27:29 +03:00
bqetl_fivetran_costs:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- lschiestl@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: lschiestl@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-01-18'
description: |
Derived tables for analyzing the Fivetran Costs. Data coming from Fivetran.
repo: bigquery-etl
schedule_interval: 0 5 * * *
tags:
- impact/tier_3
feat(MP-267): Move popularities export to Airflow (#3598) * added mdn_yari_derived namespace along with mdn_popularities_v1 query to support mdn_popularities DAG inside telemetry-airflow * added query.py as an alternative to exporting the data * removed query.sql for mdn_yari.mdn_popularities_v1 * Updated query.py for mdn_popularities_v1 to move the blob to target location and clean up after * made changes as requested in PR#3598 by akkomar
2023-02-23 12:38:12 +03:00
bqetl_mdn_yari:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
fix: replace MDN contact email address (#3678) `mdn@mozilla.com` does not actually exist.
2023-03-27 22:02:25 +03:00
- mdn-infra@mozilla.com
feat(MP-267): Move popularities export to Airflow (#3598) * added mdn_yari_derived namespace along with mdn_popularities_v1 query to support mdn_popularities DAG inside telemetry-airflow * added query.py as an alternative to exporting the data * removed query.sql for mdn_yari.mdn_popularities_v1 * Updated query.py for mdn_popularities_v1 to move the blob to target location and clean up after * made changes as requested in PR#3598 by akkomar
2023-02-23 12:38:12 +03:00
- fmerz@mozilla.com
chore(): updating DAG owner for kignasiak to kik to keep it consistent (#4178) * updating DAG owner for kignasiak to kik to keep it consistent * regenerated dags
2023-08-14 17:34:12 +03:00
- kik@mozilla.com
feat(MP-267): Move popularities export to Airflow (#3598) * added mdn_yari_derived namespace along with mdn_popularities_v1 query to support mdn_popularities DAG inside telemetry-airflow * added query.py as an alternative to exporting the data * removed query.sql for mdn_yari.mdn_popularities_v1 * Updated query.py for mdn_popularities_v1 to move the blob to target location and clean up after * made changes as requested in PR#3598 by akkomar
2023-02-23 12:38:12 +03:00
email_on_failure: true
email_on_retry: false
end_date: null
owner: fmerz@mozilla.com
retries: 1
retry_delay: 5m
start_date: '2023-02-01'
description: |
Monthly data exports of MDN 'Popularities'. This aggregates and counts total
page visits and normalizes them agains the max.
schedule_interval: 0 0 1 * *
tags:
- impact/tier_3
- triage/record_only
[Bug 1826618] bqetl status checks (#3706) * Add check for running SQL query * Add check for monitoring python script runs
2023-04-06 01:24:09 +03:00
bqetl_status_check:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
end_date: null
owner: ascholtz@mozilla.com
retries: 0
start_date: '2023-04-01'
description: |
This DAG checks if bigquery-etl is working properly. Dummy ETL tasks are executed to detect
breakages as soon as possible.
*Triage notes*
None of these tasks should fail. If they do it is very likely that other/all ETL tasks will
subsequently fail as well. Any failures should be communicated to the Data Infra Working Group
as soon as possible.
schedule_interval: "1h"
tags:
- impact/tier_1
DENG-775 downloads_with_attribution_v2 (#3716) * DENG-775 Added session_id to JOIN between GA data and stub_attr.stdout. Also expanded date range on GA session data to [download_date - 2 days, download_date + 1 day] * Updated query to handle missing GA download_session_id. It effectively applies V1 logic to the MISSING_GA_CLIENT dl_tokens.
2023-04-26 21:47:34 +03:00
Firefox mobile installs from Adjust (#3735)
2023-05-04 07:07:16 +03:00
bqetl_adjust:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
Ds 2944 create external adjust table redux (#3907) * add metadata, schema yamls and query.py * created adjust_derived namespace * add query.py, metadata, schema, dataset for testing * delete extraneous file, update DAG name * modify bqetl_adjust DAG redux * update DAG name, take out '_derived' * update table name in view * standardize table names across files * regenerate DAG * update schema in both locations * add query.py, metadata, schema yaml files * take put extraneous print statements, update datasets to be 'adjust' or 'adjust_derived' * add submission date to date_partition_parameter * update table name to be just one table * add DAG for adjust_derived * add bq_etl adjust_derived DAG to yaml file * add note about API token * revert changes to bqetl.adjust.py * use proper tast_id * fix start dates * add python command and docker image * add python command and docker image * delete extraneous code * comment out docker part in old adjust dag * add whitespace, delete extraneous code * Update sql/moz-fx-data-shared-prod/adjust/adjust_derived/view.sql Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * updated logic to check if response dictionary is not empty, moved view out of nested folder, added token ownership statement to metadata file, turned off email retry in dags.yaml, separated out clean up of json to its own function * take out extraneous if statement and move else statement * reorder where comment is to make more sense * more description as to why we're using mhirose's API token * take out periods * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * combine adjust DAGs * change logic for query_export check loop continuance, adapt metadata.yamls * add blank parameters test * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * add arguments to metadata.yaml * remove external table reference * refactor to add date parameter * refactor based on Circle CI's advice * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * take out TODO comment --------- Co-authored-by: kik-kik <kignasiak@mozilla.com> Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-07-08 01:32:03 +03:00
- mhirose@mozilla.com
Firefox mobile installs from Adjust (#3735)
2023-05-04 07:07:16 +03:00
email_on_failure: true
email_on_retry: true
end_date: null
Ds 2944 create external adjust table redux (#3907) * add metadata, schema yamls and query.py * created adjust_derived namespace * add query.py, metadata, schema, dataset for testing * delete extraneous file, update DAG name * modify bqetl_adjust DAG redux * update DAG name, take out '_derived' * update table name in view * standardize table names across files * regenerate DAG * update schema in both locations * add query.py, metadata, schema yaml files * take put extraneous print statements, update datasets to be 'adjust' or 'adjust_derived' * add submission date to date_partition_parameter * update table name to be just one table * add DAG for adjust_derived * add bq_etl adjust_derived DAG to yaml file * add note about API token * revert changes to bqetl.adjust.py * use proper tast_id * fix start dates * add python command and docker image * add python command and docker image * delete extraneous code * comment out docker part in old adjust dag * add whitespace, delete extraneous code * Update sql/moz-fx-data-shared-prod/adjust/adjust_derived/view.sql Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * updated logic to check if response dictionary is not empty, moved view out of nested folder, added token ownership statement to metadata file, turned off email retry in dags.yaml, separated out clean up of json to its own function * take out extraneous if statement and move else statement * reorder where comment is to make more sense * more description as to why we're using mhirose's API token * take out periods * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * combine adjust DAGs * change logic for query_export check loop continuance, adapt metadata.yamls * add blank parameters test * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * add arguments to metadata.yaml * remove external table reference * refactor to add date parameter * refactor based on Circle CI's advice * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * take out TODO comment --------- Co-authored-by: kik-kik <kignasiak@mozilla.com> Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-07-08 01:32:03 +03:00
owner: mhirose@mozilla.com
Firefox mobile installs from Adjust (#3735)
2023-05-04 07:07:16 +03:00
retries: 2
retry_delay: 30m
Ds 2944 create external adjust table redux (#3907) * add metadata, schema yamls and query.py * created adjust_derived namespace * add query.py, metadata, schema, dataset for testing * delete extraneous file, update DAG name * modify bqetl_adjust DAG redux * update DAG name, take out '_derived' * update table name in view * standardize table names across files * regenerate DAG * update schema in both locations * add query.py, metadata, schema yaml files * take put extraneous print statements, update datasets to be 'adjust' or 'adjust_derived' * add submission date to date_partition_parameter * update table name to be just one table * add DAG for adjust_derived * add bq_etl adjust_derived DAG to yaml file * add note about API token * revert changes to bqetl.adjust.py * use proper tast_id * fix start dates * add python command and docker image * add python command and docker image * delete extraneous code * comment out docker part in old adjust dag * add whitespace, delete extraneous code * Update sql/moz-fx-data-shared-prod/adjust/adjust_derived/view.sql Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * updated logic to check if response dictionary is not empty, moved view out of nested folder, added token ownership statement to metadata file, turned off email retry in dags.yaml, separated out clean up of json to its own function * take out extraneous if statement and move else statement * reorder where comment is to make more sense * more description as to why we're using mhirose's API token * take out periods * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * combine adjust DAGs * change logic for query_export check loop continuance, adapt metadata.yamls * add blank parameters test * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * add arguments to metadata.yaml * remove external table reference * refactor to add date parameter * refactor based on Circle CI's advice * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * take out TODO comment --------- Co-authored-by: kik-kik <kignasiak@mozilla.com> Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-07-08 01:32:03 +03:00
start_date: '2023-07-06'
description: |
Derived tables built on Adjust data downloaded from https://api.adjust.com/kpis/v1/<app_token>
Using mhirose's API token - no Adjust API token for service accounts, just users.
Firefox mobile installs from Adjust (#3735)
2023-05-04 07:07:16 +03:00
repo: bigquery-etl
schedule_interval: 0 4 * * *
tags:
- impact/tier_2
- repo/bigquery-etl
DENG-775 downloads_with_attribution_v2 (#3716) * DENG-775 Added session_id to JOIN between GA data and stub_attr.stdout. Also expanded date range on GA session data to [download_date - 2 days, download_date + 1 day] * Updated query to handle missing GA download_session_id. It effectively applies V1 logic to the MISSING_GA_CLIENT dl_tokens.
2023-04-26 21:47:34 +03:00
bqetl_download_funnel_attribution:
description: Daily aggregations of data exported from Google Analytics joined with Firefox download data.
default_args:
depends_on_past: false
email:
- gleonard@mozilla.com
- telemetry-alerts@mozilla.com
end_date: null
owner: gleonard@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-04-10'
schedule_interval: 0 23 * * *
tags:
- impact/tier_1
DENG-871 - Add installs_by_country for Fenix (#3767) * Add installs_by_country for Fenix * Dont dryrun the query * Dont dryrun view * Add chanel to query
2023-05-09 05:24:10 +03:00
bqetl_fenix_external:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
- frank@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
owner: frank@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-05-07"
tags:
- impact/tier_1
- repo/bigquery-etl
feat(DENG-789): making apple ads data accessible (#3847) * added apple_ads_derived for copying over apple_ad data from the fivetran dataset, and apple_ads views now read from it * added bqetl_fivetran_apple_ads.py DAG responsible for copying apple_ads data from the fivetran project over to moz-fx-data-shared-prod * now dryrun skips apple_ads_derived instead of apple_ads as the query now accesses restricted dataset * added schema files for apple_ads_derived datasets * added descriptions to schema.yaml files for apple_ads_derived namespace * added dataset_metadata for apple_ads_derived to include a link to the dbt transformations * fixed apple_ads view definitions * removed application label and referenced_tables section inside metadata.yaml for apple_ads as requsted by srose in PR#3847 * corrected source project for apple_ads views * renamed apple_ads_derived to apple_ads_external * added * to apple_ads_external namespace name to skip in the dryrun due to integration test deployment * made tweaks to apple_ads and apple_ads_external datasets/namespaces as requested by whd * updated apple_ads_external skip rule to the way it is meant to be defined, this will work once a fix is rolled out for dryrun * fixed dag bqetl_fivetran_apple_ads description and updated the schedule to run once a day
2023-05-26 18:32:19 +03:00
bqetl_fivetran_apple_ads:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- frank@mozilla.com
- kik@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: kik@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-05-25'
description: |
Copies over apple_ads data coming from Fivetran
into our data BQ project. Fivetran syncs this data
every hour. We copy the data every 3 hours to our project.
schedule_interval: 0 3 * * *
tags:
- impact/tier_2
Copy App Store tables from Fivetran (#4030) * Copy App Store tables from Fivetran * Move dryrun to config * Generate DAG * Dont dryrun views * Add schemas
2023-07-07 21:24:03 +03:00
bqetl_fivetran_copied_tables:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- frank@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: frank@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-07-04'
description: |
Copy over Fivetran data to shared-prod.
schedule_interval: 0 3 * * *
tags:
- impact/tier_2
Shredder prototype (#3977) * Setup query and metadata templates to generate the queries for active_users_aggregates_deletion_request tables. * Separate the active_users KPIs from the calculation based on deletion requests. * Move mobile view outside of the browsers loop. * Update DAG, move partition_date to the end of the query. * Update DAG, move partition_date to the end of the query, change from parameters in the metadata to filter dates in the query. * Revert change to fenix_derived. * Remove 7-day window update and update tests. Fix indentation. * Set query parameters using jija template. * Set query parameters using jija template. * Using parameters in the query. * Formatting. * Update sql_generators/active_users_deletion_requests/templates/mobile_deletion_request_query.sql Co-authored-by: Daniel Thorn <dthorn@mozilla.com> * Formatting. * Search table's date filter. * Query only searches for clients with deletion request. Add required filter on submission_timestamp for table deletion_request. * Format --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2023-07-31 20:47:15 +03:00
bqetl_kpis_shredder:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- lvargas@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: lvargas@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-05-16'
updated bqetl_kpis_shredder DAG description to indicate why it is paused (#4280)
2023-09-12 16:15:23 +03:00
description: |
This DAG calculates KPIs for shredder client_ids
Shredder prototype (#3977) * Setup query and metadata templates to generate the queries for active_users_aggregates_deletion_request tables. * Separate the active_users KPIs from the calculation based on deletion requests. * Move mobile view outside of the browsers loop. * Update DAG, move partition_date to the end of the query. * Update DAG, move partition_date to the end of the query, change from parameters in the metadata to filter dates in the query. * Revert change to fenix_derived. * Remove 7-day window update and update tests. Fix indentation. * Set query parameters using jija template. * Set query parameters using jija template. * Using parameters in the query. * Formatting. * Update sql_generators/active_users_deletion_requests/templates/mobile_deletion_request_query.sql Co-authored-by: Daniel Thorn <dthorn@mozilla.com> * Formatting. * Search table's date filter. * Query only searches for clients with deletion request. Add required filter on submission_timestamp for table deletion_request. * Format --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2023-07-31 20:47:15 +03:00
repo: bigquery-etl
Bug 1852517 - Add execution delta to metadata & dag (#4239) * Add execution delta to metdata & dag * Spelling fix * Update execution hour --------- Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
2023-10-12 19:18:24 +03:00
schedule_interval: 0 2 */28 * *
Shredder prototype (#3977) * Setup query and metadata templates to generate the queries for active_users_aggregates_deletion_request tables. * Separate the active_users KPIs from the calculation based on deletion requests. * Move mobile view outside of the browsers loop. * Update DAG, move partition_date to the end of the query. * Update DAG, move partition_date to the end of the query, change from parameters in the metadata to filter dates in the query. * Revert change to fenix_derived. * Remove 7-day window update and update tests. Fix indentation. * Set query parameters using jija template. * Set query parameters using jija template. * Using parameters in the query. * Formatting. * Update sql_generators/active_users_deletion_requests/templates/mobile_deletion_request_query.sql Co-authored-by: Daniel Thorn <dthorn@mozilla.com> * Formatting. * Search table's date filter. * Query only searches for clients with deletion request. Add required filter on submission_timestamp for table deletion_request. * Format --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2023-07-31 20:47:15 +03:00
tags:
- impact/tier_3
- repo/bigquery-etl
Default DAG for bqetl queries (#4143) * DENG-1314 Implement changes to bqetl and create default DAG. * DENG-1314. Update Documentation. * DENG-1314. Dummy query to enable generating DAG and run tests. * DENG-1314. Update tests. * Update bigquery_etl/cli/query.py Raise exception when scheduling information is missing. Co-authored-by: Daniel Thorn <dthorn@mozilla.com> * DENG-1314. Update tests. * DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests. * Default task and tests update. * Default task and tests update. * 3650 - Remove default DAG option, update DAG template comment & tests. * 3650 - Condition for DAG warning. * 3650 - Update docs. * Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * Update docs/cookbooks/creating_a_derived_dataset.md Co-authored-by: Anna Scholtz <anna@scholtzan.net> --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: Daniel Thorn <dthorn@mozilla.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-08-29 15:32:52 +03:00
bqetl_default:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
end_date: null
owner: telemetry-alerts@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-09-01'
description: This is a default DAG to schedule tasks with lower business impact
DS-2947. Update DAG bqetl_defaul description to remove unavailable option. (#4220) Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-08-31 13:15:15 +03:00
or that don't require a new or existing DAG. Queries are automatically scheduled
in this DAG during creation when no dag name is specified using option --dag.
See [related documentation in the cookbooks](https://mozilla.github.io/bigquery-etl/cookbooks/creating_a_derived_dataset/)
Default DAG for bqetl queries (#4143) * DENG-1314 Implement changes to bqetl and create default DAG. * DENG-1314. Update Documentation. * DENG-1314. Dummy query to enable generating DAG and run tests. * DENG-1314. Update tests. * Update bigquery_etl/cli/query.py Raise exception when scheduling information is missing. Co-authored-by: Daniel Thorn <dthorn@mozilla.com> * DENG-1314. Update tests. * DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests. * Default task and tests update. * Default task and tests update. * 3650 - Remove default DAG option, update DAG template comment & tests. * 3650 - Condition for DAG warning. * 3650 - Update docs. * Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> * Update docs/cookbooks/creating_a_derived_dataset.md Co-authored-by: Anna Scholtz <anna@scholtzan.net> --------- Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: Daniel Thorn <dthorn@mozilla.com> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-08-29 15:32:52 +03:00
repo: bigquery-etl
schedule_interval: 0 4 * * *
tags:
- impact/tier_3
- triage/no_triage
RS-587 Add macroeconomic_indices table (#4323) * Add macroeconomic_indices table * Fix schema * Remove empty clustering entry
2023-09-20 18:55:09 +03:00
bqetl_reference:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- cmorales@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: cmorales@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-09-18'
description: DAG to build reference data
repo: bigquery-etl
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
Funnel generators POC (#4390) * Add funnel generation logic * Example funnel config * Fix funnel columns * funnel generation dimensions * Optimize segmenting generated funnels * Add funnel generation docs * Schedule generated funnels * Skip DAGs with no tasks * Add background info funnel generator * Add funnel generation tests * Fix join_previous_step_on * Add funnel example config
2023-10-13 00:05:08 +03:00
bqetl_generated_funnels:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- ascholtz@mozilla.com
email_on_failure: true
email_on_retry: true
owner: ascholtz@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-10-14'
description: DAG scheduling funnels defined in sql_generators/funnels
repo: bigquery-etl
schedule_interval: 0 5 * * *
tags:
- impact/tier_3
- triage/no_triage
bqetl_serp:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- akommasani@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: akommasani@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-10-01'
description: DAG to build serp events data
repo: bigquery-etl
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
DS3244 - Add derived datasets for review checker data (#4447) * Add review checker derived datasets * Add bqetl_review_checker dag Fix * Fix CI validate dag step * Incorporate feedback from Alex * Fix CI * change client last seen to clients first seen change client last seen to clients first seen * fix dag
2023-10-18 23:57:15 +03:00
bqetl_review_checker:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- akommasani@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: akommasani@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-10-01'
description: DAG to build review checker data
repo: bigquery-etl
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
[RS-826] New job to calculate newtab visits -> activity stream sessions (#4387) * New job to calculate newtab visits -> activity stream sessions * Removing newline chars at end of file * Removing newline chars at end of file * Removing newline chars at end of file * Addressing comment suggestions * Format * Add bqetl_ads DAG * Add ACL to nt_visits_to_sessions_conversion_factors_daily_v1 * Add metadata files * Add view to dry_run skip list * Oops, fix the view --------- Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2023-10-24 19:51:14 +03:00
bqetl_ads:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- cmorales@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: cmorales@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-10-10'
description: Tables related to ads
repo: bigquery-etl
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
Add ga_sessions_v1 table & view (#4554) * Add ga_sessions_v1 table & view This table aggregates session-level data from GA. * Rename nullify string func * Apply suggestions from code review Co-authored-by: Alexander <anicholson@mozilla.com> * Add upstream backfill deps * Move depends_on to correct section --------- Co-authored-by: Alexander <anicholson@mozilla.com>
2023-11-16 23:58:33 +03:00
bqetl_mozilla_org_derived:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
- frank@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
owner: frank@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-11-13"
tags:
- impact/tier_1
Separate Airflow tasks for glean_usage (#4588) * Add support for assigning Airflow tasks to task groups * Generate separate Airflow tasks for glean_usage * Remove Airflow dependencies from old glean_usage tasks
2023-11-30 20:48:17 +03:00
bqetl_glean_usage:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
- ascholtz@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
owner: ascholtz@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-11-20"
tags:
- impact/tier_1
Public GLAM datasets (#4606) * Public GLAM datasets * Remove Fenix GLAM datasets
2023-12-08 01:56:55 +03:00
bqetl_glam_export:
chore(glam): accommodate for dag reschedule (#5664)
2024-05-27 21:12:44 +03:00
schedule_interval: 0 22 * * *
Public GLAM datasets (#4606) * Public GLAM datasets * Remove Fenix GLAM datasets
2023-12-08 01:56:55 +03:00
default_args:
depends_on_past: false
email:
- ascholtz@mozilla.com
- efilho@mozilla.com
email_on_failure: true
email_on_retry: true
owner: ascholtz@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-11-28"
tags:
- impact/tier_2
description: DAG to prepare GLAM data for public export.
DENG-965 - symbolicate and signaturize crash pings (#4642)
2023-12-12 19:57:52 +03:00
bqetl_crash:
schedule_interval: 0 2 * * *
default_args:
depends_on_past: false
email:
- dthorn@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
owner: dthorn@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-12-10"
tags:
- impact/tier_2
Fb 1866834 new use counter data (#4713) * 1866834 - add new DAG bqetl_use_counter_analysis * 1866834 - adding fenix & firefox use_counters_v1 tables and telemetry use_counters view * 1866834 - fixing yaml file formatting * 1866834 - fix dags.yaml format - remove trailing space * 1866834 - add cast to numeric for rate to match schema for rate * 1866834 - Remove unnecessary alias * 1866834 - remove unnecessary alias * Update and rename view.sql to view.sql * 1866834 - test removing view * 1866834 - update table names * 1866834 - fixing fenix_and_firefox_use_counters view name and source table references
2023-12-15 22:13:36 +03:00
bqetl_use_counter_analysis:
schedule_interval: 0 8 * * *
default_args:
depends_on_past: false
email:
- kwindau@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
owner: kwindau@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2023-12-13"
tags:
- impact/tier_2
description: DAG to prepare use counter data for Firefox Desktop & Fenix for visualization
feat(): Added new data to mobile feature usage tables (logins data for iOS and Fenix) (#4648) * update * This is the modified codes following suggestions in https://github.com/mozilla/bigquery-etl/pull/4467. This pull request aims to update codes that create four mobile feature usage tables * updated codes following comments from kik in https://github.com/mozilla/bigquery-etl/commit/72d6d7191056f703bec99607802fe13622909569 * update feature usage table codes following suggestions * added new data to mobile feature usage table (logins data for iOS) and modified to add unnest for nested values * update codes based on suggestion from https://github.com/mozilla/bigquery-etl/pull/4648/files * update metadata.yaml and dags.yaml according to suggestions * update metadata.yaml and dags.yaml according to suggestions * remove clustering and references in all the metadata files, add LEFT JOIN to all the UNNEST, modify SQL files to folllow the incremental submission date format, and rename dau as events(metrics)_ping_client_count * updated schema to reflect the name change for dau * update according to comments on dec 15 * update the distinct client count name * fix the yaml errors * fix no new line error fordags.yaml --------- Co-authored-by: Ruoxi Zhao <rzhao@rzhao-37509.local> Co-authored-by: Ruoxi Zhao <rzhao@rzhao-37509.lan>
2023-12-16 02:08:58 +03:00
bqetl_mobile_feature_usage:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- rzhao@mozilla.com
email_on_failure: true
email_on_retry: false
end_date: null
owner: rzhao@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-10-24'
description: Schedule run for mobile feature usage tables
schedule_interval: 0 6 * * *
tags:
- impact/tier_3
DENG-1728 adding glean_app metrics to bigquery-etl (#4720) * adding telemetry_dev_cycle_derived to bigquery_etl * Update dags.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * changes after code review * move to _external dataset * rename table * fix defaults * schema from file --------- Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2023-12-20 22:14:11 +03:00
bqetl_telemetry_dev_cycle:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- leli@mozilla.com
email_on_failure: true
email_on_retry: false
end_date: null
owner: leli@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2023-12-19'
doc: DENG-3361 add documentation to telemetry dev cyle DAG (#5349)
2024-04-09 19:51:08 +03:00
description: |
DAG for Telemetry Dev Cycle Dashboard
Airflow Triage Note:
The tables are build every day so only the last run needs to be successful.
DENG-1728 adding glean_app metrics to bigquery-etl (#4720) * adding telemetry_dev_cycle_derived to bigquery_etl * Update dags.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * changes after code review * move to _external dataset * rename table * fix defaults * schema from file --------- Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2023-12-20 22:14:11 +03:00
repo: bigquery-etl
schedule_interval: 0 18 * * *
tags:
- impact/tier_3
- repo/bigquery-etl
DENG-2262 - Creating new desktop_installs_v1 table (#4754) * DENG-2262 - add new DAG bqetl_desktop_installs_v1 * Initial commit for DENG-2262 * DENG-2262 - reformatted query.sql * DENG-2262 add empty new line to end of view.sql
2023-12-28 22:15:04 +03:00
bqetl_desktop_installs_v1:
Fixing DAG start time (#4762)
2023-12-29 20:17:45 +03:00
schedule_interval: 55 23 * * *
DENG-2262 - Creating new desktop_installs_v1 table (#4754) * DENG-2262 - add new DAG bqetl_desktop_installs_v1 * Initial commit for DENG-2262 * DENG-2262 - reformatted query.sql * DENG-2262 add empty new line to end of view.sql
2023-12-28 22:15:04 +03:00
default_args:
depends_on_past: false
email:
- kwindau@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
owner: kwindau@mozilla.com
retries: 2
retry_delay: 30m
Fixing DAG start time (#4762)
2023-12-29 20:17:45 +03:00
start_date: "2023-12-28"
DENG-2262 - Creating new desktop_installs_v1 table (#4754) * DENG-2262 - add new DAG bqetl_desktop_installs_v1 * Initial commit for DENG-2262 * DENG-2262 - reformatted query.sql * DENG-2262 add empty new line to end of view.sql
2023-12-28 22:15:04 +03:00
tags:
- impact/tier_2
description: DAG to build mozdata-fx-data-shared-prod.firefox_desktop_derived.desktop_installs_v1 table
Add new table www_site_metrics_summary_v2 using new GA4 data (#4776) * Initial draft * adding new GA4 dag * fixing source for browser * work in progress * change group by to be explicit * adding non_fx_sessions * adding non-fx-sessions * adding campaign to query * adding new column to group by * added ad_content column * Adding downloads column * Adding non fx downloads * reformat the SQL * Add owner to new DAG * Updata data types in the schema file * Add missing comma * Update query.sql * Update start date and reduce from tier1 -> tier2 * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * update source and medium columns to use last touch attribution instead of first touch attribution * update group by --------- Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-05 00:01:47 +03:00
bqetl_google_analytics_derived_ga4:
schedule_interval: 0 12 * * *
Update bqetl_google_analytics_derived_ga4 yaml configs (#4780) * Update bqetl_google_analytics_derived_ga4 yaml configs * Switch to using countifs for more readability * reformat and switch to countifs for readability
2024-01-05 00:44:28 +03:00
description: Daily aggregations of data exported from Google Analytics 4
Add new table www_site_metrics_summary_v2 using new GA4 data (#4776) * Initial draft * adding new GA4 dag * fixing source for browser * work in progress * change group by to be explicit * adding non_fx_sessions * adding non-fx-sessions * adding campaign to query * adding new column to group by * added ad_content column * Adding downloads column * Adding non fx downloads * reformat the SQL * Add owner to new DAG * Updata data types in the schema file * Add missing comma * Update query.sql * Update start date and reduce from tier1 -> tier2 * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * update source and medium columns to use last touch attribution instead of first touch attribution * update group by --------- Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-05 00:01:47 +03:00
default_args:
Update bqetl_google_analytics_derived_ga4 yaml configs (#4780) * Update bqetl_google_analytics_derived_ga4 yaml configs * Switch to using countifs for more readability * reformat and switch to countifs for readability
2024-01-05 00:44:28 +03:00
depends_on_past: false
Add new table www_site_metrics_summary_v2 using new GA4 data (#4776) * Initial draft * adding new GA4 dag * fixing source for browser * work in progress * change group by to be explicit * adding non_fx_sessions * adding non-fx-sessions * adding campaign to query * adding new column to group by * added ad_content column * Adding downloads column * Adding non fx downloads * reformat the SQL * Add owner to new DAG * Updata data types in the schema file * Add missing comma * Update query.sql * Update start date and reduce from tier1 -> tier2 * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * update source and medium columns to use last touch attribution instead of first touch attribution * update group by --------- Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-05 00:01:47 +03:00
owner: kwindau@mozilla.com
email:
- kwindau@mozilla.com
- telemetry-alerts@mozilla.com
Update bqetl_google_analytics_derived_ga4 yaml configs (#4780) * Update bqetl_google_analytics_derived_ga4 yaml configs * Switch to using countifs for more readability * reformat and switch to countifs for readability
2024-01-05 00:44:28 +03:00
email_on_failure: true
email_on_retry: false
Add new table www_site_metrics_summary_v2 using new GA4 data (#4776) * Initial draft * adding new GA4 dag * fixing source for browser * work in progress * change group by to be explicit * adding non_fx_sessions * adding non-fx-sessions * adding campaign to query * adding new column to group by * added ad_content column * Adding downloads column * Adding non fx downloads * reformat the SQL * Add owner to new DAG * Updata data types in the schema file * Add missing comma * Update query.sql * Update start date and reduce from tier1 -> tier2 * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/metadata.yaml Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com> * update source and medium columns to use last touch attribution instead of first touch attribution * update group by --------- Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-05 00:01:47 +03:00
start_date: "2024-01-03"
retries: 2
retry_delay: 30m
tags:
- impact/tier_2
GLAM historical aggregates tables (#4795)
2024-01-11 00:29:35 +03:00
bqetl_glam_refresh_aggregates:
default_args:
depends_on_past: false
email:
- efilho@mozilla.com
email_on_failure: true
email_on_retry: false
owner: efilho@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2024-01-10'
description: Refresh GLAM tables that are serving data.
repo: bigquery-etl
schedule_interval: 0 8 * * *
tags:
- impact/tier_2
- repo/bigquery-etl
Add ETLs and views for Google Search Console data (DENG-1733) (#4892) * Add ETLs for historical Google Search Console data synced by Fivetran. * Fix formatting of `CASE` subclauses like `WHEN` inside Jinja blocks. * Add ETLs for current Google Search Console data exported directly to BigQuery. * Add views for Google Search Console data.
2024-02-07 23:53:32 +03:00
bqetl_google_search_console:
schedule_interval: 0 8 * * *
description: |
ETLs using data exported from Google Search Console.
The Google Search Console exports for a date typically complete by 08:00 UTC two days after that date,
so these ETLs should generally specify `date_partition_offset: -1` in their scheduling metadata.
default_args:
owner: srose@mozilla.com
email:
- srose@mozilla.com
- telemetry-alerts@mozilla.com
start_date: '2024-01-26'
retries: 2
retry_delay: 30m
tags:
- impact/tier_1
feat(DENG-2864): data-observability-dev define tables, schemas, and add ETL for updating them (#5222) * feat: add fenix_derived test tables to the data-observability-dev project * feat: add event_monitoring_live_v1 to data-observability-dev * feat: add new temp DAG for running copy to data-observability-dev tasks * fix: using fully qualified source table name in fenix_derived.events_daily_v1 inside data-observabiolity project * fix: correct event queries used to copy data into data-observability-dev project
2024-03-19 19:10:59 +03:00
bqetl_data_observability_test_data_copy:
schedule_interval: 0 8 * * *
description: |
ETL used to copy over data to data-observability-dev BQ project used
for testing out different data observability platforms.
default_args:
owner: kik@mozilla.com
email:
- kik@mozilla.com
- akommasani@mozilla.com
- ascholtz@mozilla.com
start_date: '2024-03-20'
retries: 2
retry_delay: 30m
tags:
- triage/no_triage
fix: add missing impact tag to the bqetl_data_observability_test_data_copy DAG (#5236)
2024-03-19 20:28:24 +03:00
- impact/tier_3
DENG-3161 create full suppression list Accoustic (#5246) Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-03-20 19:25:55 +03:00
bqetl_acoustic_suppression_list:
schedule_interval: 0 9 * * *
description: |
ETL for Acoustic suppression list.
default_args:
owner: leli@mozilla.com
email:
- leli@mozilla.com
fix: DENG-3287 change start date for DAGs acoustic (#5335)
2024-04-05 17:39:52 +03:00
start_date: '2024-04-03'
DENG-3161 create full suppression list Accoustic (#5246) Co-authored-by: Marlene Hirose <92952117+Marlene-M-Hirose@users.noreply.github.com>
2024-03-20 19:25:55 +03:00
retries: 1
retry_delay: 30m
tags:
- impact/tier_3
Create Braze derived models (#5362) * DENG-3008 removing misspelled file * DENG-3008 adding suppressions model * adding products model * updating dataset * adding newsletter model * adding waitlist model * removing create statement and adding modified timestamp * updating name and description for newsletters * updating description * adding users model * adding braze dag * changing subscribed to boolean * updating type to date * updating to full refresh * updating email id to external id for standardization * updating format * updating nullable fields * updating format * updating nullable fields * adding subscription map model * adding subscriptions model * adding user profiles model * updating format * formatting * removing last modified date in favor of only timestamp * updating table name
2024-04-16 05:27:31 +03:00
bqetl_braze:
Deng 3763 increase schedule frequency (#5590) * changing to 8 hours to align with schedules * updating schedule * updating schedule and aligning checks * updating error text
2024-05-29 23:25:50 +03:00
schedule_interval: 30 4,12,20 * * *
Create Braze derived models (#5362) * DENG-3008 removing misspelled file * DENG-3008 adding suppressions model * adding products model * updating dataset * adding newsletter model * adding waitlist model * removing create statement and adding modified timestamp * updating name and description for newsletters * updating description * adding users model * adding braze dag * changing subscribed to boolean * updating type to date * updating to full refresh * updating email id to external id for standardization * updating format * updating nullable fields * updating format * updating nullable fields * adding subscription map model * adding subscriptions model * adding user profiles model * updating format * formatting * removing last modified date in favor of only timestamp * updating table name
2024-04-16 05:27:31 +03:00
description: |
ETL for Braze workflows.
Add description to the bqetl_braze DAG (#5566)
2024-05-14 21:01:33 +03:00
## Triage notes:
Don't rerun this DAG!
Ping Chelsey Beck or Leli Schiestl.
If both are out, follow the [workbook](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/730234942/bqetl+braze+DAG+workbook) in confluence.
Create Braze derived models (#5362) * DENG-3008 removing misspelled file * DENG-3008 adding suppressions model * adding products model * updating dataset * adding newsletter model * adding waitlist model * removing create statement and adding modified timestamp * updating name and description for newsletters * updating description * adding users model * adding braze dag * changing subscribed to boolean * updating type to date * updating to full refresh * updating email id to external id for standardization * updating format * updating nullable fields * updating format * updating nullable fields * adding subscription map model * adding subscriptions model * adding user profiles model * updating format * formatting * removing last modified date in favor of only timestamp * updating table name
2024-04-16 05:27:31 +03:00
default_args:
owner: cbeck@mozilla.com
email:
- cbeck@mozilla.com
Add description to the bqetl_braze DAG (#5566)
2024-05-14 21:01:33 +03:00
- leli@mozilla.com
Create Braze derived models (#5362) * DENG-3008 removing misspelled file * DENG-3008 adding suppressions model * adding products model * updating dataset * adding newsletter model * adding waitlist model * removing create statement and adding modified timestamp * updating name and description for newsletters * updating description * adding users model * adding braze dag * changing subscribed to boolean * updating type to date * updating to full refresh * updating email id to external id for standardization * updating format * updating nullable fields * updating format * updating nullable fields * adding subscription map model * adding subscriptions model * adding user profiles model * updating format * formatting * removing last modified date in favor of only timestamp * updating table name
2024-04-16 05:27:31 +03:00
start_date: '2024-04-15'
retries: 1
retry_delay: 30m
tags:
- impact/tier_2
DENG-3412 braze sync hard bounce to bigquery with bqetl (#5375)
2024-04-16 20:54:33 +03:00
bqetl_braze_currents:
schedule_interval: 0 2 * * *
description: |
Load Braze current data from GCS into BigQuery
default_args:
owner: leli@mozilla.com
email:
- leli@mozilla.com
start_date: '2024-04-15'
retries: 1
retry_delay: 30m
tags:
- impact/tier_2
DENG-2621: Add pageload derived tables (#5359) * Added nightly, experiments and 1pct tables
2024-04-23 18:44:38 +03:00
Deng 3471 suppression list out of campaign monitor api (#5416) * DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> * after seans coder review --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-04-24 14:55:43 +03:00
bqetl_marketing_suppression_list:
DENG-3471 suppression list from campaign monitor (#5412) * DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-04-23 20:39:25 +03:00
schedule_interval: 0 3 * * *
description: |
Deng 3471 suppression list out of campaign monitor api (#5416) * DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> * after seans coder review --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-04-24 14:55:43 +03:00
Ingest marketing suppression lists into BigQuery
DENG-3471 suppression list from campaign monitor (#5412) * DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-04-23 20:39:25 +03:00
default_args:
owner: leli@mozilla.com
email:
- leli@mozilla.com
start_date: '2024-04-21'
retries: 1
retry_delay: 30m
tags:
- impact/tier_2
DENG-2621: Add pageload derived tables (#5359) * Added nightly, experiments and 1pct tables
2024-04-23 18:44:38 +03:00
bqetl_pageload_v1:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- wichan@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: wichan@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2024-04-01'
description: DAG to build pageload tables
repo: bigquery-etl
schedule_interval: daily
tags:
- impact/tier_1
- repo/bigquery-etl
DENG-3186 new desktop engagement model (#5415) * DENG-3186 new desktop engagement model * DENG-3186 add distribution ID to eng client V1 query * DENG-3186 - add distribution ID to the schema.yaml for eng client * DENG-3186 add distribution ID to the agg table * DENG-3186 adding locale to both client & agg tables * DENG-3186 add app_version to client & agg tables * DENG-3168 move lifecycle stage from engagement agg table to engagement agg view * DENG-3186 - add new desktop_engagement_client_v1 table to shredder * DENG-3186 update formatting of config.py * DENG-3186 switch to get country, os, os version, and build # from clients last seen and remove join to clients daily
2024-04-24 19:57:11 +03:00
bqetl_desktop_engagement_model:
schedule_interval: 0 12 * * *
description: Loads the desktop engagement model tables
default_args:
depends_on_past: false
owner: kwindau@mozilla.com
email:
- kwindau@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
start_date: "2024-04-24"
retries: 2
retry_delay: 30m
tags:
- impact/tier_2
DS-3501 - Create New iOS App Campaign Reporting Table (#5515) * DS-3501 initial commit * DS-3501 add new dag * DS-3501 update metadata.yaml * DS-3501 - update metadata.yaml * DS-3501 add backticks around column named date
2024-05-07 22:44:59 +03:00
Deng 3187 desktop retention model (#5496) * initial commit for desktop_retention_clients view * initial commit for desktop_retention_clients view - after formatting * initial add of query script for telemetry_derived.retention_v1 table * change view to table as I need to use the submission_date query parameter * move view to query for desktop_retention_clients * reformat file and add COALESCE on submission_date for new_profiles * take out app_name from retention_v1 query. Change isp_name to isp, take out mozfun UDF on normalized_os_version in CTE and update main query from retention_clients * run formatting on retention_v1/query.sql * move files from retention to desktop_retention, retention_clietns to desktop_retention_clients * add newline to end of schema.yaml file * reinstate retention_v1 deprecated folder/metadata.yaml file * refactor desktop_retention_clients_v1/desktop_retention metadata.yaml - add in clustering, take out extra space indent * refactor metadata.yaml, add schema.yaml, change query.sql to pull from telemetry_derived.desktop_retention_clients_v1 not view * add in metric_date for desktop_retention_v1 partition, take out require partition date filtering for both retention and retention_clients * add desktop_retention_model to dags.yaml * remove retention_clients_v1 * change column name is_new_profile to new_profile_metric_date * take out 'app_name' from group by * add telemetry_derived.desktop_retention_clients_v1 to shredder config
2024-05-14 23:50:15 +03:00
bqetl_desktop_retention_model:
schedule_interval: 0 12 * * *
description: Loads the desktop retention model tables
default_args:
depends_on_past: false
owner: mhirose@mozilla.com
email:
- mhirose@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
start_date: "2024-05-14"
retries: 2
retry_delay: 30m
tags:
- impact/tier_2
DS-3501 - Create New iOS App Campaign Reporting Table (#5515) * DS-3501 initial commit * DS-3501 add new dag * DS-3501 update metadata.yaml * DS-3501 - update metadata.yaml * DS-3501 add backticks around column named date
2024-05-07 22:44:59 +03:00
bqetl_ios_campaign_reporting:
schedule_interval: 0 12 * * *
description: Loads the apple ads ios_app_campaign_stats table
default_args:
depends_on_past: false
owner: kwindau@mozilla.com
email:
- kwindau@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
start_date: "2024-05-08"
retries: 2
retry_delay: 30m
tags:
- impact/tier_2
feat: add bqetl_mobile_kpi_metrics DAG and schedule mobile_kpi_metrics using it (#5715)
2024-06-04 15:47:23 +03:00
bqetl_mobile_kpi_metrics:
schedule_interval: 0 12 * * *
description: Generates support metrics for mobile KPI's
default_args:
depends_on_past: false
owner: kik@mozilla.com
email:
- kik@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: false
feat: update bqetl_mobile_kpi_metrics DAG start date (#5731)
2024-06-04 20:03:25 +03:00
start_date: '2024-06-03'
feat: add bqetl_mobile_kpi_metrics DAG and schedule mobile_kpi_metrics using it (#5715)
2024-06-04 15:47:23 +03:00
retries: 1
retry_delay: 30m
tags:
- impact/tier_1