mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
bigquery-etl/dags.yaml

789 строки
20 KiB
YAML
Исходник Обычный вид История

Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
---
Add version to error_aggregates query
2020-06-03 20:49:11 +03:00
bqetl_error_aggregates:
schedule_interval: 3h
default_args:
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
owner: wkahngreene@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
email:
[
"telemetry-alerts@mozilla.com",
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
"wkahngreene@mozilla.com",
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
]
start_date: "2019-11-01"
Add version to error_aggregates query
2020-06-03 20:49:11 +03:00
retries: 1
retry_delay: 20m
depends_on_past: false
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
KPI dashboard generated Airflow DAG
2020-05-22 03:01:42 +03:00
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
bqetl_ssl_ratios:
More task re-scheduling
2020-07-10 20:33:34 +03:00
schedule_interval: 0 2 * * *
Add Markdown descriptions to DAGs
2021-01-30 02:04:42 +03:00
description: The DAG schedules SSL ratios queries.
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
default_args:
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
owner: chutten@mozilla.com
start_date: "2019-07-20"
email: ["telemetry-alerts@mozilla.com", "chutten@mozilla.com"]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
bqetl_amo_stats:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
# yamllint disable rule:line-length
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Add-on download and install statistics to power the
[addons.mozilla.org](https://addons.mozilla.org) (AMO) stats pages.
See the [post on the Add-Ons Blog](https://blog.mozilla.org/addons/2020/06/10/improvements-to-statistics-processing-on-amo/).
# yamllint enable rule:line-length
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
default_args:
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
owner: jklukas@mozilla.com
start_date: "2020-06-01"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add bqetl_amo_stats DAG
2020-06-03 16:44:02 +03:00
bqetl_core DAG
2020-06-05 21:36:53 +03:00
bqetl_core:
More task re-scheduling
2020-07-10 20:33:34 +03:00
schedule_interval: 0 2 * * *
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
description:
Tables derived from the legacy telemetry `core` ping sent by various
mobile applications.
bqetl_core DAG
2020-06-05 21:36:53 +03:00
default_args:
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
bqetl_core DAG
2020-06-05 21:36:53 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_core DAG
2020-06-05 21:36:53 +03:00
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
bqetl_nondesktop:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 3 * * *
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
default_args:
Update DAG owners
2022-04-21 00:50:59 +03:00
owner: "jklukas@mozilla.com"
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
updated owner of socorro_import DAG as per tags google sheet
2022-01-03 14:53:05 +03:00
email: [
"telemetry-alerts@mozilla.com",
]
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_nondesktop DAG
2020-06-05 22:41:31 +03:00
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
bqetl_mobile_search:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
email:
- "telemetry-alerts@mozilla.com"
- "anicholson@mozilla.com"
- "akomar@mozilla.com"
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
bqetl_mobile_search DAG
2020-06-08 23:04:50 +03:00
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
bqetl_fxa_events:
schedule_interval: 30 1 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Copies data from a Firefox Accounts (FxA) project. Those source tables
are populated via Cloud Logging (Stackdriver). We hash various fields
as part of the import.
The DAG also provides daily aggregations on top of the raw log data,
which eventually power high-level reporting about FxA usage.
Bug 1707571 - Ignore isverified in fxa_auth_events (#1975) * Bug 1707571 Ignore isverified in fxa_auth_events This is hopefully a temporary workaround while we understand the incompatible schema change in the source table. Very similar to the fix in https://github.com/mozilla/bigquery-etl/pull/1683 * Add docs about this failure mode.
2021-04-26 17:50:00 +03:00
Tasks here have occasionally failed due to incompatible schema changes
in the tables populated by Cloud Logging.
See https://github.com/mozilla/bigquery-etl/issues/1684 for an example
mitigation.
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
default_args:
Change ownership of FxA log ETL (#3051) As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1776156#c4
2022-06-23 21:36:20 +03:00
owner: dthorn@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-03-01"
Change ownership of FxA log ETL (#3051) As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1776156#c4
2022-06-23 21:36:20 +03:00
email: ["telemetry-alerts@mozilla.com", "dthorn@mozilla.com"]
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
retries: 1
retry_delay: 10m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add bqetl_fxa_events DAG (#1163) * Add bqetl_fxa_events DAG Replaces the fxa_events DAG in telemetry-airflow Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 14:58:11 +03:00
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
bqetl_subplat:
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
schedule_interval: 45 1 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
Daily imports for Subscription Platform data from Stripe and the Mozilla VPN
operational DB as well as derived tables based on that data.
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
Depends on `bqetl_fxa_events`, so is scheduled to run a bit after that.
Bug 1768419 - Add note to subplat dag docs for airflow triage (#2959) * Bug 1768419 - Add note to subplat dag docs for airflow triage * enriched the bqetl_subplat DAG description with a bit more detail regarding expected failures Co-authored-by: kik-kik <kignasiak01@gmail.com>
2022-06-01 16:54:19 +03:00
Stripe data retrieved by stripe_external__itemized_payout_reconciliation__v5
task has highly viariable availability timing, so it is possible for it to
fail with the following type of error:
`Error: Request req_OTssZ0Zv1cEmmm: Data for the report type
payout_reconciliation.itemized.5 is only available through
2022-05-08 12:00:00 UTC; you requested `interval_end`
= 2022-05-09 00:00:00 UTC.`
In such cases the failure is expected, the task will continue to retry every
30 minutes until the data becomes available. If failure observed looks
different then it should be reported using the Airflow triage process.
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
Combine multiple dags into bqetl_subplat (#2206)
2021-07-21 23:39:01 +03:00
start_date: "2021-07-20"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Delay VPN dag (#1734) We get alert emails every day about the wait_for_firefox_accounts_derived task, since this DAG starts at midnight and the upstream DAG doesn't start until 1:30. This reschedules the VPN DAG to occur 15 minutes after the upstream DAG to avoid these alerts.
2021-02-02 17:07:16 +03:00
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
bqetl_mozilla_vpn_site_metrics:
wait for GA data (#1973) Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2021-04-27 22:19:23 +03:00
schedule_interval: 0 15 * * *
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
description: |
Daily extracts from the Google Analytics tables for Mozilla VPN as well as
derived tables based on that data.
Depends on Google Analytics exports, which have highly variable timing, so
wait for GA data (#1973) Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2021-04-27 22:19:23 +03:00
queries depend on site_metrics_empty_check_v1, which retries every 30
minutes to wait for data to be available.
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
start_date: "2021-04-22"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add vpn tables to support data studio dashboard (#1960)
2021-04-22 23:48:59 +03:00
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
bqetl_gud:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
description: Optimized tables that power the
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
[Mozilla Growth and Usage Dashboard](https://gud.telemetry.mozilla.org).
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
default_args:
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
Move smoot queries to bqetl_gud DAG
2020-06-09 01:39:19 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
bqetl_messaging_system:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Daily aggregations on top of pings sent for the `messaging_system`
namespace by desktop Firefox.
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
default_args:
owner: najiang@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "najiang@mozilla.com"]
DAG for messaging system queries
2020-06-10 23:18:48 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
bqetl_activity_stream:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Daily aggregations on top of pings sent for the `activity_stream`
namespace by desktop Firefox. These are largely related to activity
on the newtab page and engagement with Pocket content.
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
default_args:
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2019-07-25"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
Create bqetl_activity_stream DAG
2020-06-12 02:19:20 +03:00
retries: 1
retry_delay: 5m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
bqetl_search:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
email:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
- "telemetry-alerts@mozilla.com"
- "anicholson@mozilla.com"
- "akomar@mozilla.com"
Add bqetl_search DAG
2020-06-12 01:54:40 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
bqetl_addons:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 4 * * *
description: |
Daily rollups of addon data from `main` pings.
Depends on `bqetl_search`, so is scheduled after that DAG.
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
default_args:
Updates to owners of queries and DAGs (#2289) * Updates to owners of queries and DAGs This is intended to generate some discussion about who new owners should be and to get that codified. * Make ascholtz owner of experiments daily active clients * Add klukas as addons owner + generate dags
2021-08-31 21:38:22 +03:00
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
Updates to owners of queries and DAGs (#2289) * Updates to owners of queries and DAGs This is intended to generate some discussion about who new owners should be and to get that codified. * Make ascholtz owner of experiments daily active clients * Add klukas as addons owner + generate dags
2021-08-31 21:38:22 +03:00
email: ["telemetry-alerts@mozilla.com"]
Move addons queries to bqetl_addons DAG
2020-06-13 00:14:57 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
DAGs for client queries
2020-06-23 19:22:41 +03:00
bqetl_devtools:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
Summarizes usage of the Dev Tools component of desktop Firefox.
DAGs for client queries
2020-06-23 19:22:41 +03:00
default_args:
owner: jklukas@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
email: ["telemetry-alerts@mozilla.com", "jklukas@mozilla.com"]
DAGs for client queries
2020-06-23 19:22:41 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
DAGs for client queries
2020-06-23 19:22:41 +03:00
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
bqetl_main_summary:
Rescheduling
2020-07-10 22:35:45 +03:00
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add descriptions for several DAGs (#1739) Follow-up to #1534
2021-02-02 22:53:13 +03:00
General-purpose derived tables for analyzing usage of desktop Firefox.
This is one of our highest-impact DAGs and should be handled carefully.
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
default_args:
owner: dthorn@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
email:
[
"telemetry-alerts@mozilla.com",
"dthorn@mozilla.com",
"jklukas@mozilla.com",
]
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Support multipart queries and main_summary
2020-06-24 23:40:24 +03:00
Refactoring
2020-06-25 22:44:37 +03:00
bqetl_experiments_daily:
Reschedule and set main_summary priority
2020-07-09 19:27:02 +03:00
schedule_interval: 0 3 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Add Markdown descriptions to DAGs
2021-01-30 02:04:42 +03:00
The DAG schedules queries that query experimentation related
metrics (enrollments, search, ...) from stable tables to finalize
numbers of experiment monitoring datasets for a specific date.
DAG for daily experiments clients
2020-06-25 00:42:55 +03:00
default_args:
Experiment enrollment monitoring queries (#1656) * Experiment enrollment aggregates hourly * Experiment enrollments recents query * Add execution_delay support for tasks * Experiment enrollment aggregates base query * Schedule experiment enrollment cumulative population estimate and active population * Experiment enrollment monitoring queries as views * Script for exporting experiment monitoring data to GCS * Export experiment monitoring data script aggregating data of longer running experiments * Parallelize experiment monitoring data export * init.sql for experiment enrollment monitoring queries * Use Airflow ds_format macro for hourly destination table * Use Airflow macros for experiments monitoring hourly execution delay * experiment_enrollment_cumulative_population_estimate as query instead of view * Fix referenced tables in enrollment_aggregates_hourly metadata and add comment * Simplify cumulative population estimate query
2021-01-14 00:53:32 +03:00
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2018-11-27"
Experiment enrollment monitoring queries (#1656) * Experiment enrollment aggregates hourly * Experiment enrollments recents query * Add execution_delay support for tasks * Experiment enrollment aggregates base query * Schedule experiment enrollment cumulative population estimate and active population * Experiment enrollment monitoring queries as views * Script for exporting experiment monitoring data to GCS * Export experiment monitoring data script aggregating data of longer running experiments * Parallelize experiment monitoring data export * init.sql for experiment enrollment monitoring queries * Use Airflow ds_format macro for hourly destination table * Use Airflow macros for experiments monitoring hourly execution delay * experiment_enrollment_cumulative_population_estimate as query instead of view * Fix referenced tables in enrollment_aggregates_hourly metadata and add comment * Simplify cumulative population estimate query
2021-01-14 00:53:32 +03:00
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
DAG for daily experiments clients
2020-06-25 00:42:55 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
# DAG for exporting query data marked as public to GCS
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
# queries should not be explicitly assigned to this DAG (done automatically)
Use consistent naming for DAG publishing JSON
2020-06-01 22:40:03 +03:00
bqetl_public_data_json:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 5 * * *
description: |
Daily exports of query data marked as public to GCS.
Depends on the results of several upstream DAGs, the latest of which
runs at 04:00 UTC.
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
default_args:
owner: ascholtz@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-04-14"
email: ["telemetry-alerts@mozilla.com", "ascholtz@mozilla.com"]
Handle public data export in DAGs
2020-05-30 01:42:22 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
bqetl_internet_outages:
updated schedule datetime to run later due to query running with missing data (#2757) Co-authored-by: kik <=>
2022-02-28 15:01:20 +03:00
schedule_interval: 0 7 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Update DAGs with Markdown description
2021-01-30 02:25:05 +03:00
DAG for building the internet outages datasets.
See [bug 1640204](https://bugzilla.mozilla.org/show_bug.cgi?id=1640204).
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
default_args:
owner: aplacitelli@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-01-01"
Updates to owners of queries and DAGs (#2289) * Updates to owners of queries and DAGs This is intended to generate some discussion about who new owners should be and to get that codified. * Make ascholtz owner of experiments daily active clients * Add klukas as addons owner + generate dags
2021-08-31 21:38:22 +03:00
email: ["aplacitelli@mozilla.com"]
Create an ETL job for the Internet Outages (#1058) * Add aggregation by country * Copy the initial Italy focus query This initial commit provides a baseline for the next commits to ease review, since this initial code was already reviewed. * Cleanup the country list and replace FULL OUTER with LEFT joins * Aggregate by city for cities with more than 15k inhabitants The actual 15k limit is enforced at ingestion time. This further limits the resulting cities to ones with at least 1000 active daily users. * Produce hourly aggregates * Move the query to the `internet_outage` dataset * Provide automatic daily scheduling through AirFlow * Tweak the SQL addressing review comments This additionally changes the `CAST` to `SAFE_CAST` to account for weirdnesses in the data. * Add ssl_error_prop * Add missing_dns_success * Add missing_dns_failure * Lower the minimum reported bucket size to 50 This allows us to match the EDA by Saptarshi and to have a better comparable baseline. * Document the oddities around `submission_timestamp_min`
2020-07-01 07:44:40 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Add query for monitoring deletion request volume (#1106)
2020-07-13 21:11:06 +03:00
bqetl_deletion_request_volume:
schedule_interval: 0 1 * * *
default_args:
owner: dthorn@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-06-29"
email: ["telemetry-alerts@mozilla.com", "dthorn@mozilla.com"]
Add query for monitoring deletion request volume (#1106)
2020-07-13 21:11:06 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Create DAG for events rollup (#1301) * Create DAG for events rollup * Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-11 01:50:45 +03:00
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
bqetl_fenix_event_rollup:
schedule_interval: 0 2 * * *
Create DAG for events rollup (#1301) * Create DAG for events rollup * Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml Co-authored-by: Anna Scholtz <anna@scholtzan.net> Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-11 01:50:45 +03:00
default_args:
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
owner: wlachance@mozilla.com
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-09-09"
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
email: ["wlachance@mozilla.com"]
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Bug 1635906 Add bqetl support for scripts and script for AET lookup (#1323) * Bug 1635906 Add bqetl support for scripts and script for AET lookup There are some code changes here for DAG generation and for testing. * Apply suggestions from code review Co-authored-by: Anna Scholtz <anna@scholtzan.net> * DAG fixups Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 22:28:25 +03:00
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
bqetl_org_mozilla_fenix_derived:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
schedule_interval: 0 2 * * *
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
default_args:
depends_on_past: false
email:
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
- amiyaguchi@mozilla.com
- telemetry-alerts@mozilla.com
Add daily schedule for geckoview versions (#1447) * Add daily schedule for geckoview versions * Remove unnecessary parameters
2020-10-17 00:37:30 +03:00
email_on_failure: true
email_on_retry: true
owner: amiyaguchi@mozilla.com
retries: 2
retry_delay: 30m
Ran YAMLlint on all yaml files and resolved linting issues (fixes #1297) (#1481) * "Ran YAMLlint on all yaml files" * "Moved product info metadata table to README file" * "Reformatted yaml lists" * "Updated line breaks so script runs" * "Updated line breaks so script runs" * "Undid line breaks" * "Created custom config file" * "Removed base document id" * "Undid line breaks" * "Reformatted code" * "Trimmed whitespace" * "Undid line break" * "Introduced newline" * "Trimmed whitespace" * "Added yamillint to config file" * "Added yamllint to config file" * "Moved up yamllint test" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Trimmed whitespace" * "Removing hyphen to fix CI error" * "Indentation to remove CI error" * "Included yamllint install in build run" * "Added yamllint in requirements.txt and .in file" * "Moved install yamllint step to its own stage" * "Updated yamllint test" * "Updated circleci step" * "Reformatted code" * "Added yamllint to circleci steps" * "Added checkout block to yamllint step" * "Trimmed whitespace" * "Undid yamllint step" * "Specified directory name for yamllint test" * "Fixed yamlint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Fixed yamllint errors" * "Ignore pathway in linting" * "Added ignore venv pathway during linting" * "Updated ignore block" * "Updated ignore block" * "Removed ignore block" * "Updated ignore block" * "Indented base as a list" * "Indented base item" * Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Resolved linting errors" * "Referenced tables put back on same line" * "Fixed linting error" * Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com> * "Fixed linting error" Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-30 03:24:55 +03:00
start_date: "2020-10-18"
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
bqetl_google_analytics_derived:
Change GA dag to run later (#1515)
2020-11-04 22:47:32 +03:00
schedule_interval: 0 23 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Daily aggregations of data exported from Google Analytics.
The GA export runs at 15:00 UTC, so there's an effective 2-day delay
for user activity to appear in these tables.
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
default_args:
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
owner: ascholtz@mozilla.com
email:
- ascholtz@mozilla.com
- telemetry-alerts@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-10-31"
Add derived tables for blog.m.o google analytics (#1492)
2020-10-29 00:40:45 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
bqetl_monitoring:
schedule_interval: 0 2 * * *
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Update DAGs with Markdown description
2021-01-30 02:25:05 +03:00
This DAG schedules queries and scripts for populating datasets
used for monitoring of the data platform.
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
default_args:
owner: ascholtz@mozilla.com
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
email: ["ascholtz@mozilla.com"]
start_date: "2018-10-30"
Add event_types query generation
2020-11-12 17:10:07 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add event_types query generation
2020-11-12 17:10:07 +03:00
bqetl_event_rollup:
schedule_interval: 0 3 * * *
default_args:
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
owner: wlachance@mozilla.com
Add event_types query generation
2020-11-12 17:10:07 +03:00
start_date: "2020-11-03"
Bug 1730190 - filter out overactive clients in events_daily queries (#2333) * Bug 1730190 - filter out overactive clients in events_daily queries * Update events_daily DAGs owners Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-09-16 18:04:26 +03:00
email: ["wlachance@mozilla.com"]
Schedule monitoring tasks in bqetl_monitoring
2020-11-10 23:44:03 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add queries and views for fetch marketing spend data (#1575)
2020-11-30 23:34:55 +03:00
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
bqetl_iprospect:
schedule_interval: 0 4 * * *
description: |
This DAG imports iProspect data from moz-fx-data-marketing-prod-iprospect.
depends_on_past: false
default_args:
owner: ascholtz@mozilla.com
Update notifications for bqetl_iprospect
2021-11-03 22:32:58 +03:00
email:
[
"ascholtz@mozilla.com",
"echo@mozilla.com",
"shong@mozilla.com"
]
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
start_date: "2021-04-19"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add script to import iProspect data
2021-07-06 23:31:29 +03:00
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
bqetl_search_dashboard:
default_args:
depends_on_past: false
email:
Enforce more yaml lint rules (#1878)
2021-03-10 01:25:01 +03:00
- telemetry-alerts@mozilla.com
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
- akomar@mozilla.com
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
email_on_failure: true
email_on_retry: true
Update query and dag owners (#2292)
2021-09-01 22:51:29 +03:00
owner: akomar@mozilla.com
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
retries: 2
retry_delay: 30m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-12-14"
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Bug 1673979 - Add Search dashboard queries (#1619) * Add search dashboard desktop search aggregates by userstate table and schedule daily * Add desktop_search_aggregates_for_searchreport * Add mobile_search_aggregates_for_searchreport Co-authored-by: Sunah Suh <ssuh@mozilla.com>
2020-12-15 02:27:14 +03:00
Adding accessibility clients query for platform accessibility clients (#1612) Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-12-11 23:05:50 +03:00
bqetl_desktop_platform:
schedule_interval: 0 3 * * *
default_args:
owner: jklukas@mozilla.com
Updated accessibility clients query and start date. (#1623)
2020-12-15 20:41:30 +03:00
start_date: "2018-11-01"
Adding accessibility clients query for platform accessibility clients (#1612) Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-12-11 23:05:50 +03:00
email:
[
"telemetry-alerts@mozilla.com",
"jklukas@mozilla.com",
"yzenevich@mozilla.com",
]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
bqetl_internal_tooling:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
This DAG schedules queries for populating queries related to Mozilla's
internal developer tooling (e.g. mozregression).
default_args:
depends_on_past: false
email:
Enforce more yaml lint rules (#1878)
2021-03-10 01:25:01 +03:00
- wlachance@mozilla.com
- telemetry-alerts@mozilla.com
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
email_on_failure: true
email_on_retry: true
end_date: null
owner: wlachance@mozilla.com
retries: 2
retry_delay: 30m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2020-06-01"
Initial version of a mozregression aggregates dataset (#1760)
2021-02-11 17:43:34 +03:00
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_3
Bug 1647328 - Schedule performance release criteria queries (#1596)
2021-03-10 22:27:32 +03:00
bqetl_release_criteria:
schedule_interval: daily
default_args:
owner: perf-pmo@mozilla.com
start_date: "2020-12-03"
email:
- telemetry-alerts@mozilla.com
- dthorn@mozilla.com
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
bqetl_pocket:
default_args:
depends_on_past: false
email:
- jklukas@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
owner: jklukas@mozilla.com
retries: 2
retry_delay: 30m
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
start_date: "2021-03-10"
Bug 1695336 Pocket MAU import (#1883) * Bug 1695336 Pocket MAU import See https://bugzilla.mozilla.org/show_bug.cgi?id=1695336 * Remove null in dags.yaml * Add 1 day offset
2021-03-12 00:55:21 +03:00
description: |
Import of data from Pocket's Snowflake warehouse.
Originally created for [Bug 1695336](
https://bugzilla.mozilla.org/show_bug.cgi?id=1695336).
schedule_interval: 0 12 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Create adMarketplace Daily Engagements table (#1891)
2021-03-17 21:03:54 +03:00
Schedule queries
2021-03-16 18:49:17 +03:00
bqetl_desktop_funnel:
DAG scheduling and docs cleanup (#1919) * DAG scheduling and docs cleanup This adjusts some DAG schedules in an attempt to minimize the number of BQSensor rescheduling emails we receive under normal circumstances. Of note, the `copy_deduplicate` DAG is often taking a little longer than an hour to complete, meaning that DAGs starting at 02:00 are likely to hit reschedules. We do not address that problem here in hopes that performance improvements to copy_deduplicate can bring performance back under 1 hour. We also make some documentation fixups, including consistently using `|` rather than `>` or `>-` as our multi-line string indicator. This preserves whitespace that is relevant for markdown processing. * Update stripe schedule
2021-03-29 16:15:24 +03:00
description: |
Schedule queries
2021-03-16 18:49:17 +03:00
This DAG schedules desktop funnel queries used to power the
[Numbers that Matter dashboard](https://protosaur.dev/numbers-that-matter/)
Add view for desktop funnel activation
2021-03-19 22:20:05 +03:00
schedule_interval: 0 4 * * *
Schedule queries
2021-03-16 18:49:17 +03:00
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-01-01"
email:
[
"telemetry-alerts@mozilla.com",
"ascholtz@mozilla.com",
]
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add DAG for iOS Firefox ETL (#1905) * Add DAG for iOS Firefox ETL * Format dags.yaml file * Rename dag to betl_firefox_ios * Update schedule_interval to align with other queries
2021-03-30 02:23:50 +03:00
bqetl_firefox_ios:
default_args:
depends_on_past: false
email:
- amiyaguchi@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: amiyaguchi@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2021-03-18"
description: Schedule daily ios firefox ETL
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Schedule release data import
2021-04-14 22:54:28 +03:00
bqetl_releases:
default_args:
depends_on_past: false
email:
- ascholtz@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: ascholtz@mozilla.com
retries: 2
retry_delay: 30m
start_date: "2021-04-14"
description: |
Schedule release data import from https://product-details.mozilla.org/1.0
Add metadata for public release dataset views
2021-04-15 19:28:14 +03:00
For more context, see
https://wiki.mozilla.org/Release_Management/Product_details
Schedule release data import
2021-04-14 22:54:28 +03:00
schedule_interval: 0 4 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
bqetl_ctxsvc_derived:
default_args:
depends_on_past: false
email:
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
- ctroy@mozilla.com
- wstuckey@mozilla.com
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
owner: ctroy@mozilla.com
Bug 1708486 - Create aggregate table for contextual services (#2001)
2021-05-05 21:32:20 +03:00
retries: 2
retry_delay: 30m
start_date: '2021-05-01'
description: Contextual services derived tables
schedule_interval: 0 3 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Add command to import subscriber report from apple (#2116)
2021-06-15 01:16:38 +03:00
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
bqetl_search_terms_daily:
default_args:
depends_on_past: false
email:
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
- ctroy@mozilla.com
- wstuckey@mozilla.com
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
- rburwei@mozilla.com
- telemetry-alerts@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
owner: ctroy@mozilla.com
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
retries: 2
retry_delay: 30m
start_date: '2021-09-20'
description: |
Derived tables on top of search terms data.
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
Note that the tasks for populating `suggest_impression_sanitized_v*` are
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
particularly important because the source unsanitized dataset has only
a 2-day retention period, so errors fairly quickly become unrecoverable
and can impact reporting to partners. If this task errors out, it could
indicate trouble with an upstream task that runs in a restricted project
Set owners for Contextual Services tasks (#3048) * Set owners for Contextual Services tasks I made some guesses here about who would make the most sense to own individual pieces; treat this as a starting point and we can discuss in PR comments to get this to final state. * Generate DAGs after rebase * Respond to review comments Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> Co-authored-by: whd <whd@users.noreply.github.com>
2022-07-21 23:18:23 +03:00
outside of Airflow. Contact `ctroy`, `wstuckey`, `whd`, and `jbuck`.
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
schedule_interval: 0 3 * * *
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
Update impact tag and docs for bqetl_search_terms_daily (#2593) * Update impact tag and docs for bqetl_search_terms_daily * Line length
2022-01-10 17:24:49 +03:00
- impact/tier_1
new derived table: search_queries_daily (#2359) * new derived table: search_queries_daily * Restructuring * Add rburwei to owners lists * Update table and field names * Remove old init.sql file * Format sql Co-authored-by: Jeff Klukas <jeff@klukas.net> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-09-22 16:01:24 +03:00
Schedule experimenter_experiments_import
2021-07-22 01:40:18 +03:00
bqetl_experimenter_experiments_import:
schedule_interval: "*/10 * * * *"
description: |
Imports experiments from the Experimenter V4 and V6 API.
Imported experiment data is used for experiment monitoring in
[Grafana](https://grafana.telemetry.mozilla.org/d/XspgvdxZz/experiment-enrollment).
default_args:
owner: ascholtz@mozilla.com
start_date: "2020-10-09"
retries: 0
email:
- ascholtz@mozilla.com
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Move feature_usage_v2 to separate DAG
2021-08-03 19:18:50 +03:00
bqetl_feature_usage:
schedule_interval: 0 5 * * *
description: |
Daily aggregation of browser features usages from `main` pings,
`event` pings and addon data.
Depends on `bqetl_addons` and `bqetl_main_summary`, so is scheduled after.
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-01-01"
email:
- "telemetry-alerts@mozilla.com"
- "ascholtz@mozilla.com"
- "loines@mozilla.com"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
bqetl_urlbar:
schedule_interval: 0 3 * * *
description: |
Daily aggregation of metrics related to urlbar usage.
default_args:
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
owner: anicholson@mozilla.com
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
start_date: "2021-08-01"
email:
- "telemetry-alerts@mozilla.com"
Update search DAGs ownership (#2571)
2021-12-13 19:12:53 +03:00
- "anicholson@mozilla.com"
Create urlbar_clients_daily table (#1537)
2021-09-02 01:43:51 +03:00
- "akomar@mozilla.com"
- "tbrooks@mozilla.com"
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
updated tiers to reflect the airflow tag google sheet
2021-12-15 16:21:56 +03:00
- impact/tier_2
Schedule query for unified metrics
2021-10-13 01:59:24 +03:00
bqetl_unified:
schedule_interval: 0 3 * * *
description: |
Schedule queries that unify metrics across all products.
default_args:
owner: ascholtz@mozilla.com
start_date: "2021-10-12"
email:
- "telemetry-alerts@mozilla.com"
- "ascholtz@mozilla.com"
- "loines@mozilla.com"
Create derived dataset and DAG for the aggregation of active users. (#2952) * Create derived dataset and DAG for the aggregation of active users. * Change APROX_DISTINCT_COUNT for COUNT(DISTINCT) * Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity. * Improve descriptions * Add clustering by channel * DAG update * Clustering based on users most common filtering * Add notification for analytics DAG * agg_active_users to query from telemetry_derived. * Add query to create agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Add date to init.sql * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/mozfun/bytes/zero_right/metadata.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Remove CAST from init.sql for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql * Update project in ini.sql to `moz-fx-data-shared-prod`. * Update name of aggregation and DAGS for consistency * Add active_users_aggregates_v1 to align with current naming convention * Update to active_users_aggregates * Update query for active_users_aggregates * Add uri_count and active_hours query for active_users_aggregates * Format query, update DAG to remove agg_active_users_v1 * Generate DAG to correct CI error. Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 17:23:50 +03:00
- "lvargas@mozilla.com"
Schedule query for unified metrics
2021-10-13 01:59:24 +03:00
retries: 2
retry_delay: 30m
tags added to dag generation, for now impact defaulted to tier1
2021-12-08 19:12:11 +03:00
tags:
making sure that the tag follows the agreed name convention
2021-12-14 14:46:50 +03:00
- impact/tier_1
Add regrets reporter summary table (fixes #2572) (#2574) * added regrets_reporter_summary query * regrets_reporter_summary init added * submission date modified to be dynamic @submission_date * added labels * Apply suggestions from code review by @wlach Fixing SQL logic Co-authored-by: Will Lachance <wlachance@mozilla.com> * Apply suggestions from code review by @jklukas Co-authored-by: Jeff Klukas <jklukas@mozilla.com> * added metadata.yaml for regrets_reporter_summary view * added regrets_reporter_summary to dry_run exclusion * regrets_reporter_summary dag updated Co-authored-by: = <=> Co-authored-by: Will Lachance <wlachance@mozilla.com> Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-12-16 21:14:51 +03:00
bqetl_regrets_reporter_summary:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- kignasiak@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: kignasiak@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2021-12-12'
description: Measure usage of the regrets reporter addon
schedule_interval: 0 4 * * *
tags:
- impact/tier_1
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
bqetl_cjms_nonprod:
DVPN-89 - Catch all flows at end of day for cjms nonprod (#2873)
2022-04-07 00:31:21 +03:00
schedule_interval: 0 * * * *
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
description: |
Hourly ETL for cjms nonprod.
default_args:
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
owner: srose@mozilla.com
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
start_date: "2022-03-24"
Make @sean-rose owner for subplat ETL (#3068)
2022-06-30 19:07:33 +03:00
email: ["telemetry-alerts@mozilla.com", "srose@mozilla.com"]
DVPN-119 - Add ETL for Non-Prod CJMS (#2823)
2022-03-26 00:56:11 +03:00
retries: 2
retry_delay: 5m
tags:
- impact/tier_3
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
bqetl_acoustic_contact_export:
pushing bqetl acoustic dags by 1 hour to see if that resolves sensor issues (#3002)
2022-06-02 16:18:41 +03:00
schedule_interval: 0 9 * * *
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
description: |
Processing data loaded by
fivetran_acoustic_contact_export
DAG to clean up the data loaded from Acoustic.
default_args:
owner: kignasiak@mozilla.com
start_date: "2021-03-01"
email: ["telemetry-alerts@mozilla.com", "kignasiak@mozilla.com"]
retries: 2
retry_delay: 5m
tags:
- impact/tier_3
bqetl_acoustic_raw_recipient_export:
pushing bqetl acoustic dags by 1 hour to see if that resolves sensor issues (#3002)
2022-06-02 16:18:41 +03:00
schedule_interval: 0 9 * * *
BQETL processes related to Acoustic Fivetran load, new UDF also added (#2900)
2022-05-04 12:40:48 +03:00
description: |
Processing data loaded by
fivetran_acoustic_raw_recipient_export
DAG to clean up the data loaded from Acoustic.
default_args:
owner: kignasiak@mozilla.com
start_date: "2022-03-01"
email: ["telemetry-alerts@mozilla.com", "kignasiak@mozilla.com"]
retries: 2
retry_delay: 5m
tags:
- impact/tier_3
Create derived dataset and DAG for the aggregation of active users. (#2952) * Create derived dataset and DAG for the aggregation of active users. * Change APROX_DISTINCT_COUNT for COUNT(DISTINCT) * Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity. * Improve descriptions * Add clustering by channel * DAG update * Clustering based on users most common filtering * Add notification for analytics DAG * agg_active_users to query from telemetry_derived. * Add query to create agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Add date to init.sql * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Update sql/mozfun/bytes/zero_right/metadata.yaml Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com> * Modify metadata.yaml for agg_active_users * Remove CAST from init.sql for agg_active_users * Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql * Update project in ini.sql to `moz-fx-data-shared-prod`. * Update name of aggregation and DAGS for consistency * Add active_users_aggregates_v1 to align with current naming convention * Update to active_users_aggregates * Update query for active_users_aggregates * Add uri_count and active_hours query for active_users_aggregates * Format query, update DAG to remove agg_active_users_v1 * Generate DAG to correct CI error. Co-authored-by: Lucia Vargas <lvargas@mozilla.com> Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 17:23:50 +03:00
bqetl_analytics_aggregations:
default_args:
depends_on_past: false
email:
- "telemetry-alerts@mozilla.com"
- "lvargas@mozilla.com"
- "gkaberere@mozilla.com"
email_on_failure: true
email_on_retry: true
end_date: null
owner: lvargas@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-05-12'
description: Scheduler to populate the aggregations required for
analytics engineering and reports optimization.
It provides data to build growth, search and usage metrics, as well
as acquisition and retention KPIs, in a model that facilitates
reporting in Looker.
schedule_interval: 0 1 * * *
tags:
- impact/tier_1
Decision support metrics for fog initial commit (#2984) * decision support metrics for fog initial commit * reverting overwrite * init.sql to initialize table * changing to main_1pct for the time being * init sql correction * missing as keyword * updating dag order * trying main 1pct * removed init.sql, added schema.yaml * update dag to reflect different table wait * re-generated DAG * fix tag * fix tags in dag * no clustering
2022-06-02 00:00:29 +03:00
bqetl_fog_decision_support:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- pmcmanis@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: pmcmanis@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-05-25'
description: This DAG schedules queries for calculating FOG decision
support metrics
schedule_interval: 0 4 * * *
tags:
- impact/tier_3
- repo/bigquery-etl
RS-306 - ETL for newtab interactions (#3073)
2022-07-18 23:32:03 +03:00
bqetl_newtab:
default_args:
depends_on_past: false
email:
- telemetry-alerts@mozilla.com
- anicholson@mozilla.com
email_on_failure: true
email_on_retry: true
end_date: null
owner: anicholson@mozilla.com
retries: 2
retry_delay: 30m
start_date: '2022-07-01'
description: Schedules newtab related queries.
schedule_interval: daily
tags:
- impact/tier_1