6d002f4f71
Fixed back the variable name and set to notriage
2024-06-06 17:58:02 -04:00
47986405d6
Changed env variable xmatters_url to generate an error and test Jira ticket creation on failure.
2024-06-05 13:50:03 -04:00
1417232ed4
Fixed environment variable name and on_failure_callback
2024-06-04 19:12:27 -04:00
ad8f7ef714
Fixed format
2024-06-04 09:39:14 -04:00
76dbce1329
Fixed image string
2024-06-04 09:28:03 -04:00
378d160201
feat:[ASP-4631] Workday - XMatters integration ( #2006 )
...
* Added secrets to the DAG
* Format changes
* Fixed a ruff error
* Fixed format
---------
Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-03 08:08:08 -07:00
f9d466c985
feat(dags): Add Workday - XMatters integration DAG (ASP-4546) ( #2005 )
...
* First commit with the workday-xmatters DAG
* First commit with the workday-xmatters DAG
* Changed the jira_conn_id name
* Fixed ruff errors
* Fixed a ruff error
* Fixed black format
* Added apache-airflow-providers-atlassian-jira to requirements
* Fixed requirements.txt
* Fixed pytest error
* Fixed top-level code
* Changed the task definition back to what it was before
* Reformated file
* Added new line
---------
Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-05-31 11:49:28 -04:00
a726a86bba
fix(dags): Revert feat:[ASP 4546 JCM]-Workday - XMatters integration DAG ( #1996 ) ( #2004 )
2024-05-29 17:56:34 -04:00
9240183c16
feat:[ASP 4546 JCM]-Workday - XMatters integration DAG ( #1996 )
...
* First commit with the workday-xmatters DAG
* First commit with the workday-xmatters DAG
* Changed the jira_conn_id name
* Fixed ruff errors
* Fixed a ruff error
* Fixed black format
* Added apache-airflow-providers-atlassian-jira to requirements
* Fixed requirements.txt
* Fixed pytest error
* Update dags/eam_workday_xmatters_integration.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-05-29 08:52:30 -04:00
b20a871a5c
fix: Set bhr_collection spark memory to 20g ( #2003 )
2024-05-28 14:54:27 -04:00
3a72448d84
fix(glam): fix wait for copy dedup exec delta ( #2001 )
2024-05-28 14:41:16 -04:00
0907d75068
fix: Fix SQL path typos for data observability dev DAGs ( #2002 )
2024-05-28 11:22:04 -07:00
867d452cbc
feat:Add POC dags for triggering data observability checks ( #1999 )
...
* feat: Add POC dags for triggering data observability checks
* Bump keyring constraint
* Add observability connections
* Add requirements-override.txt
* Update Dockerfile
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
* Update requirements-override.txt
* Update .circleci/config.yml
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
* pip-install-local overrides
* Update dags/bigeye_poc.py
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
---------
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2024-05-28 10:32:20 -07:00
02e1f2371f
fix(glam): reschedule to give fenix and fog more time ( #2000 )
2024-05-27 14:10:39 -04:00
0b4755a383
fix(ci): Actually exit with error when `manual-publish` runs from main. ( #1998 )
2024-05-27 13:08:39 -04:00
0eff7b7ccc
chore(ci): Filter manual-publish workflows. ( #1997 )
2024-05-27 11:50:17 -04:00
b9a2fc850c
fix: Increase bhr_collection spark memory ( #1995 )
2024-05-24 15:54:32 -04:00
f17fa691c9
chore(ci): Prepare CI for CD proposal implementation. ( #1993 )
2024-05-24 14:14:22 -04:00
9097e5a10f
feat(plugins): Add Dockerflow-style version endpoint. ( #1994 )
2024-05-24 12:06:37 -04:00
0d3d2f28af
fix(bqetl_backfill): sandbox arguments are still in main...need to be removed ( #1991 )
2024-05-21 13:49:27 -05:00
4515cfa142
feat(bqetl_backfill): remove sandbox references from previous PR ( #1990 )
...
* feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill
* feat(bqetl_backfill): shelex quote was not needed
* fix(bqetl_backfill): remove sandbox references
2024-05-21 13:25:56 -05:00
485c8d864f
feat(bqetl_backfill): applying shelex quote removal for scheduling overrides handling that was missing from previous PR ( #1989 )
...
* feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill
* feat(bqetl_backfill): shelex quote was not needed
2024-05-21 11:59:55 -05:00
41c18a0eac
feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill ( #1987 )
2024-05-20 11:26:01 -05:00
50381fd25b
chore(glam): re-schedule glam legacy etl ( #1988 )
...
* chore(glam): re-schedule glam legacy etl
* Adjust execution_delta for upstream
* Remove unused tasks from glam_glean_imports
* Update dags/glam.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-05-17 17:03:45 -04:00
15fb7b1f4e
fix: [DENG-3335] Add shredder task for experiments ( #1985 )
2024-05-16 15:12:59 -04:00
c1b567de2c
chore: update probe-scraper emails ( #1986 )
2024-05-16 12:09:08 +02:00
28bfadea04
fix(glam): explicit params ( #1984 )
2024-05-13 15:59:29 -04:00
fe2f653f46
fix(glam): add missing fully qualified dataset param ( #1983 )
2024-05-10 23:24:24 -04:00
0ee18d3f64
feat: [DENG-3096] fix string formatting ( #1981 )
2024-05-08 11:55:58 -05:00
4a3a584197
feat: [DENG-3096] Fix string formatting errors ( #1980 )
2024-05-08 08:27:48 -05:00
aeaaa38047
fix(glam): move GLAM ETL from shared-prod to glam-prod ( #1978 )
...
* Move GLAM DAG to moz-fx-data-glam-prod-fca7
* fix(glam): move glam etl from shared-prod to glam-prod - lint
2024-05-06 12:09:46 -04:00
a6091acb62
feat(dags): Add DAP Collector dag for PPA Dev ( #1976 )
...
* feat(dags): Add DAP Collector dag for PPA Dev
* Update default args for ppa dev job
* fix ruff failure
---------
Co-authored-by: wil stuckey <wil.stuckey@gmail.com>
2024-05-03 14:58:20 -05:00
a0971a5ba9
feat: DENG-3096 Create 3 new cloudflare DAGs ( #1979 )
2024-05-03 13:32:06 -05:00
d8dac74167
feat(bqetl_backfill): adding the bqetl_backfill scheduling_parameters_override parameter ( #1974 )
...
* feat(bqetl_backfill): adding the bqetl_backfill scheduling_parameters_override parameter
* style: running make fixes
2024-04-26 12:12:13 -05:00
f80259c40d
fix: Bug 1892258 - Separate bhr_collection into two jobs ( #1975 )
2024-04-26 10:47:27 -04:00
a4183b99ea
fix: Increase bhr_collection spark memory allocation ( #1972 )
2024-04-22 10:41:23 -04:00
b8c132b9be
feat: [DENG-3277] Replace dataproc usage in public hardware report ( #1960 )
2024-04-16 17:29:17 -04:00
488f30e686
fix: DENG-3412 braze to bq remove the DAG ( #1969 )
2024-04-16 21:11:19 +02:00
cd874fe793
fix: DENG-3412 braze to bq add gcp_conn_id again ( #1968 )
2024-04-16 19:15:21 +02:00
509c40e394
fix: DENG-3412 braze to bq change project id ( #1967 )
2024-04-16 16:02:18 +02:00
34739f5785
fix: DENG-3412 braze to bq add gcp_conn_id ( #1966 )
...
* fix: DENG-3412 braze to bq add gcp_conn_id
* run black
2024-04-16 13:41:44 +02:00
55183da9e8
fix: DENG-3412 braze to bq add source format ( #1964 )
2024-04-15 21:26:48 +02:00
6007297985
feat: DENG-3412 braze sync hard bounce to bigquery ( #1963 )
...
* DENG-3412 braze sync hard bounce to bigquery
* run black
* move dag documentation out of docstring
* run black
* move import to top
2024-04-15 20:35:52 +02:00
3234be57c6
feat: Default reattach_on_restart parameter to True in bigquery_etl_query function ( #1961 )
...
We've been hit (again) by Airflow losing track of a running pod, restarting a task and failing with `More than one pod running with labels ...` error.
We previously set some of the tasks to reattach to running pods on restart ([1], [2], [3]) which solved this problem for them. We didn't set it by default for all tasks back then to avoid situations where not-autodeleted pods are stuck with the old code. Now that DEs have permissions required to delete pods it seems safe to revisit this.
[1] https://github.com/mozilla/telemetry-airflow/pull/1820
[2] https://github.com/mozilla/telemetry-airflow/pull/1834
[3] https://github.com/mozilla/telemetry-airflow/pull/1837
2024-04-08 18:40:45 +02:00
09acddb2b3
fix(managed-backfills): typo in Slack message, add owners ( #1959 )
2024-04-02 15:09:11 -04:00
6487786f9f
fix(dags): mad_server secrets are managed on GSM. ( #1958 )
...
* mad_server secrets are managed on GSM
* lint fixes
2024-03-26 16:58:20 -04:00
bd2bc2bd95
feat(managed-backfills): DENG-989 - Use overwatch Slack app for managed backfills ( #1957 )
2024-03-26 14:29:24 -04:00
22920ef969
feat(managed-backfills): DENG-983 Add DAGs for initiating and completing backfills ( #1956 )
...
* feat(managed-backfills): DENG-983 Add DAGs for initiating and completing backfills
2024-03-26 13:25:20 -04:00
4752c5d846
fix: Remove dev requirements from Docker image and improve `Makefile` ( #1953 )
...
* Define dev requirements in `requirements-dev.in`.
* Add `pip-tools` as a dev requirement.
* Remove `pip-compile` as a prerequisite for `pip-install-local`.
To avoid the bootstrapping issue in a fresh Python environment where `pip-compile` isn't installed yet.
* Update `.PHONY` config in `Makefile` to include all non-file targets.
* Update `help` target in `Makefile` to include all available commands.
* Replace tabs in `Makefile` strings with spaces.
* Update CI instructions for fixing lint errors.
2024-03-19 11:49:23 -07:00
ad49849fc7
feat: Create shredder backfill dag ( #1941 )
2024-03-19 11:38:52 -04:00
Julio Cezar Moscon
JCMOSCON1976
Mikaël Ducharme
Ben Wu
Eduardo Filho
Anna Scholtz
Braunk
whd
Katie Windau
Brendan Birdsong
Leli
akkomar
Alexander
Sean Rose