Граф коммитов

2250 Коммитов

Автор SHA1 Сообщение Дата
Eduardo Filho b666c25bce
fix(glam_imports): Wait on TaskGroup instead of task (#2043) 2024-07-09 15:49:30 -04:00
Eduardo Filho bd9ccb0cc4
fix(glam): build extract tables as TaskGroup (#2042)
* GLAM build extract tables as TaskGroup

* fix(glam): build extract tables as TaskGroup
2024-07-08 18:46:28 -04:00
Jared Snyder fea332afb2
fix: for search_forecasting remove execution delta as upstream job already applies 7 day lag (#2040)
* remove execution delta as upstream job already applies 7 day lag

* fix: remove execution delay from search_forecast dag
2024-07-08 15:48:14 -04:00
Daniel Mueller e90004c558
feat(dags): Add DAP collector for PPA Prod (#2034)
* creat prod dap collector job

* feat(dags): Add DAP collector for PPA Prod

* adjust how secrets are handled
2024-07-01 11:24:40 -05:00
Jared Snyder 8ba2f594af
feat: add search_forecasting dag (#2036)
* add search_forecasting dag

* feat: add new search_forecasting DAG and update docs

* docs: update README

* Update README.md

Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com>

* specified forked PR is just for people without write access

* docs: added comment to clarify usage of kpi_forecasting script

* fixes requested by kik

* Update README.md

Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com>

* updated start_date

* added wait task sensor

* fix redundant task

* moved wait task inside context manager

* fix schedule_interval to match upstream job

---------

Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com>
2024-07-01 16:09:05 +02:00
JCMOSCON1976 04566b23cc
fix(DAG): Workday-Everfi [ASP-4747] - New DAG (#2038)
* New DAG eam-workday-everfi-integration

* Make fixes applied

* feat: Added dag schedule time

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-28 13:57:27 -07:00
Anna Scholtz 5e66d8a2ad
feat: Separate DAG for generating Looker artifacts (#2031) 2024-06-27 11:29:53 -07:00
Ben Wu 5d1882653a
fix(bhr): Update bhr_collection_child worker count and sampling (#2035) 2024-06-26 12:23:12 -04:00
Eduardo Filho 7a70dccfb8
chore(glam): don't export to GCS since aggregates are read from BQ (#2019)
* chore(glam): remove  useless steps

* remove extra blank lines

* Reformat
2024-06-25 15:36:23 -04:00
Marlene Hirose a9c598b3a0
single quotes not needed around date. This caused errors in running the Airflow DAG (#2033) 2024-06-25 09:42:45 -07:00
Marlene Hirose ee6234d7ec
fix: microsoft store DAG update when DAG runs to day before actual date (#2032)
* update when DAG runs to day before actual date

* Update dags/microsoft_store.py

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

---------

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-06-24 10:40:57 -07:00
Daniel Thorn 1edc654ed3
fix: bug 1579266 fix fetch_schema in socorro_import_crash_data.py (#2030)
* fix: bug 1579266 fix fetch_schema in socorro_import_crash_data.py

it should not read from s3 anymore

also fix the code for reading from github

* black format and fix ruff
2024-06-21 10:19:16 -04:00
Daniel Thorn 5432d2054c
fix: bug-1579266 update socorro import to pull from gcs (#2022) 2024-06-20 18:37:15 -04:00
Lucia 723be3a7af
fix: Update table names in DAG microsoft_store (#2029)
* Update table names

* Updated with the suggestion to make it easier to maintain.

* fix(dag): Remove space.

* Adding _v1 to the table name in the query

* f string
2024-06-19 12:25:03 -04:00
Marlene Hirose 8a4177a423
take out 'microsoft' from the microsoft_store DAG (#2027) 2024-06-18 20:04:24 -07:00
Marlene Hirose b942a1170a
feat(deng 2361 microsoft) take out arguments from DAG (#2025)
* take out arguments from DAG

* take out unnecessary airlfow_gke_prod_kwargs

* add in the variable devfinitions that are needed
2024-06-18 13:53:07 -07:00
Marlene Hirose 0e170cfd9d
reconstruct command strings (#2024) 2024-06-18 12:03:53 -07:00
Marlene Hirose 5217731b21
add spaces in templating sections (#2023) 2024-06-18 11:23:11 -07:00
Marlene Hirose 2a17c546d3
feat(microsoft_store): initial add microsoft_store DAG (#2021)
* initial add microsoft_store DAG

* take out read_gke functions, add secrets = [<secrets>] to each task

* reformat file

* get rid of unused imports, get rid of useless expresions

* get rid of useless expressions

* take out '+' sign to test if that will fix ruff error

* take out remaing '+' signs

* add in impact in TAGS

* add import Tags line

* add tags=tags to DAG
2024-06-18 13:27:49 -03:00
JCMOSCON1976 1ebe9a9de2
feat(DAG): Workday-XMatters [ASP-4792] - Changed the Jira Issue Type to Bug and Assignee to Julio (#2018)
* Changed Jira issuetype to Bug and assignee to Julio

* Added comments

* Fixed assignee format

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-12 11:53:08 -07:00
JCMOSCON1976 7544eefb4b
fix(DAG): Workday-XMatters [ASP-4631] - Changes in the on_failure_callback (#2017)
* Moved on_failure_callback to default_args

* Fix on_failure_callback function to list.

* Fix on_failure_callback

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-11 05:41:34 -07:00
JCMOSCON1976 2373cd3155
fix(DAG): Workday-XMatters [ASP-4631] - Changes in the on_failure_callback function (#2016)
* Changed the on_failure_callback function

* Make fixes

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-10 15:30:59 -04:00
Katie Windau 88376779ce
feat: DENG-3096 remove from telemetry-airflow so we can move to bqetl (#2015) 2024-06-07 13:08:49 -05:00
JCMOSCON1976 2cc4d69b93
fix(DAG): Workday-XMatters [ASP-4631] - Reverting changes (#2013)
* Fixed back the XMATTERS_URL variable name

* Adding tier_3 tag

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-07 08:28:08 -04:00
JCMOSCON1976 07bf1f9578
fix(DAG): Workday-XMatters [ASP-4631] testing Jira ticket creation on failure (#2012)
* Fixed image string

* Fixed format

* Fixed environment variable name and on_failure_callback

* Changed env variable xmatters_url to generate an error and test Jira ticket creation on failure.

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-05 12:57:26 -07:00
Katie Windau 9318abeabf
feat: [DENG-3096] update to correct gcp conn id (#1982) 2024-06-05 14:06:45 -05:00
Winnie Chan d686559bcf
feat: DENG-3961 Added dataset id to managed backfill staging table name (#2011)
* Added dataset id
2024-06-05 10:11:56 -07:00
JCMOSCON1976 ace66e8b95
fix(DAG): Workday-XMatters [ASP-4631] - (#2010)
* Fixed image string

* Fixed format

* Fixed environment variable name and on_failure_callback

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-05 08:23:33 -04:00
JCMOSCON1976 7788a42164
fix(DAG): Workday-XMatters [ASP-4631] - Fixed image link string (#2008)
* Fixed image string

* Fixed format

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-04 15:42:39 -04:00
Anna Scholtz 1bc54e78bb
feat: Add Monte Carlo POC (#2009) 2024-06-04 12:02:32 -07:00
Winnie Chan 5254b94e51
feat: DENG-3983 Added qualified table name to managed backfill slack notification (#2007)
* Added qualified table name
2024-06-04 07:59:58 -07:00
JCMOSCON1976 378d160201
feat:[ASP-4631] Workday - XMatters integration (#2006)
* Added secrets to the DAG

* Format changes

* Fixed a ruff error

* Fixed format

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-06-03 08:08:08 -07:00
JCMOSCON1976 f9d466c985
feat(dags): Add Workday - XMatters integration DAG (ASP-4546) (#2005)
* First commit with the workday-xmatters DAG

* First commit with the workday-xmatters DAG

* Changed the jira_conn_id name

* Fixed ruff errors

* Fixed a ruff error

* Fixed black format

* Added  apache-airflow-providers-atlassian-jira to requirements

* Fixed requirements.txt

* Fixed pytest error

* Fixed top-level code

* Changed the task definition back to what it was before

* Reformated file

* Added new line

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-05-31 11:49:28 -04:00
Mikaël Ducharme a726a86bba
fix(dags): Revert feat:[ASP 4546 JCM]-Workday - XMatters integration DAG (#1996) (#2004) 2024-05-29 17:56:34 -04:00
JCMOSCON1976 9240183c16
feat:[ASP 4546 JCM]-Workday - XMatters integration DAG (#1996)
* First commit with the workday-xmatters DAG

* First commit with the workday-xmatters DAG

* Changed the jira_conn_id name

* Fixed ruff errors

* Fixed a ruff error

* Fixed black format

* Added  apache-airflow-providers-atlassian-jira to requirements

* Fixed requirements.txt

* Fixed pytest error

* Update dags/eam_workday_xmatters_integration.py

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-05-29 08:52:30 -04:00
Ben Wu b20a871a5c
fix: Set bhr_collection spark memory to 20g (#2003) 2024-05-28 14:54:27 -04:00
Eduardo Filho 3a72448d84
fix(glam): fix wait for copy dedup exec delta (#2001) 2024-05-28 14:41:16 -04:00
Anna Scholtz 0907d75068
fix: Fix SQL path typos for data observability dev DAGs (#2002) 2024-05-28 11:22:04 -07:00
Anna Scholtz 867d452cbc
feat:Add POC dags for triggering data observability checks (#1999)
* feat: Add POC dags for triggering data observability checks

* Bump keyring constraint

* Add observability connections

* Add requirements-override.txt

* Update Dockerfile

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>

* Update requirements-override.txt

* Update .circleci/config.yml

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>

* pip-install-local overrides

* Update dags/bigeye_poc.py

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

---------

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2024-05-28 10:32:20 -07:00
Eduardo Filho 02e1f2371f
fix(glam): reschedule to give fenix and fog more time (#2000) 2024-05-27 14:10:39 -04:00
Mikaël Ducharme 0b4755a383
fix(ci): Actually exit with error when `manual-publish` runs from main. (#1998) 2024-05-27 13:08:39 -04:00
Mikaël Ducharme 0eff7b7ccc
chore(ci): Filter manual-publish workflows. (#1997) 2024-05-27 11:50:17 -04:00
Ben Wu b9a2fc850c
fix: Increase bhr_collection spark memory (#1995) 2024-05-24 15:54:32 -04:00
Mikaël Ducharme f17fa691c9
chore(ci): Prepare CI for CD proposal implementation. (#1993) 2024-05-24 14:14:22 -04:00
Mikaël Ducharme 9097e5a10f
feat(plugins): Add Dockerflow-style version endpoint. (#1994) 2024-05-24 12:06:37 -04:00
Braunk 0d3d2f28af
fix(bqetl_backfill): sandbox arguments are still in main...need to be removed (#1991) 2024-05-21 13:49:27 -05:00
Braunk 4515cfa142
feat(bqetl_backfill): remove sandbox references from previous PR (#1990)
* feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill

* feat(bqetl_backfill): shelex quote was not needed

* fix(bqetl_backfill): remove sandbox references
2024-05-21 13:25:56 -05:00
Braunk 485c8d864f
feat(bqetl_backfill): applying shelex quote removal for scheduling overrides handling that was missing from previous PR (#1989)
* feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill

* feat(bqetl_backfill): shelex quote was not needed
2024-05-21 11:59:55 -05:00
Braunk 41c18a0eac
feat(bqetl_backfill): adding new scheduling overrides json object param to bqetl_backfill (#1987) 2024-05-20 11:26:01 -05:00
Eduardo Filho 50381fd25b
chore(glam): re-schedule glam legacy etl (#1988)
* chore(glam): re-schedule glam legacy etl

* Adjust execution_delta for upstream

* Remove unused tasks from glam_glean_imports

* Update dags/glam.py

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-05-17 17:03:45 -04:00