Граф коммитов

33 Коммитов

Автор SHA1 Сообщение Дата
Anna Scholtz 9dd21df2c7
feat(dag): Schedule daily dbt runs (#2056) 2024-07-31 15:08:31 -07:00
Anna Scholtz 79b3199fd4
chore: Remove data observability POC DAGs (#2049) 2024-07-22 12:21:23 -07:00
Anna Scholtz 1bc54e78bb
feat: Add Monte Carlo POC (#2009) 2024-06-04 12:02:32 -07:00
JCMOSCON1976 f9d466c985
feat(dags): Add Workday - XMatters integration DAG (ASP-4546) (#2005)
* First commit with the workday-xmatters DAG

* First commit with the workday-xmatters DAG

* Changed the jira_conn_id name

* Fixed ruff errors

* Fixed a ruff error

* Fixed black format

* Added  apache-airflow-providers-atlassian-jira to requirements

* Fixed requirements.txt

* Fixed pytest error

* Fixed top-level code

* Changed the task definition back to what it was before

* Reformated file

* Added new line

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
2024-05-31 11:49:28 -04:00
Mikaël Ducharme a726a86bba
fix(dags): Revert feat:[ASP 4546 JCM]-Workday - XMatters integration DAG (#1996) (#2004) 2024-05-29 17:56:34 -04:00
JCMOSCON1976 9240183c16
feat:[ASP 4546 JCM]-Workday - XMatters integration DAG (#1996)
* First commit with the workday-xmatters DAG

* First commit with the workday-xmatters DAG

* Changed the jira_conn_id name

* Fixed ruff errors

* Fixed a ruff error

* Fixed black format

* Added  apache-airflow-providers-atlassian-jira to requirements

* Fixed requirements.txt

* Fixed pytest error

* Update dags/eam_workday_xmatters_integration.py

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Julio Cezar Moscon <jcmoscon@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-05-29 08:52:30 -04:00
Anna Scholtz 867d452cbc
feat:Add POC dags for triggering data observability checks (#1999)
* feat: Add POC dags for triggering data observability checks

* Bump keyring constraint

* Add observability connections

* Add requirements-override.txt

* Update Dockerfile

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>

* Update requirements-override.txt

* Update .circleci/config.yml

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>

* pip-install-local overrides

* Update dags/bigeye_poc.py

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

---------

Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2024-05-28 10:32:20 -07:00
Sean Rose 4752c5d846
fix: Remove dev requirements from Docker image and improve `Makefile` (#1953)
* Define dev requirements in `requirements-dev.in`.

* Add `pip-tools` as a dev requirement.

* Remove `pip-compile` as a prerequisite for `pip-install-local`.

To avoid the bootstrapping issue in a fresh Python environment where `pip-compile` isn't installed yet.

* Update `.PHONY` config in `Makefile` to include all non-file targets.

* Update `help` target in `Makefile` to include all available commands.

* Replace tabs in `Makefile` strings with spaces.

* Update CI instructions for fixing lint errors.
2024-03-19 11:49:23 -07:00
kik-kik 7d9927edc7
feat: Upgrade Python to 3.11.8 and remove trailing whitespaces (#1942) 2024-03-14 11:07:53 +01:00
Sean Rose 94b190e3d7
feat: Upgrade Airflow from 2.7.3 to 2.8.2 (#1935)
* Upgrade Airflow from 2.7.3 to 2.8.2.

* Update CircleCI Docker orb from 2.2.0 to 2.5.0.

* Update Ruff config for version 0.2.

* fix dockerfile URL scheme to use https for gcloud CLI

* Restore `google-auth` extra, with its built-in dependency on `authlib`.

* Add missing Airflow state colors for "restarting", "deferred", and "removed".

* Explain the difference between our configured Airflow state colors and the defaults.

---------

Co-authored-by: mikaeld <mducharme@mozilla.com>
2024-03-07 14:01:07 -08:00
Mikaël Ducharme 6e6fdfb019
chore: Cleanup dependencies. (#1918) 2024-02-26 13:12:54 -05:00
kik-kik ec271e2c6d
feat(DENG-2682): Update fivetran imports to use airflow-provider-fivetran-async Airflow Fivetran provider (#1914)
* update fivetran imports to use airflow-provider-fivetran-async

* black and ruff errors fixed
2024-02-21 17:51:16 +01:00
Mikaël Ducharme ff3a2fa82f
chore(plugins): Remove unused plugins and dependencies. (#1911) 2024-02-09 12:31:02 -05:00
Mikaël Ducharme 50455cc0c1
feat: Upgrade to Airflow 2.7.3 (#1854) 2023-11-29 09:38:18 -05:00
Mikaël Ducharme 4a0419fb93
feat: Upgrade to 2.6.3 (#1789) 2023-09-07 14:04:40 -04:00
Mikaël Ducharme 3e3acc0f69
feat: Update Airflow to 2.5.3 (#1742) 2023-06-28 12:41:31 -04:00
Mikaël Ducharme cba4eeb23f
feat: Update to Python 3.10 (#1737) 2023-06-22 09:32:19 -04:00
Mikaël Ducharme d2fbb73616
fix: disable catch up and change start dates. (#1712) 2023-05-26 11:20:59 -04:00
Mikaël Ducharme ac5f33b427
chore(ci): diff based linting (#1658) 2023-03-01 14:47:21 -05:00
Mikaël Ducharme b51ab0a143
feat!: Use Dockerfile based on upstream [DSRE-1116] (#1638)
* CI runtime decreased by 70-80%

* docker-compose setup time decreased significantly. **Local use: takes a few seconds as opposed to a few minutes before**. Airflow Variables and Connections are loaded via `import` CLI command; replaces `bin/run` script using Airflow CLI.

* Improved dev secrets security by dynamically generating a Fernet key in makefile
  * `.env` file is generated from `make up`, contains UID and Fernet key. `.env` is automatically loaded into environment variables by `docker-compose`

* Retire shell scripts to use builtin features in our stack
  * `bin/run` replaced by docker-compose and CI
  * `bin/test-dag-tags` replaced by pytest unit tests
  * `bin/test-parse` replaced by pytest unit tests
2023-02-22 12:52:41 -05:00
Mikaël Ducharme e555ca12d2
chore(ci): use circleci orbs (#1619) 2022-12-20 14:51:41 -05:00
Harold Woo d2505e9ef3 update requirements, introduce constraints file and clean up for 2.3.3 2022-12-12 13:24:03 -05:00
Mikaël Ducharme 0b4b4eb418
Revert "feat(airflow): upgrade airflow from 2.1.4 to 2.3.3 [DSRE-1039] " (#1612)
* Revert "update airflow config for 2.3.3"

This reverts commit d19cc711aa.

* Revert "fix deprecation warnings, clean up and update for 2.3.3"

This reverts commit e80472ab9a.

* Revert "update requirements, introduce constraints file and clean up for 2.3.3"

This reverts commit 8e60dba783.
2022-12-07 18:25:40 -05:00
Harold Woo 8e60dba783 update requirements, introduce constraints file and clean up for 2.3.3 2022-12-07 13:30:42 -05:00
Harold Woo f19c6b80c3 Add pubsub and slack example operators 2022-04-04 12:23:16 -07:00
kik-kik e04f16b83e
added dag for trigerring acoustic fivetran connector run (#1470)
Co-authored-by: kik <Anich, aschultz>
2022-03-29 12:25:05 +02:00
kik-kik df796f6105
Airflow tags (#1430)
* Airflow tags added + tests + pytest to requirements

* added pytest execution to test stage in circleci pipeline

* moved tests out of dags/

Co-authored-by: = <=>
2022-01-06 19:13:55 +01:00
Harold Woo 1553630f73 Upgrade Airflow to 2.1.4 to fix regression 2021-12-14 11:38:43 -08:00
Harold Woo d14b003a8e Update airflow to 2.1.3 to fix Flask issues with UI User permission editing 2021-12-07 14:09:18 -08:00
Harold Woo e1518a5ff5 [DSRE-6] Upgrade Airflow (wtmo) to 2.1.1 2021-10-18 12:00:34 -07:00
Harold Woo 9bd76d28a5 Upgrade Airflow to 1.10.15 bridge release prior to 2.0 2021-07-20 11:34:26 -07:00
Harold Woo e6456b0132 Upgrade airflow WTMO to 1.10.12 2020-11-09 13:43:08 -08:00
Anthony Miyaguchi 4fb86a1d2d
Bug 1570368 - Update airflow from python2 to python3 (#1008)
* Upgrade to 1.10.10 and pip-compile using python3.8

* pip-compile with python3.7

* Update Dockerfile from 2.7-slim to 3.7-slim

* Remove deprecated option from config

* Update relative imports in backported operators

* Replace urlparse to new location

* Replace iteritems with items

* Add instructions for pip-compile and cut out old material

* Update unittests for plugins

* Update README.md

Co-authored-by: Daniel Thorn <dthorn@mozilla.com>

* Update README.md

Co-authored-by: Daniel Thorn <dthorn@mozilla.com>

* Compile requirements using explicit source

* Add CI for checking requirements are generated correctly

* Leave off requirements.in from pip-compile call

* Recompile requirements.txt

* Pin kombu to 4.6.3

Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2020-07-22 10:09:07 -07:00