3234be57c6
feat: Default reattach_on_restart parameter to True in bigquery_etl_query function ( #1961 )
...
We've been hit (again) by Airflow losing track of a running pod, restarting a task and failing with `More than one pod running with labels ...` error.
We previously set some of the tasks to reattach to running pods on restart ([1], [2], [3]) which solved this problem for them. We didn't set it by default for all tasks back then to avoid situations where not-autodeleted pods are stuck with the old code. Now that DEs have permissions required to delete pods it seems safe to revisit this.
[1] https://github.com/mozilla/telemetry-airflow/pull/1820
[2] https://github.com/mozilla/telemetry-airflow/pull/1834
[3] https://github.com/mozilla/telemetry-airflow/pull/1837
2024-04-08 18:40:45 +02:00
09acddb2b3
fix(managed-backfills): typo in Slack message, add owners ( #1959 )
2024-04-02 15:09:11 -04:00
6487786f9f
fix(dags): mad_server secrets are managed on GSM. ( #1958 )
...
* mad_server secrets are managed on GSM
* lint fixes
2024-03-26 16:58:20 -04:00
bd2bc2bd95
feat(managed-backfills): DENG-989 - Use overwatch Slack app for managed backfills ( #1957 )
2024-03-26 14:29:24 -04:00
22920ef969
feat(managed-backfills): DENG-983 Add DAGs for initiating and completing backfills ( #1956 )
...
* feat(managed-backfills): DENG-983 Add DAGs for initiating and completing backfills
2024-03-26 13:25:20 -04:00
4752c5d846
fix: Remove dev requirements from Docker image and improve `Makefile` ( #1953 )
...
* Define dev requirements in `requirements-dev.in`.
* Add `pip-tools` as a dev requirement.
* Remove `pip-compile` as a prerequisite for `pip-install-local`.
To avoid the bootstrapping issue in a fresh Python environment where `pip-compile` isn't installed yet.
* Update `.PHONY` config in `Makefile` to include all non-file targets.
* Update `help` target in `Makefile` to include all available commands.
* Replace tabs in `Makefile` strings with spaces.
* Update CI instructions for fixing lint errors.
2024-03-19 11:49:23 -07:00
ad49849fc7
feat: Create shredder backfill dag ( #1941 )
2024-03-19 11:38:52 -04:00
80fd8de864
fix: 1952 Acoustic Suppression List DENG 3082 date format ( #1955 )
...
* fix: 1952 Acoustic Suppression List DENG 3082 date format
* change start_date
2024-03-19 15:22:03 +01:00
50744dec59
feat: add data-observability-dev to the list of projects covered by artifact_deployment ( #1954 )
2024-03-19 14:35:44 +01:00
1c463acbd5
fix: 1951 - acoustic suppression list date format ( #1952 )
...
* fix 1951 - acoustic suppression list date format
* empty commit
2024-03-15 20:31:53 +01:00
5c8966f646
feat: add acoustic supression list export ( #1951 )
2024-03-15 16:53:09 +01:00
7d9927edc7
feat: Upgrade Python to 3.11.8 and remove trailing whitespaces ( #1942 )
2024-03-14 11:07:53 +01:00
32330c82a6
feat: Make args parameter optional in operational_monitoring_backfill ( #1949 )
2024-03-13 10:50:30 -07:00
723f473776
feat: Add DAG for running opmon backfills ( #1946 )
...
* feat: Add DAG for running opmon backfills
* Add param for specifying additional CLI arguments
* Update dags/operational_monitoring_backfill.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
2024-03-12 12:58:43 -07:00
474ab8d5b4
feat: Add repo tag ( #1947 )
...
* add repo tag
* add repo tag
* Update utils/tags.py
* Update utils/tags.py
* Fix formatting
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-03-12 12:45:53 -07:00
f317239ff2
chore: Replace gke_command usage with GKEPodOperator. ( #1948 )
2024-03-12 15:08:53 -04:00
5166835fd5
docs: update mad_server owner ( #1945 )
2024-03-12 11:13:59 -04:00
4837063903
Revert "Bring back SQL generation in artifact deployments ( #1931 )" ( #1940 )
...
This reverts commit d89bc5b408
2024-03-11 11:37:15 -07:00
0de70425a4
chore: Add PR template. ( #1943 )
2024-03-11 11:40:45 -04:00
94b190e3d7
feat: Upgrade Airflow from 2.7.3 to 2.8.2 ( #1935 )
...
* Upgrade Airflow from 2.7.3 to 2.8.2.
* Update CircleCI Docker orb from 2.2.0 to 2.5.0.
* Update Ruff config for version 0.2.
* fix dockerfile URL scheme to use https for gcloud CLI
* Restore `google-auth` extra, with its built-in dependency on `authlib`.
* Add missing Airflow state colors for "restarting", "deferred", and "removed".
* Explain the difference between our configured Airflow state colors and the defaults.
---------
Co-authored-by: mikaeld <mducharme@mozilla.com>
2024-03-07 14:01:07 -08:00
e593fafa15
chore(dags): Delete overwatch.py ( #1937 )
2024-03-07 15:49:32 -05:00
5cf735b52b
fix(dockerfille): google-cloud-sdk renamed google-cloud-cli. ( #1938 )
2024-03-07 15:42:15 -05:00
5022c25e03
Bug 1821773 - Update debugging notes for probe scraper and bqetl deploy ( #1933 )
2024-03-01 12:26:09 -05:00
5c92093f2d
Change Fivetran Airflow operators to run synchronously so the task concurrency limit is respected. ( #1932 )
2024-02-29 14:10:53 -08:00
d89bc5b408
Bring back SQL generation in artifact deployments ( #1931 )
2024-02-29 09:48:20 -08:00
9e2dbf505a
bqetl_artifact_deployment max_active_runs and remove SQL generation ( #1930 )
2024-02-28 13:44:17 -08:00
387e5969e9
DSRE-1520 Manage secrets for adm_export dag in GSM ( #1928 )
2024-02-26 15:10:52 -06:00
ed65f01eb9
DSRE-1519 Fix secrets in merino_jobs dag ( #1927 )
...
* DSRE-1519 Fix secrets in merino_jobs dag
* fix: secret import path
2024-02-26 12:25:15 -06:00
6e6fdfb019
chore: Cleanup dependencies. ( #1918 )
2024-02-26 13:12:54 -05:00
722e1a6511
Publish glam managed views ( #1926 )
2024-02-22 16:16:15 -05:00
dd5adcf346
Fix `bqetl_artifact_deployment.publish_views` to publish views in alternate projects. ( #1925 )
2024-02-22 09:44:42 -08:00
6ef0675bcb
Limit Fivetran task concurrency so it only tries to run one sync per connector at a time. ( #1923 )
2024-02-21 11:56:22 -08:00
cf98c995aa
Remove FivetranSensor to avoid double wait for sync to complete ( #1922 )
2024-02-21 19:11:21 +01:00
ec271e2c6d
feat(DENG-2682): Update fivetran imports to use airflow-provider-fivetran-async Airflow Fivetran provider ( #1914 )
...
* update fivetran imports to use airflow-provider-fivetran-async
* black and ruff errors fixed
2024-02-21 17:51:16 +01:00
68bfbe9d71
DSRE-1526 - Move probe_scraper secret to GSM ( #1921 )
2024-02-20 17:43:45 +01:00
e342665fec
DSRE-1526 - Move probe_scraper secrets to GSM ( #1917 )
...
* DSRE-1526 - Move probe_scraper secrets to GSM
* lint
* Add dev variable
2024-02-19 12:00:37 +01:00
a72518f664
Issue 4135: Added publish metadata task ( #1912 )
...
* Added publish metadata task
* Changed to gke pod operator
* Updated wild cards
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
2024-02-14 13:26:39 -08:00
050aeb4940
Remove stray && in bqetl_artifact_deployment ( #1919 )
2024-02-14 11:00:14 -08:00
bf1b839764
Add project moz-fx-data-glam-prod-fca7 to the DAG bqetl_artifact_deployment. ( #1916 )
2024-02-13 12:37:55 -05:00
ff3a2fa82f
chore(plugins): Remove unused plugins and dependencies. ( #1911 )
2024-02-09 12:31:02 -05:00
aa61933cec
Remove bugzilla DAG ( #1910 )
2024-02-08 15:37:21 -08:00
5a23bbbc0e
chore: delete code leftovers. ( #1909 )
2024-02-08 12:39:53 -05:00
486ca7ac66
chore: delete burnham and bq_sensor operator. ( #1905 )
2024-01-31 11:09:52 -05:00
013f13bd5d
GLAM move secrets outside of Airflow ( #1906 )
...
* Stop secrets from showing on Airflow UI
* Glam imports fetches secrets from GSM instead of Airflow
* Lint fixes
* Replace gke_command with GKEPodOperator
2024-01-30 15:44:00 -05:00
8402c46c7c
Remove adjust_import and android_client_history_sim dags ( #1879 )
...
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
2024-01-30 15:35:58 -05:00
71809f487c
Remove GLAM dev DAG ( #1900 )
...
* Remove glam dev dag
2024-01-26 15:27:49 -05:00
aaf20325d1
chore: Delete GKENatPodOperator. ( #1903 )
2024-01-26 14:17:46 -05:00
56dc0bfe1b
chore: Delete create_gke_cluster. ( #1904 )
2024-01-26 14:09:16 -05:00
289211b5cf
DSRE-52: Remove mozaggregator_prerelease DAG from Airflow ( #1899 )
2024-01-25 15:38:31 -05:00
4050697bd2
Have `bqetl_artifact_deployment` publish new `glean_usage`-generated tables. ( #1896 )
2024-01-10 16:47:37 -08:00
akkomar
Alexander
Eduardo Filho
Sean Rose
Ben Wu
Leli
kik-kik
Anna Scholtz
Mikaël Ducharme
wil stuckey
Winnie Chan
Lucia
Frank Bertsch