80fd8de864
fix: 1952 Acoustic Suppression List DENG 3082 date format ( #1955 )
...
* fix: 1952 Acoustic Suppression List DENG 3082 date format
* change start_date
2024-03-19 15:22:03 +01:00
50744dec59
feat: add data-observability-dev to the list of projects covered by artifact_deployment ( #1954 )
2024-03-19 14:35:44 +01:00
1c463acbd5
fix: 1951 - acoustic suppression list date format ( #1952 )
...
* fix 1951 - acoustic suppression list date format
* empty commit
2024-03-15 20:31:53 +01:00
5c8966f646
feat: add acoustic supression list export ( #1951 )
2024-03-15 16:53:09 +01:00
7d9927edc7
feat: Upgrade Python to 3.11.8 and remove trailing whitespaces ( #1942 )
2024-03-14 11:07:53 +01:00
32330c82a6
feat: Make args parameter optional in operational_monitoring_backfill ( #1949 )
2024-03-13 10:50:30 -07:00
723f473776
feat: Add DAG for running opmon backfills ( #1946 )
...
* feat: Add DAG for running opmon backfills
* Add param for specifying additional CLI arguments
* Update dags/operational_monitoring_backfill.py
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
---------
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
2024-03-12 12:58:43 -07:00
474ab8d5b4
feat: Add repo tag ( #1947 )
...
* add repo tag
* add repo tag
* Update utils/tags.py
* Update utils/tags.py
* Fix formatting
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-03-12 12:45:53 -07:00
f317239ff2
chore: Replace gke_command usage with GKEPodOperator. ( #1948 )
2024-03-12 15:08:53 -04:00
5166835fd5
docs: update mad_server owner ( #1945 )
2024-03-12 11:13:59 -04:00
4837063903
Revert "Bring back SQL generation in artifact deployments ( #1931 )" ( #1940 )
...
This reverts commit d89bc5b408
2024-03-11 11:37:15 -07:00
0de70425a4
chore: Add PR template. ( #1943 )
2024-03-11 11:40:45 -04:00
94b190e3d7
feat: Upgrade Airflow from 2.7.3 to 2.8.2 ( #1935 )
...
* Upgrade Airflow from 2.7.3 to 2.8.2.
* Update CircleCI Docker orb from 2.2.0 to 2.5.0.
* Update Ruff config for version 0.2.
* fix dockerfile URL scheme to use https for gcloud CLI
* Restore `google-auth` extra, with its built-in dependency on `authlib`.
* Add missing Airflow state colors for "restarting", "deferred", and "removed".
* Explain the difference between our configured Airflow state colors and the defaults.
---------
Co-authored-by: mikaeld <mducharme@mozilla.com>
2024-03-07 14:01:07 -08:00
e593fafa15
chore(dags): Delete overwatch.py ( #1937 )
2024-03-07 15:49:32 -05:00
5cf735b52b
fix(dockerfille): google-cloud-sdk renamed google-cloud-cli. ( #1938 )
2024-03-07 15:42:15 -05:00
5022c25e03
Bug 1821773 - Update debugging notes for probe scraper and bqetl deploy ( #1933 )
2024-03-01 12:26:09 -05:00
5c92093f2d
Change Fivetran Airflow operators to run synchronously so the task concurrency limit is respected. ( #1932 )
2024-02-29 14:10:53 -08:00
d89bc5b408
Bring back SQL generation in artifact deployments ( #1931 )
2024-02-29 09:48:20 -08:00
9e2dbf505a
bqetl_artifact_deployment max_active_runs and remove SQL generation ( #1930 )
2024-02-28 13:44:17 -08:00
387e5969e9
DSRE-1520 Manage secrets for adm_export dag in GSM ( #1928 )
2024-02-26 15:10:52 -06:00
ed65f01eb9
DSRE-1519 Fix secrets in merino_jobs dag ( #1927 )
...
* DSRE-1519 Fix secrets in merino_jobs dag
* fix: secret import path
2024-02-26 12:25:15 -06:00
6e6fdfb019
chore: Cleanup dependencies. ( #1918 )
2024-02-26 13:12:54 -05:00
722e1a6511
Publish glam managed views ( #1926 )
2024-02-22 16:16:15 -05:00
dd5adcf346
Fix `bqetl_artifact_deployment.publish_views` to publish views in alternate projects. ( #1925 )
2024-02-22 09:44:42 -08:00
6ef0675bcb
Limit Fivetran task concurrency so it only tries to run one sync per connector at a time. ( #1923 )
2024-02-21 11:56:22 -08:00
cf98c995aa
Remove FivetranSensor to avoid double wait for sync to complete ( #1922 )
2024-02-21 19:11:21 +01:00
ec271e2c6d
feat(DENG-2682): Update fivetran imports to use airflow-provider-fivetran-async Airflow Fivetran provider ( #1914 )
...
* update fivetran imports to use airflow-provider-fivetran-async
* black and ruff errors fixed
2024-02-21 17:51:16 +01:00
68bfbe9d71
DSRE-1526 - Move probe_scraper secret to GSM ( #1921 )
2024-02-20 17:43:45 +01:00
e342665fec
DSRE-1526 - Move probe_scraper secrets to GSM ( #1917 )
...
* DSRE-1526 - Move probe_scraper secrets to GSM
* lint
* Add dev variable
2024-02-19 12:00:37 +01:00
a72518f664
Issue 4135: Added publish metadata task ( #1912 )
...
* Added publish metadata task
* Changed to gke pod operator
* Updated wild cards
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
2024-02-14 13:26:39 -08:00
050aeb4940
Remove stray && in bqetl_artifact_deployment ( #1919 )
2024-02-14 11:00:14 -08:00
bf1b839764
Add project moz-fx-data-glam-prod-fca7 to the DAG bqetl_artifact_deployment. ( #1916 )
2024-02-13 12:37:55 -05:00
ff3a2fa82f
chore(plugins): Remove unused plugins and dependencies. ( #1911 )
2024-02-09 12:31:02 -05:00
aa61933cec
Remove bugzilla DAG ( #1910 )
2024-02-08 15:37:21 -08:00
5a23bbbc0e
chore: delete code leftovers. ( #1909 )
2024-02-08 12:39:53 -05:00
486ca7ac66
chore: delete burnham and bq_sensor operator. ( #1905 )
2024-01-31 11:09:52 -05:00
013f13bd5d
GLAM move secrets outside of Airflow ( #1906 )
...
* Stop secrets from showing on Airflow UI
* Glam imports fetches secrets from GSM instead of Airflow
* Lint fixes
* Replace gke_command with GKEPodOperator
2024-01-30 15:44:00 -05:00
8402c46c7c
Remove adjust_import and android_client_history_sim dags ( #1879 )
...
Co-authored-by: Mikaël Ducharme <mducharme@mozilla.com>
2024-01-30 15:35:58 -05:00
71809f487c
Remove GLAM dev DAG ( #1900 )
...
* Remove glam dev dag
2024-01-26 15:27:49 -05:00
aaf20325d1
chore: Delete GKENatPodOperator. ( #1903 )
2024-01-26 14:17:46 -05:00
56dc0bfe1b
chore: Delete create_gke_cluster. ( #1904 )
2024-01-26 14:09:16 -05:00
289211b5cf
DSRE-52: Remove mozaggregator_prerelease DAG from Airflow ( #1899 )
2024-01-25 15:38:31 -05:00
4050697bd2
Have `bqetl_artifact_deployment` publish new `glean_usage`-generated tables. ( #1896 )
2024-01-10 16:47:37 -08:00
eed962c2eb
Remove unnecessary no_partition parameter ( #1894 )
...
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-10 16:49:37 -05:00
37d5987e8c
Run query initialization before schema deploys ( #1893 )
2024-01-10 12:41:28 -08:00
03274fdad2
Fix parameter in new DAG ga4_site_metrics_summary_backfill ( #1892 )
2024-01-05 14:18:19 -06:00
ecfbd52a70
Add new DAG ga4_site_metrics_summary_backfill ( #1891 )
...
* Add new DAG ga4_site_metrics_summary_backfill - initial commit
* reformat new dag using black
2024-01-05 13:12:32 -06:00
791b40f745
FXCTO-13 Update dap_collector's output destination ( #1887 )
2023-12-22 17:37:55 +01:00
cb44e2eb43
Fixes #1889 - Add a dag definition to schedule broken-site-report-ml job ( #1890 )
2023-12-22 07:45:08 -08:00
35fc486157
Remove dthorn as owner of dags and tasks ( #1883 )
2023-12-18 11:52:52 -08:00
Leli
kik-kik
Anna Scholtz
Mikaël Ducharme
akkomar
Sean Rose
Ben Wu
wil stuckey
Eduardo Filho
Winnie Chan
Lucia
Frank Bertsch
Alexander
Katie Windau
Ksenia
Daniel Thorn