8d72cfa9fe
Move routine config to bqetl_project.yaml ( #4038 )
2023-07-11 10:52:48 -07:00
d9bda0df7e
Add ConfigLoader and move dry run skip to bqetl_project.yaml ( #4000 )
...
* Add ConfigLoader and move dry run skip to bqetl_project.yaml
* format tests
2023-07-06 10:42:29 -07:00
7c90d5f8e7
Publish view metadata ( #3909 )
2023-06-29 16:28:17 -04:00
01333782b1
DENG 946 - Update DAG generation to include ETL checks ( #3969 )
...
* CAccomodate dq checks in dag generation
* Modify the tests to include dq check
* Generate dags to include bigquery_dq_check
* rename destination to source for dq check
* Add DQ check to download attribution dag
* Update bigquery_etl/query_scheduling/templates/airflow_dag.j2
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update bigquery_etl/query_scheduling/generate_airflow_dags.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Set upstream check dependencies using upstream_dependencies
* Change bigquery_dq_check as per gcp.py utils
* remove sql_file_path in airflow jinja
* Fix download attribution dag
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-06-28 13:50:41 -04:00
1c3ba13b40
Skip non-emails in owner labels ( #3763 )
2023-05-11 17:02:46 -04:00
ac053c326a
Update dependencies missed by dependabot ( #3566 )
2023-02-06 12:14:32 -08:00
02b7e2a37a
Increase Fivetran sensors' poke interval to 30 seconds ( #3481 )
...
Using a poke interval of 5 seconds means only 7 Fivetran sensors polling for an hour would exceed Fivetran's rate limit of 5,000 API requests per hour. Increasing the poke interval to 30 seconds will allow for up to 41 Fivetran sensors polling for an hour.
2023-01-04 11:23:42 -08:00
d76408b314
Retry Fivetran syncs when they fail ( #3480 )
2023-01-04 11:01:17 -08:00
e1eb6df342
Use the `airflow-provider-fivetran` package instead of our backport. ( #3479 )
...
And use the `airflow-provider-fivetran` package's new feature to pass the return value from the Fivetran operator to the Fivetran sensor via XCom so the sensor doesn't miss syncs that finish before it can check.
2023-01-03 15:52:13 -08:00
29907d0792
Replace references of deprecated task_sensor with upstream airflows externaltasksensor ( #3447 )
2022-12-12 12:20:32 -08:00
daae227108
Add date_partition_offset tests
2022-07-05 09:26:21 -07:00
e7d1e1243c
Add task marker docs
2022-06-22 11:05:25 -07:00
2f5c6ac41a
Generate ExternalTaskMarkers for Airflow downstream dependencies
2022-06-22 11:05:25 -07:00
7a3d777e59
removed dag=dag from tempalte since context managed is used ( #2891 )
...
* removed dag=dag from tempalte since context managed is used
* regenerated dags
2022-04-19 15:02:25 +02:00
1c782cbd21
Fix whitespace around fivetran in airflow integration test ( #2863 )
2022-04-04 15:44:59 -07:00
45655229d3
Added sql_generators script to create schema.yaml files for derived views ( #2657 )
2022-01-13 15:46:06 -05:00
d5e5542a07
Support Fivetran import task generation
2022-01-11 11:23:32 -08:00
1531e8d495
when generating dags repo tag automatically set to repo/bigquery-etl
2021-12-23 12:29:09 +00:00
7e683af588
fixed test
2021-12-16 15:13:27 +01:00
feca518031
making sure that the tag follows the agreed name convention
2021-12-16 15:13:27 +01:00
9644a89ef7
bqetl dag create now support --tag and tests updated
2021-12-16 15:13:27 +01:00
cc6bc5bf7b
Handle manual and automatic airflow dependencies consistently ( #2330 )
2021-09-20 17:46:48 +00:00
cb84f91d9f
Revert "Revert "Add view tests""
...
This reverts commit 63764c72cc
2021-08-04 14:49:38 -07:00
63764c72cc
Revert "Add view tests"
...
This reverts commit d0dfeb9701
2021-07-29 10:47:24 -07:00
d0dfeb9701
Add view tests
2021-07-29 10:07:53 -07:00
92281854cd
Use ExternalTaskCompletedSensor in generated Airflow DAGs
2021-07-06 08:34:26 -07:00
1c8180b085
Use ExternalTaskCompletedSensor for bqetl_public_data_json DAG
2021-06-24 08:34:20 -07:00
4886f8fe66
Revert "Add sla and execution_timeout parameters to Airflow tasks"
...
This reverts commit 011dbd6367
2021-06-17 12:27:37 -07:00
151f799713
Revert "Test sla and execution_timeout on public data dag"
...
This reverts commit e1f530d168
2021-06-17 12:27:37 -07:00
e1f530d168
Test sla and execution_timeout on public data dag
2021-06-17 08:30:29 -07:00
011dbd6367
Add sla and execution_timeout parameters to Airflow tasks
2021-06-17 08:30:29 -07:00
8c82b48e87
Bug 1713631 - Use bigquery_etl image from private GCR repository
2021-06-08 17:07:38 +02:00
288e0a687b
Revert "Use dedicated pool for bqetl_public_data_json"
...
This reverts commit 63517fad94
2021-05-27 09:57:38 -07:00
63517fad94
Use dedicated pool for bqetl_public_data_json
2021-05-26 12:03:05 -07:00
3e2e611ce6
Add tests
2021-05-19 12:51:11 -07:00
7c1b03934b
Default branch ( #1939 )
...
* Rename default branch
* Rename branch
* Update circleci for default branch name
2021-04-06 21:15:21 +00:00
dfeea39ac5
Enforce more yaml lint rules ( #1878 )
2021-03-09 17:25:01 -05:00
b46034cd97
Add schema tests
2021-02-19 09:34:15 -08:00
5d07beaca7
Use zetasql to get dependencies for dag generation ( #1817 )
2021-02-18 17:49:46 -05:00
8e90f0b063
Update DAGs with Markdown description
2021-02-01 13:16:17 -08:00
2d2cb6b4a9
Support DAG end_date
2020-12-16 13:06:58 -08:00
9c12adb506
Update generated DAGs, gke_command added
2020-11-10 14:36:07 -08:00
269e0ff803
Add tests for python script query scheduling
2020-11-10 14:36:07 -08:00
1ace0fe2b7
Ran YAMLlint on all yaml files and resolved linting issues ( fixes #1297 ) ( #1481 )
...
* "Ran YAMLlint on all yaml files"
* "Moved product info metadata table to README file"
* "Reformatted yaml lists"
* "Updated line breaks so script runs"
* "Updated line breaks so script runs"
* "Undid line breaks"
* "Created custom config file"
* "Removed base document id"
* "Undid line breaks"
* "Reformatted code"
* "Trimmed whitespace"
* "Undid line break"
* "Introduced newline"
* "Trimmed whitespace"
* "Added yamillint to config file"
* "Added yamllint to config file"
* "Moved up yamllint test"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Removing hyphen to fix CI error"
* "Indentation to remove CI error"
* "Included yamllint install in build run"
* "Added yamllint in requirements.txt and .in file"
* "Moved install yamllint step to its own stage"
* "Updated yamllint test"
* "Updated circleci step"
* "Reformatted code"
* "Added yamllint to circleci steps"
* "Added checkout block to yamllint step"
* "Trimmed whitespace"
* "Undid yamllint step"
* "Specified directory name for yamllint test"
* "Fixed yamlint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Ignore pathway in linting"
* "Added ignore venv pathway during linting"
* "Updated ignore block"
* "Updated ignore block"
* "Removed ignore block"
* "Updated ignore block"
* "Indented base as a list"
* "Indented base item"
* Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Resolved linting errors"
* "Referenced tables put back on same line"
* "Fixed linting error"
* Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Fixed linting error"
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-29 17:24:55 -07:00
89d48cd99e
Add multiple bug test
2020-10-28 09:01:43 -07:00
88b0e2066e
Listify review_bugs in tests and metadata files
2020-10-28 09:01:43 -07:00
2a1454c309
Update metadata validation logic (issue #924 ) ( #1463 )
...
* Validate metadata with attr
* Update and add tests
* Add check if file exists and update tests
* Format code and update validate metadata
* Revert changes to is_metadata_file()
* Remove format error: whitespaces
* Format test files
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-10-20 16:03:50 -07:00
8d1683c628
Fix DAG generation for different projects
2020-10-13 12:07:25 -07:00
5a8311e2af
Update UDF parsing
2020-10-08 10:30:22 -07:00
fc95e32a31
Update DAGs
2020-10-05 12:59:58 -07:00
Anna Scholtz
Linh Nguyen
Alekhya
Alexander
Daniel Thorn
Sean Rose
haroldwoo
kik-kik
=
Arkadiusz Komarzewski
whd
Rhys
jailang