daae227108
Add date_partition_offset tests
2022-07-05 09:26:21 -07:00
e7d1e1243c
Add task marker docs
2022-06-22 11:05:25 -07:00
2f5c6ac41a
Generate ExternalTaskMarkers for Airflow downstream dependencies
2022-06-22 11:05:25 -07:00
7a3d777e59
removed dag=dag from tempalte since context managed is used ( #2891 )
...
* removed dag=dag from tempalte since context managed is used
* regenerated dags
2022-04-19 15:02:25 +02:00
1c782cbd21
Fix whitespace around fivetran in airflow integration test ( #2863 )
2022-04-04 15:44:59 -07:00
45655229d3
Added sql_generators script to create schema.yaml files for derived views ( #2657 )
2022-01-13 15:46:06 -05:00
d5e5542a07
Support Fivetran import task generation
2022-01-11 11:23:32 -08:00
1531e8d495
when generating dags repo tag automatically set to repo/bigquery-etl
2021-12-23 12:29:09 +00:00
7e683af588
fixed test
2021-12-16 15:13:27 +01:00
feca518031
making sure that the tag follows the agreed name convention
2021-12-16 15:13:27 +01:00
9644a89ef7
bqetl dag create now support --tag and tests updated
2021-12-16 15:13:27 +01:00
cc6bc5bf7b
Handle manual and automatic airflow dependencies consistently ( #2330 )
2021-09-20 17:46:48 +00:00
cb84f91d9f
Revert "Revert "Add view tests""
...
This reverts commit 63764c72cc
2021-08-04 14:49:38 -07:00
63764c72cc
Revert "Add view tests"
...
This reverts commit d0dfeb9701
2021-07-29 10:47:24 -07:00
d0dfeb9701
Add view tests
2021-07-29 10:07:53 -07:00
92281854cd
Use ExternalTaskCompletedSensor in generated Airflow DAGs
2021-07-06 08:34:26 -07:00
1c8180b085
Use ExternalTaskCompletedSensor for bqetl_public_data_json DAG
2021-06-24 08:34:20 -07:00
4886f8fe66
Revert "Add sla and execution_timeout parameters to Airflow tasks"
...
This reverts commit 011dbd6367
2021-06-17 12:27:37 -07:00
151f799713
Revert "Test sla and execution_timeout on public data dag"
...
This reverts commit e1f530d168
2021-06-17 12:27:37 -07:00
e1f530d168
Test sla and execution_timeout on public data dag
2021-06-17 08:30:29 -07:00
011dbd6367
Add sla and execution_timeout parameters to Airflow tasks
2021-06-17 08:30:29 -07:00
8c82b48e87
Bug 1713631 - Use bigquery_etl image from private GCR repository
2021-06-08 17:07:38 +02:00
288e0a687b
Revert "Use dedicated pool for bqetl_public_data_json"
...
This reverts commit 63517fad94
2021-05-27 09:57:38 -07:00
63517fad94
Use dedicated pool for bqetl_public_data_json
2021-05-26 12:03:05 -07:00
3e2e611ce6
Add tests
2021-05-19 12:51:11 -07:00
7c1b03934b
Default branch ( #1939 )
...
* Rename default branch
* Rename branch
* Update circleci for default branch name
2021-04-06 21:15:21 +00:00
dfeea39ac5
Enforce more yaml lint rules ( #1878 )
2021-03-09 17:25:01 -05:00
b46034cd97
Add schema tests
2021-02-19 09:34:15 -08:00
5d07beaca7
Use zetasql to get dependencies for dag generation ( #1817 )
2021-02-18 17:49:46 -05:00
8e90f0b063
Update DAGs with Markdown description
2021-02-01 13:16:17 -08:00
2d2cb6b4a9
Support DAG end_date
2020-12-16 13:06:58 -08:00
9c12adb506
Update generated DAGs, gke_command added
2020-11-10 14:36:07 -08:00
269e0ff803
Add tests for python script query scheduling
2020-11-10 14:36:07 -08:00
1ace0fe2b7
Ran YAMLlint on all yaml files and resolved linting issues ( fixes #1297 ) ( #1481 )
...
* "Ran YAMLlint on all yaml files"
* "Moved product info metadata table to README file"
* "Reformatted yaml lists"
* "Updated line breaks so script runs"
* "Updated line breaks so script runs"
* "Undid line breaks"
* "Created custom config file"
* "Removed base document id"
* "Undid line breaks"
* "Reformatted code"
* "Trimmed whitespace"
* "Undid line break"
* "Introduced newline"
* "Trimmed whitespace"
* "Added yamillint to config file"
* "Added yamllint to config file"
* "Moved up yamllint test"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Removing hyphen to fix CI error"
* "Indentation to remove CI error"
* "Included yamllint install in build run"
* "Added yamllint in requirements.txt and .in file"
* "Moved install yamllint step to its own stage"
* "Updated yamllint test"
* "Updated circleci step"
* "Reformatted code"
* "Added yamllint to circleci steps"
* "Added checkout block to yamllint step"
* "Trimmed whitespace"
* "Undid yamllint step"
* "Specified directory name for yamllint test"
* "Fixed yamlint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Ignore pathway in linting"
* "Added ignore venv pathway during linting"
* "Updated ignore block"
* "Updated ignore block"
* "Removed ignore block"
* "Updated ignore block"
* "Indented base as a list"
* "Indented base item"
* Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Resolved linting errors"
* "Referenced tables put back on same line"
* "Fixed linting error"
* Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Fixed linting error"
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-29 17:24:55 -07:00
89d48cd99e
Add multiple bug test
2020-10-28 09:01:43 -07:00
88b0e2066e
Listify review_bugs in tests and metadata files
2020-10-28 09:01:43 -07:00
2a1454c309
Update metadata validation logic (issue #924 ) ( #1463 )
...
* Validate metadata with attr
* Update and add tests
* Add check if file exists and update tests
* Format code and update validate metadata
* Revert changes to is_metadata_file()
* Remove format error: whitespaces
* Format test files
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-10-20 16:03:50 -07:00
8d1683c628
Fix DAG generation for different projects
2020-10-13 12:07:25 -07:00
5a8311e2af
Update UDF parsing
2020-10-08 10:30:22 -07:00
fc95e32a31
Update DAGs
2020-10-05 12:59:58 -07:00
87f1a4e19f
Update tests
2020-10-05 12:59:58 -07:00
06233819ab
Remove sql/ directory
2020-10-05 12:59:58 -07:00
9827b725e0
Alternate project support for query scheduling
2020-10-05 12:59:58 -07:00
efb49cf738
Dryrun stored procedures ( #1367 )
...
* Dryrun stored procedures
* Reformat SQL
* Update tests for reformatted sql
* Fix doc format again
* Let black run past max-line-length
Black does not mandate line-length, it only tries to adhere to it.
For that reason, we can't have flake8 fail at too-long line length.
https://black.readthedocs.io/en/stable/the_black_code_style.html#line-length
* Why doesn't pydoctest run locally?
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-10-01 16:48:28 -04:00
3caa5eb382
Event analysis udfs ( #1343 )
...
* WIP: Event analysis UDFs
* WIP: Some stored procedures for working with events
* Add UDF descriptions && move datasets
* Add capabilities for testing procedures
* Add procedures for creating funnel views
* Dryrun stored procedures
* Add support for count queries to events view
* Remove legacy stored procedures
* Remove dryruns
* Add explicit count support
* Change SQL to work around sqlparse bug
See https://github.com/andialbrecht/sqlparse/issues/583
* Run black
* Reformat SQL
* Reformat assertion
* Fix doc issues
* Add stored procedure test file
* Flake8 issue
* Reformat last file
* Ignore pre-colon whitespace failure
* Also ignore line breaks before binary operators
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Fix sql lint
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-29 16:33:38 -04:00
268ea08e81
Add tests for UDF description publishing
2020-09-16 10:40:42 -07:00
26c67c7ee8
Upgrade to pytest 6.0.1 ( #1281 )
...
Also upgrade and fix pytest plugins
2020-09-02 11:30:14 -07:00
6c7ad57b55
Prefix test data UDFs with test_
2020-09-02 10:24:38 -07:00
3bdb418588
Add tests for parsing UDFs
2020-09-02 10:24:38 -07:00
1d5483afb5
Update DAG generation tests
2020-08-21 11:10:58 -07:00
Anna Scholtz
kik-kik
Daniel Thorn
Alexander Nicholson
=
Arkadiusz Komarzewski
whd
Rhys
jailang
Frank Bertsch