75484bc37e
[AIRFLOW-5446] Rewrite Google KMS Hook to Google Cloud Python ( #6065 )
2019-09-10 14:12:01 +02:00
455eb41ba6
[AIRFLOW-5437] Better python version detection/explanation. ( #6060 )
...
We have fairly complex python version detection in our CI scripts.
They have to handle several cases:
1) Running builds on DockerHub (we cannot pass different environment
variables there, so we detect python version based on the image
name being build (airflow:master-python3.7 -> PYTHON_VERSION=3.7)
2) Running builds on Travis CI. We use python version determined
from default python3 version available on the path. This way we
do not have to specify PYTHON_VERSION separately in each job,
we just specify which host python version is used for that job.
This makes a nice UI experience where you see python version in
Travis UI.
3) Running builds locally via scripts where we can pass PYTHON_VERSION
as environment variable.
4) Running builds locally for the first time with Breeze. By default
we determine the version based on default python3 version we have
in the host system (3.5, 3.6 or 3.7) and we use this one.
5) Selecting python version with Breeze's --python switch. This will
override python version but it will also store the last used version
of python in .build directory so that it is automatically used next
time.
This change adds necessary explanations to the code that works for
all the cases and fixes some of the edge-cases we had. It also
extracts the code to common directory.
2019-09-10 10:16:10 +02:00
b21be80235
[AIRFLOW-5402] Remove deprecated logger ( #6006 )
...
* [AIRFLOW-5402] Remove deprecated logger
* Remove the related test
* Less is more
2019-09-09 15:18:38 +02:00
6b2a290ddf
[AIRFLOW-5443] Use alpine image in Kubernetes's sidecar ( #6059 )
2019-09-09 14:42:06 +02:00
d31c09236c
[AIRFLOW-3601] Update operators to BigQuery to support location ( #6020 )
2019-09-09 11:10:37 +02:00
6db9f86925
[AIRFLOW-4983] Add ability for DataflowPythonOperator to submit jobs w/ python3 ( #5602 )
2019-09-09 08:02:38 +02:00
0ffc14c30a
[AIRFLOW-5425] Use logging not printing in LoggingCommandExecutor ( #6032 )
...
This will show proper message in log (yellow color) which will be harder to miss.
2019-09-08 10:36:50 +02:00
867cd0f714
[AIRFLOW-5368] Display DAG from the CLI
2019-09-07 18:13:07 +02:00
cb6dfdfabe
[AIRFLOW-2842] Add GoogleCloudStorageSynchronizeBuckets operator
2019-09-07 17:20:41 +02:00
ae0d03eba7
[AIRFLOW-5401] Add support for project_id from connection in GKE
...
This change adds support for reading project_id value from
connection configuration GKE operators.
2019-09-07 17:14:36 +02:00
004f353342
[AIRFLOW-3511][AIRFLOW-3512] Add Cloud Memorystore integration
2019-09-07 17:12:52 +02:00
1b1d79c0a6
[AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator ( #5769 )
...
* [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator
2019-09-06 12:48:42 +02:00
0076e17a91
[AIRFLOW-4758] Add GcsToGDriveOperator operator ( #5822 )
...
* [AIRFLOW-4758] Add GcsToGDriveOperator operator
2019-09-06 12:45:15 +02:00
6b82b9ef91
[AIRFLOW-5318] Option to specify location of the new BQ dataset ( #5923 )
2019-09-06 12:41:40 +02:00
f4b64b3557
[AIRFLOW-5412] Add get_conn/get_client to hooks tests ( #6018 )
...
* [AIRFLOW-5412] Add get_conn/get_client to hooks tests
2019-09-06 12:20:38 +02:00
dd36d90cad
[AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook ( #5671 )
...
* AIRFLOW-5049 Add validation for src_fmt_configs in bigquery hook
Adds validation for the src_fmt_configs arguments in the bigquery hook. Otherwise wrong src_fmt_configs would be silently ignored which is non-desireable.
* [AIRFLOW-5049] Update - Add validation for src_fmt_configs in bigquery hook
Adds a common method for validating the src_ftm_configs
2019-09-06 08:50:26 +02:00
4ab6982f0b
[AIRFLOW-5072] gcs_hook should download once ( #5685 )
...
When a user supplied a filename the expected behaviour is that airflow
downloads the file and does not return it's content as a string.
2019-09-05 17:54:07 +02:00
756069766d
[AIRFLOW-3804] Extend MySQL to GCS operator tests ( #5993 )
...
This commit adds additional test related to problem mentioned in initial
issue https://issues.apache.org/jira/browse/AIRFLOW-3804\#
2019-09-05 17:49:19 +02:00
0948bcf868
[AIRFLOW-5319] Add system tests for Dataproc ( #5924 )
...
* [AIRFLOW-5319] Add system tests for Dataproc
2019-09-05 17:44:32 +02:00
7018f47460
[AIRFLOW-5361] Add system tests for BigQuery ( #5968 )
2019-09-05 17:43:04 +02:00
ad61f3272a
[AIRFLOW-5332] Add system tests for Datastore ( #5937 )
2019-09-05 17:35:31 +02:00
0f548b8cf2
[AIRFLOW-5403] Fix input check in GKE Operator ( #6004 )
2019-09-05 16:48:14 +02:00
9165739e90
[AIRFLOW-5409] Fix BigQuery hook tests ( #6017 )
2019-09-05 15:37:07 +02:00
57f77e0c09
[AIRFLOW-5308] Pass credentials object to pandas_gbq ( #5911 )
2019-09-05 12:55:06 +02:00
a4ebd9d2eb
[AIRFLOW-5358] Improved mocking of the project ID ( #5964 )
...
* [AIRFLOW-5358] Improved mocking of the project ID
2019-09-05 12:49:45 +02:00
17d4179db2
[AIRFLOW-4851] Refactor K8S codebase with k8s API models ( #5481 )
...
* [AIRLFOW-4851] refactor Airflow kubernetes
* [AIRFLOW-4851] refactor Airflow k8s models
* [AIRFLOW-4851] Fix linting and tests
* Refactor and add some tests
* [AIRLFOW-4851] Add assertions to PodOperator tests
2019-09-04 16:24:31 -05:00
9143893c99
[AIRFLOW-5347] Add system tests for GoogleCloudStorage ( #5951 )
2019-09-04 22:21:15 +02:00
c9e2d04fde
[AIRFLOW-4085] FileSensor now takes glob patterns for `filepath` ( #5358 )
2019-09-04 11:02:28 +01:00
ba9e521e71
[AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator ( #5948 )
...
`spark2-submit` supports `--proxy-user` parameter which should be handled by SparkSubmitOperator.
```
$ spark2-submit --help 2>&1 | grep proxy
--proxy-user NAME User to impersonate when submitting the application.
```
2019-09-04 10:47:09 +01:00
13843151fb
[Airflow-4668] Make airflow/contrib/utils Pylint compatible ( #5916 )
2019-09-04 09:23:57 +01:00
6e53281af6
[AIRFLOW-5399] Add invoke operator for GCP Functions ( #5995 )
2019-09-04 09:07:45 +02:00
fa9e91227b
[AIRFLOW-5386] Move Google Dataproc to core ( #5986 )
...
For more information check AIP-21.
2019-09-03 23:00:38 +02:00
f497d1d5aa
[AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration ( #5495 )
...
1. Issue old conf method deprecation warnings properly and remove current old conf method usages.
2. Unify the way to use conf as `from airflow.configuration import conf`
2019-09-03 17:08:55 +01:00
2ae5256df6
[AIRFLOW-5388] Add airflow version label to newly created buckets ( #5987 )
...
To tag and track GCP resources spawned from Airflow, we have
been adding airflow specific label(s) to GCP API service calls
whenever possible and applicable.
2019-09-02 20:54:55 +02:00
7ee2b99229
[AIRFLOW-5314] Create test for new import paths ( #5920 )
2019-09-01 02:51:28 +02:00
7a59358ffd
[AIRFLOW-5356] Fix GCP Datastore unit tests ( #5961 )
...
This commit adds additional mocking for Datastore tests which
tests fallback_to_default_project_id decorator.
2019-08-30 18:17:28 +02:00
6138208749
[AIRFLOW-5335] Simplify GCSHook test ( #5958 )
2019-08-30 16:50:54 +01:00
bc5c8f32a1
[AIRFLOW-5320] Add system tests for PubSub ( #5925 )
2019-08-30 15:28:12 +02:00
1b69a0b120
[AIRFLOW-5353] Simplify GKEClusterHook ( #5960 )
2019-08-30 14:26:48 +02:00
7c0d9b46ce
[AIRFLOW-5303] Use project_id from GCP credentials ( #5907 )
2019-08-30 01:11:34 +02:00
890adde7bd
[AIRFLOW-5350] Fix bug in the num_retires field in BigQueryHook ( #5955 )
2019-08-29 23:05:10 +01:00
b420ada05b
[AIRFLOW-5351] Move all GCP Cloud SQL tests in 1 file ( #5956 )
2019-08-29 23:04:00 +01:00
5fb9972672
[AIRFLOW-5330] Add project_id to Datastore hook and operators ( #5935 )
2019-08-29 23:04:39 +02:00
8eeedf538c
[AIRFLOW-5346] Add system tests for GKECluster ( #5947 )
2019-08-29 22:54:30 +02:00
88a94cb1bb
[AIRFLOW-5341] Use more precise mock of time.sleep ( #5950 )
...
Some tests were mocking 'time.sleep' instead of using more precise
'airflow.module.path.time.sleep' mock.
2019-08-29 22:52:10 +02:00
3d14a51cdb
[AIRFLOW-4833] Allow to set Jinja env options in DAG declaration ( #5943 )
2019-08-29 20:06:17 +01:00
b1d3d55da9
[AIRFLOW-5335] Update GCSHook methods so they need min IAM perms ( #5939 )
2019-08-29 16:40:33 +01:00
fd544eb0cb
[AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator ( #5821 )
2019-08-28 21:12:30 +02:00
fb0910e847
[AIRFLOW-5275] Add support for template parameters in DataprocWorkflowTemplateInstantiateOperator ( #5877 )
2019-08-28 21:06:30 +02:00
382b21e234
[AIRFLOW-5322] Fix flaky test - GCP Transfer Service hook ( #5931 )
2019-08-28 15:56:38 +02:00
TobKed
Jarek Potiuk
Fokko Driesprong
Kamil Breguła
Tomek
Jacob Ferriero
Kamil Breguła
Mohannad Albanayosy
Diederik Greveling
Tobias Kaymak
davlum
Zacharya
Igor Khrol
Omkar Desai
Hao Liang
Tomek
Kaxil Naik
Géraud
Igor Dralyuk
Michal Brys