58c3542ed2
[AIRFLOW-5231] Fix S3Hook.delete_objects method ( #7375 )
2020-02-12 16:44:08 +01:00
9e09fa3c94
[AIRFLOW-XXXX] Add Changelog & Updating.md section for 1.10.9 ( #7385 )
2020-02-08 16:06:15 +05:30
54768c899c
[AIRFLOW-XXXX] Add section for 1.10.8 in Updating.md ( #7384 )
2020-02-08 12:40:07 +05:30
a02265382d
[AIRFLOW-XXXX] Move LatestOnlyOperator change to Master ( #7350 )
2020-02-03 23:21:42 +05:30
486054bc48
[AIRFLOW-6711] Drop plugin support for stat_name_handler ( #7335 )
2020-02-03 09:28:06 +01:00
97a429f9d0
[AIRFLOW-6714] Remove magic comments about UTF-8 ( #7338 )
2020-02-02 22:18:19 +01:00
373c6aa4a2
[AIRFLOW-6682] Move GCP classes to providers package ( #7295 )
...
* [AIP-21] Move gcp.hooks.automl providers.google.cloud.hooks.automl
* [AIP-21] Move gcp.hooks.bigquery providers.google.cloud.hooks.bigquery
* [AIP-21] Move gcp.hooks.bigquery_dts providers.google.cloud.hooks.bigquery_dts
* [AIP-21] Move gcp.hooks.bigtable providers.google.cloud.hooks.bigtable
* [AIP-21] Move gcp.hooks.cloud_build providers.google.cloud.hooks.cloud_build
* [AIP-21] Move gcp.hooks.cloud_memorystore providers.google.cloud.hooks.cloud_memorystore
* [AIP-21] Move gcp.hooks.cloud_sql providers.google.cloud.hooks.cloud_sql
* [AIP-21] Move gcp.hooks.cloud_storage_transfer_service providers.google.cloud.hooks.cloud_storage_transfer_service
* [AIP-21] Move gcp.hooks.compute providers.google.cloud.hooks.compute
* [AIP-21] Move gcp.hooks.dataflow providers.google.cloud.hooks.dataflow
* [AIP-21] Move gcp.hooks.datastore providers.google.cloud.hooks.datastore
* [AIP-21] Move gcp.hooks.dlp providers.google.cloud.hooks.dlp
* [AIP-21] Move gcp.hooks.functions providers.google.cloud.hooks.functions
* [AIP-21] Move gcp.hooks.gcs providers.google.cloud.hooks.gcs
* [AIP-21] Move gcp.hooks.kms providers.google.cloud.hooks.kms
* [AIP-21] Move gcp.hooks.kubernetes_engine providers.google.cloud.hooks.kubernetes_engine
* [AIP-21] Move gcp.hooks.mlengine providers.google.cloud.hooks.mlengine
* [AIP-21] Move gcp.hooks.spanner providers.google.cloud.hooks.spanner
* [AIP-21] Move gcp.hooks.speech_to_text providers.google.cloud.hooks.speech_to_text
* [AIP-21] Move gcp.hooks.text_to_speech providers.google.cloud.hooks.text_to_speech
* [AIP-21] Move gcp.hooks.translate providers.google.cloud.hooks.translate
* [AIP-21] Move gcp.hooks.discovery_api providers.google.cloud.hooks.discovery_api
* [AIP-21] Move gcp.hooks.video_intelligence providers.google.cloud.hooks.video_intelligence
* [AIP-21] Move gcp.sensors.bigquery providers.google.cloud.sensors.bigquery
* [AIP-21] Move gcp.sensors.bigquery_dts providers.google.cloud.sensors.bigquery_dts
* [AIP-21] Move gcp.sensors.bigtable providers.google.cloud.sensors.bigtable
* [AIP-21] Move gcp.sensors.cloud_storage_transfer_service providers.google.cloud.sensors.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.automl providers.google.cloud.operators.automl
* [AIP-21] Move gcp.operators.bigquery providers.google.cloud.operators.bigquery
* [AIP-21] Move gcp.operators.bigquery_dts providers.google.cloud.operators.bigquery_dts
* [AIP-21] Move gcp.operators.bigtable providers.google.cloud.operators.bigtable
* [AIP-21] Move gcp.operators.cloud_build providers.google.cloud.operators.cloud_build
* [AIP-21] Move gcp.operators.cloud_memorystore providers.google.cloud.operators.cloud_memorystore
* [AIP-21] Move gcp.operators.cloud_sql providers.google.cloud.operators.cloud_sql
* [AIP-21] Move gcp.operators.cloud_storage_transfer_service providers.google.cloud.operators.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.compute providers.google.cloud.operators.compute
* [AIP-21] Move gcp.operators.datastore providers.google.cloud.operators.datastore
* [AIP-21] Move gcp.operators.dlp providers.google.cloud.operators.dlp
* [AIP-21] Move gcp.operators.functions providers.google.cloud.operators.functions
* [AIP-21] Move gcp.operators.gcs providers.google.cloud.operators.gcs
* [AIP-21] Move gcp.operators.kubernetes_engine providers.google.cloud.operators.kubernetes_engine
* [AIP-21] Move gcp.operators.mlengine providers.google.cloud.operators.mlengine
* [AIP-21] Move gcp.operators.spanner providers.google.cloud.operators.spanner
* [AIP-21] Move gcp.operators.speech_to_text providers.google.cloud.operators.speech_to_text
* [AIP-21] Move gcp.operators.tasks providers.google.cloud.operators.tasks
* [AIP-21] Move gcp.operators.text_to_speech providers.google.cloud.operators.text_to_speech
* [AIP-21] Move gcp.operators.translate providers.google.cloud.operators.translate
* [AIP-21] Move gcp.operators.translate_speech providers.google.cloud.operators.translate_speech
* [AIP-21] Move gcp.operators.video_intelligence providers.google.cloud.operators.video_intelligence
* [AIP-21] Move operators.adls_to_gcs providers.google.cloud.operators.adls_to_gcs
* [AIP-21] Move operators.bigquery_to_bigquery providers.google.cloud.operators.bigquery_to_bigquery
* [AIP-21] Move operators.bigquery_to_gcs providers.google.cloud.operators.bigquery_to_gcs
* [AIP-21] Move operators.bigquery_to_mysql providers.google.cloud.operators.bigquery_to_mysql
* [AIP-21] Move operators.cassandra_to_gcs providers.google.cloud.operators.cassandra_to_gcs
* [AIP-21] Move operators.gcs_to_bq providers.google.cloud.operators.gcs_to_bigquery
* [AIP-21] Move operators.gcs_to_gcs providers.google.cloud.operators.gcs_to_gcs
* [AIP-21] Move operators.gcs_to_sftp providers.google.cloud.operators.gcs_to_sftp
* [AIP-21] Move operators.local_to_gcs providers.google.cloud.operators.local_to_gcs
* [AIP-21] Move operators.mssql_to_gcs providers.google.cloud.operators.mssql_to_gcs
* [AIP-21] Move operators.mysql_to_gcs providers.google.cloud.operators.mysql_to_gcs
* [AIP-21] Move operators.postgres_to_gcs providers.google.cloud.operators.postgres_to_gcs
* [AIP-21] Move operators.sql_to_gcs providers.google.cloud.operators.sql_to_gcs
* Update docs
* [AIP-21] Move gcp.hooks.base providers.google.cloud.hooks.base
* Move system tests to providers package
* Fix deprecation message
* Remove gcp.{operators,hooks,sensors}
* Move gcp.example_dags to providers.google.cloud.example_dags
* Move gcp.utils to providers.google.cloud.utils
* Remove airflow.gcp
* Fix import paths in tests
* Remove reference to airflow.gcp
* Fix utils path in test_mlengine_utils.py
2020-01-30 21:59:38 +01:00
b568f745a9
[AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator ( #5970 )
...
LatestOnlyOperator is a special case of a BranchOperator, thus it should inherit from it.
This fixes an issue where the skipping behaviour of LatestOnlyOperator is inconsistent with other operators,
by forcefully skipping all downstream tasks recursively ignoring trigger rules.
2020-01-27 14:10:59 +05:30
a3eb58f56b
[AIRFLOW-6473] Show conf in response of dag_state cli command ( #7186 )
2020-01-18 09:54:00 +01:00
8604c43f83
[AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch ( #7183 )
2020-01-15 21:03:56 +00:00
b38766e834
[AIRFLOW-XXXX] Fix typo in UPDATING.md ( #7182 )
2020-01-15 21:39:10 +01:00
5d874911c4
[AIRFLOW-XXXX] Move UPDATING changes into correct versions ( #7166 )
2020-01-15 20:08:49 +00:00
7c8bf94b3d
[AIRFLOW-6534] - BigQuery - move methods from BigQueryBaseCursor to BigQueryHook ( #7131 )
...
* [AIRFLOW-6534] Add backward relation, update tests
* Move create_empty_table method, simplify BigQueryCreateEmptyTableOperator
* Move create_empty_dataset method, simplify BigQueryCreateEmptyDatasetOperator
* Move get_dataset_tables method, simplify BigQueryGetDatasetTablesOperator
* Move delete_dataset method, simplify BigQueryDeleteDatasetOperator
* Move create_external_table method, simplify BigQueryCreateExternalTableOperator
* Move patch_table method
* Move insert_all method
* Move update_dataset method, simplify BigQueryUpdateDatasetOperator
* Move patch_dataset method, simplify BigQueryPatchDatasetOperator
* Move get_dataset_tables_list method
* Move get_datasets_list method
* Move get_dataset method, simplify BigQueryGetDatasetOperator
* Move run_grant_dataset_view_access
* Move run_table_upsert method
* Move run_table_delete method, simplify BigQueryDeleteTableOperator
* Move get_tabledata method, simplify TestBigQueryGetDataOperator
* Move get_schema method
* Move poll_job_complete method
* Move cancel_query method
* Move run_with_configuration method
* Move run_load method
* Move run_copy method
* Move run_extract method
* Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
2020-01-14 09:58:10 +01:00
6b1986ec58
[AIRFLOW-6107] [AIP-21] Rename GCP container operators ( #7154 )
2020-01-14 09:51:36 +01:00
f4d3e5e545
[AIRFLOW-6102] [AIP-21] Rename Dataproc operators ( #7151 )
2020-01-13 16:22:48 +01:00
e7bf8ecb48
[AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors ( #7125 )
...
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules:
* GoogleCloudStorageHook
* GoogleCloudStorageDeleteOperator
* GoogleCloudStorageDownloadOperator
* GoogleCloudStorageListOperator
* GoogleCloudStorageCreateBucketOperator
* GoogleCloudStorageBucketCreateAclEntryOperator
* GoogleCloudStorageObjectCreateAclEntryOperator
* GoogleCloudStorageObjectSensor
* GoogleCloudStorageObjectUpdatedSensor
* GoogleCloudStoragePrefixSensor
* GoogleCloudStorageUploadSessionCompleteSensor
2020-01-13 13:37:15 +01:00
8ad6160950
[AIRFLOW-6111] [AIP-21] Rename GCP spanner operator and hook ( #7004 )
2020-01-08 12:26:49 +01:00
4f8592ae8f
[AIRFLOW-6118] [AIP-21] Rename Pubsub operators and hook ( #7046 )
...
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules
* adde deprecation warnings to the contrib modules
* fixed tests
* updated UPDATING.md
2020-01-08 12:08:38 +01:00
78d8fe6944
[AIRFLOW-6245] Add custom waiters for AWS batch jobs ( #6811 )
...
- add AwsBatchWaiters
- the waiters are based on botocore, but not yet
available for AWS batch services
- refactor AwsBatchOperator:
- use an optional waiters object to wait for
batch job status indicators
- split execute into submit_job and monitor_job
- use job_id with an optional init-parameter;
discard jobId and jobName (already has job_name)
- inherit from AwsBatchClient
- add notes to UPDATING.md
- extract class for AwsBatchClient
- move responsibility for batch API calls and
response parsing to this client
- move responsibility for default wait and
polling to this client
- rename BatchProtocol to AwsBatchProtocol [AIP-21]
- test backward compatibility
- add PROTOCOLS to tests/test_core_to_contrib.py
- add notes to UPDATING.md
- split up polling for job status into steps:
- poll for a JobExists
- poll for a JobRunning
- poll for a JobComplete
- use random jitter for wait-polling delays for
high concurrency job polling
- modify the exponential backoff delay for the
existing polling functions
- revise and update unit tests
2020-01-08 11:52:28 +01:00
5cc81d0512
[AIRFLOW-6116] [AIP-21] Rename MLEngine operators ( #7021 )
2020-01-07 14:51:10 +01:00
cf3c4b34f7
[AIRFLOW-6113] [AIP-21] Rename GCP transfer operator ( #7000 )
...
changes in howto refs in cloud_storage_transfer_service.rst
2020-01-07 01:25:34 +01:00
24fa9380cf
[AIRFLOW-5621] - Failure callback is not triggered when marked Failed on UI ( #7025 )
2020-01-06 12:26:37 +01:00
848022abdd
[AIRFLOW-6442] BigQuery hook - standardize handling http exceptions ( #7028 )
2020-01-03 22:25:01 +01:00
20299473f1
[AIRFLOW-6115] [AIP-21] Rename GCP vision operators ( #7020 )
...
ignred unused imports in gcp_vision_operator and fixed test_core_to_contrib
2020-01-03 22:19:49 +01:00
e79d8d2238
[AIRFLOW-6112] [AIP-21] Rename GCP SQL operators and hooks ( #7006 )
...
fixed tests
2020-01-03 09:42:45 +01:00
e9e0203ebc
[AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers ( #6950 )
...
There is a hidden cyclic dependency between baseoperator and helpers module.
It's hidden by local import but it is detected when baseoperator/helpers are
removed from pylint_todo.txt (and it's really there).
The dependency comes from BaseOperator using helpers and two helpers methods
(chain and cross_downstream) using BaseOperator. This can be solved by
converting the chain and cross_downstream methods to be static methods in
BaseOperator class.
2019-12-31 17:39:13 +01:00
235a87d2ee
[AIRFLOW-6108] [AIP-21] Rename GCP DLP operators ( #6966 )
2019-12-31 16:46:00 +01:00
95087af140
[AIRFLOW-6110] [AIP-21] Rename natural_language service ( #6968 )
2019-12-31 10:05:51 +01:00
7176a830fd
[AIRFLOW-6396] Use tempfile.TemporaryDirectory instead of custom one ( #6957 )
2019-12-31 09:55:14 +01:00
3b5b9560b5
[AIRFLOW-XXX] Add tips for writing a note in UPDATIND.md ( #6960 )
2019-12-30 02:32:44 +00:00
8e037f31f6
[AIRFLOW-6318] Change python3 as Dataflow default interpreter ( #6945 )
2019-12-29 12:37:45 +01:00
2ae10e98ed
[AIRFLOW-XXX] Remove trailing whitespaces from UPDATING.md ( #6940 )
2019-12-28 18:53:18 +01:00
046b36fab7
[AIRFLOW-6332] Extract logging options to new section ( #6887 )
...
* [AIRFLOW-6332] Extract logging options to new section
2019-12-25 11:21:28 +01:00
8734dfa220
[AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor ( #6877 )
...
* [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
2019-12-24 10:18:26 +01:00
19939c92f8
[AIRFLOW-6105] [AIP-21] Rename Bigtable operators ( #6862 )
2019-12-21 14:17:26 +01:00
f2a703a8a2
[AIRFLOW-6100] [AIP-21] Rename bigquery service ( #6854 )
2019-12-19 15:52:10 +01:00
57f7e49ffb
[AIRFLOW-6101] [AIP-21] Rename dataflow service ( #6852 )
2019-12-19 15:51:50 +01:00
9b79c3963c
[AIRFLOW-6147] [AIP-21] Rename GoogleCloudStorageToS3Operator ( #6826 )
2019-12-19 15:08:14 +01:00
ce6e616ace
[AIRFLOW-5903] Remove serve_logs command from CLI ( #6843 )
...
* [AIRFLOW-5903] Remove serve_logs command
2019-12-19 14:25:30 +01:00
1d5b850c8b
[AIRFLOW-6126] [AIP-21] Rename GCP Speech operators ( #6827 )
...
Also these modules have been changed GcpTextToSpeechSynthesizeOperator, GCPSpeechToTextHook, GCPTextToSpeechHook
2019-12-19 10:40:53 +01:00
7502cad284
[AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] ( #6764 )
...
- conform to AIP-21
- see https://issues.apache.org/jira/browse/AIRFLOW-4733
- see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
- use airflow.providers.amazon.aws.operators.batch.AwsBatchOperator
- deprecate airflow.contrib.operators.awsbatch_operator.AWSBatchOperator
- fix pylint for airflow/providers/amazon/aws/operators/batch.py
2019-12-17 16:15:47 +00:00
55f4a07456
[AIRFLOW-6170] BranchPythonOperator does not do XCom push of returned value ( #6726 )
...
* [AIRFLOW-6170] BranchPythonOperator XCom push
2019-12-13 23:16:42 +01:00
add3b5cec0
[AIRFLOW-6122] Rename CloudKMS service classes ( #6798 )
2019-12-13 12:01:37 +01:00
1a58880626
[AIRFLOW-6121][API-21] Rename Cloud Build service class ( #6771 )
2019-12-12 09:34:23 +01:00
53aa9753c1
[AIRFLOW-6209] Drop gcp_service_account_keys option ( #6768 )
...
* [AIRFLOW-6209] Drop gcp_service_account_keys option
* fixup! [AIRFLOW-6209] Drop gcp_service_account_keys option
2019-12-10 14:05:37 -08:00
a36cfe049a
[AIRFLOW-6004] Untangle Executors class to avoid cyclic imports ( #6596 )
...
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI
It was not deterministic because pylint usually uses as many processors as
many are available and it splits the list of .py files between the separate
pylint processors - depending on how the split is done, pylint check might
or might not detect it. The cycle is always detected when all files are used.
In order to make it more deterministic, all pylint and mypy errors were resolved
in all executors package and in dag_processor.
At the same time plugins_manager had also been moved out of the executors
and all of the operators/hooks/sensors/macros because it was also causing
cyclic dependencies and it's far easier to untangle those dependencies
in executor when we move the intialisation of all plugins to plugins_manager.
Additionally require_serial is set in pre-commit configuration to
make sure cycle detection is deterministic.
2019-12-03 16:02:20 +01:00
03c870a617
[AIRFLOW-6010] Remove cyclic imports and pylint hacks ( #6601 )
2019-11-26 22:19:45 +00:00
10052e52c2
[AIRFLOW-3682] Use aws_default in EMR related operators ( #4465 )
2019-11-25 20:02:19 +01:00
da086661f7
[AIRFLOW-6026] Use contextlib to redirect stderr and stdout ( #6624 )
2019-11-21 19:57:00 +01:00
a45a209f38
[AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API ( #6380 )
...
No need to require a user to pass in replace_microseconds to the
request body; instead we should use exactly the date that is given.
We will still replace the microseconds on execution_date if none is
passed in (and the param is True, which is the default)
2019-11-20 17:27:28 +00:00
Cyril Shcherbin
Kaxil Naik
Kamil Breguła
Cedrik Neumann
tooptoop4
Matthew Bowden
TobKed
Michał Słowikowski
Darren Weber
Bharath Palaksha
Jarek Potiuk
Tomek Urbaszek
Xinbin Huang
Bjorn Olsen
laakejs
Austin C Roos