4bde99f132
Make airflow/providers pylint compatible ( #7802 )
2020-03-23 10:17:49 +01:00
9575706509
[AIRFLOW-6860] Default ignore_first_depends_on_past to True ( #7610 )
2020-03-20 11:15:24 +00:00
ff3700d2fc
[AIRFLOW-6959] Use NULL as dag.description default value ( #7593 )
2020-03-20 09:00:56 +01:00
d0cf42162b
[AIRFLOW-XXXX] Fix typo in UPDATING.md ( #7751 )
2020-03-17 18:52:08 +00:00
137896f326
[AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op ( #7685 )
...
* [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
2020-03-12 08:26:04 +01:00
2ea9278f76
[AIRFLOW-6860] Default ignore_first_depends_on_past to True ( #7490 )
2020-03-02 14:06:40 -08:00
f85cb8709b
[AIRFLOW-6472] Correct short and long option in cli ( #7148 )
...
* [AIRFLOW-6472] Correct short and long option in cli
* Make Airflow cli short options be single character only
* Make Airflow cli long options be kebab-case style
* Add test for airflow/bin
* Update UPDATING.md
2020-03-01 17:06:58 +02:00
37a8f6a910
[AIRFLOW-6939] Executor configuration via import path ( #7563 )
2020-02-27 16:04:06 +01:00
3320e432a1
[AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched ( #7517 )
...
`from airflow import DAG` is _such_ a common pattern that it's worth
making sure this stays working.
Since we are now on Python 3 we can use PEP-562 to have officially
supported lazy-loading of attributes on a module.
Since the example_dags will be referred to/copied by users I have
updated them all back to import from the airflow module, but have left
any internal uses untouched.
2020-02-24 14:55:52 +00:00
f0a4aa3e8d
[AIRFLOW-XXXX] Fix grammar in UPDATING.md ( #7509 )
2020-02-23 05:51:50 +01:00
4d03e33c11
[AIRFLOW-6817] remove imports from `airflow/__init__.py`, replaced implicit imports with explicit imports, added entry to `UPDATING.MD` - squashed/rebased ( #7456 )
2020-02-22 08:21:19 +01:00
6cd37da0fd
[AIRFLOW-6821] Success callback not called when task marked as success from UI ( #7447 )
2020-02-20 23:52:39 +00:00
4c91cf87d7
[AIRFLOW-6765] Add CLI "airflow dags test" command ( #7426 )
...
* Add DAG return type
* Add airflow dag test CLI command for running a complete DAG with CLI
* Added new command note to updating.md
* Add airflow dags test example to testing.rst
* Fix rst inline code mistake
2020-02-16 17:56:25 +01:00
58c3542ed2
[AIRFLOW-5231] Fix S3Hook.delete_objects method ( #7375 )
2020-02-12 16:44:08 +01:00
9e09fa3c94
[AIRFLOW-XXXX] Add Changelog & Updating.md section for 1.10.9 ( #7385 )
2020-02-08 16:06:15 +05:30
54768c899c
[AIRFLOW-XXXX] Add section for 1.10.8 in Updating.md ( #7384 )
2020-02-08 12:40:07 +05:30
a02265382d
[AIRFLOW-XXXX] Move LatestOnlyOperator change to Master ( #7350 )
2020-02-03 23:21:42 +05:30
486054bc48
[AIRFLOW-6711] Drop plugin support for stat_name_handler ( #7335 )
2020-02-03 09:28:06 +01:00
97a429f9d0
[AIRFLOW-6714] Remove magic comments about UTF-8 ( #7338 )
2020-02-02 22:18:19 +01:00
373c6aa4a2
[AIRFLOW-6682] Move GCP classes to providers package ( #7295 )
...
* [AIP-21] Move gcp.hooks.automl providers.google.cloud.hooks.automl
* [AIP-21] Move gcp.hooks.bigquery providers.google.cloud.hooks.bigquery
* [AIP-21] Move gcp.hooks.bigquery_dts providers.google.cloud.hooks.bigquery_dts
* [AIP-21] Move gcp.hooks.bigtable providers.google.cloud.hooks.bigtable
* [AIP-21] Move gcp.hooks.cloud_build providers.google.cloud.hooks.cloud_build
* [AIP-21] Move gcp.hooks.cloud_memorystore providers.google.cloud.hooks.cloud_memorystore
* [AIP-21] Move gcp.hooks.cloud_sql providers.google.cloud.hooks.cloud_sql
* [AIP-21] Move gcp.hooks.cloud_storage_transfer_service providers.google.cloud.hooks.cloud_storage_transfer_service
* [AIP-21] Move gcp.hooks.compute providers.google.cloud.hooks.compute
* [AIP-21] Move gcp.hooks.dataflow providers.google.cloud.hooks.dataflow
* [AIP-21] Move gcp.hooks.datastore providers.google.cloud.hooks.datastore
* [AIP-21] Move gcp.hooks.dlp providers.google.cloud.hooks.dlp
* [AIP-21] Move gcp.hooks.functions providers.google.cloud.hooks.functions
* [AIP-21] Move gcp.hooks.gcs providers.google.cloud.hooks.gcs
* [AIP-21] Move gcp.hooks.kms providers.google.cloud.hooks.kms
* [AIP-21] Move gcp.hooks.kubernetes_engine providers.google.cloud.hooks.kubernetes_engine
* [AIP-21] Move gcp.hooks.mlengine providers.google.cloud.hooks.mlengine
* [AIP-21] Move gcp.hooks.spanner providers.google.cloud.hooks.spanner
* [AIP-21] Move gcp.hooks.speech_to_text providers.google.cloud.hooks.speech_to_text
* [AIP-21] Move gcp.hooks.text_to_speech providers.google.cloud.hooks.text_to_speech
* [AIP-21] Move gcp.hooks.translate providers.google.cloud.hooks.translate
* [AIP-21] Move gcp.hooks.discovery_api providers.google.cloud.hooks.discovery_api
* [AIP-21] Move gcp.hooks.video_intelligence providers.google.cloud.hooks.video_intelligence
* [AIP-21] Move gcp.sensors.bigquery providers.google.cloud.sensors.bigquery
* [AIP-21] Move gcp.sensors.bigquery_dts providers.google.cloud.sensors.bigquery_dts
* [AIP-21] Move gcp.sensors.bigtable providers.google.cloud.sensors.bigtable
* [AIP-21] Move gcp.sensors.cloud_storage_transfer_service providers.google.cloud.sensors.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.automl providers.google.cloud.operators.automl
* [AIP-21] Move gcp.operators.bigquery providers.google.cloud.operators.bigquery
* [AIP-21] Move gcp.operators.bigquery_dts providers.google.cloud.operators.bigquery_dts
* [AIP-21] Move gcp.operators.bigtable providers.google.cloud.operators.bigtable
* [AIP-21] Move gcp.operators.cloud_build providers.google.cloud.operators.cloud_build
* [AIP-21] Move gcp.operators.cloud_memorystore providers.google.cloud.operators.cloud_memorystore
* [AIP-21] Move gcp.operators.cloud_sql providers.google.cloud.operators.cloud_sql
* [AIP-21] Move gcp.operators.cloud_storage_transfer_service providers.google.cloud.operators.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.compute providers.google.cloud.operators.compute
* [AIP-21] Move gcp.operators.datastore providers.google.cloud.operators.datastore
* [AIP-21] Move gcp.operators.dlp providers.google.cloud.operators.dlp
* [AIP-21] Move gcp.operators.functions providers.google.cloud.operators.functions
* [AIP-21] Move gcp.operators.gcs providers.google.cloud.operators.gcs
* [AIP-21] Move gcp.operators.kubernetes_engine providers.google.cloud.operators.kubernetes_engine
* [AIP-21] Move gcp.operators.mlengine providers.google.cloud.operators.mlengine
* [AIP-21] Move gcp.operators.spanner providers.google.cloud.operators.spanner
* [AIP-21] Move gcp.operators.speech_to_text providers.google.cloud.operators.speech_to_text
* [AIP-21] Move gcp.operators.tasks providers.google.cloud.operators.tasks
* [AIP-21] Move gcp.operators.text_to_speech providers.google.cloud.operators.text_to_speech
* [AIP-21] Move gcp.operators.translate providers.google.cloud.operators.translate
* [AIP-21] Move gcp.operators.translate_speech providers.google.cloud.operators.translate_speech
* [AIP-21] Move gcp.operators.video_intelligence providers.google.cloud.operators.video_intelligence
* [AIP-21] Move operators.adls_to_gcs providers.google.cloud.operators.adls_to_gcs
* [AIP-21] Move operators.bigquery_to_bigquery providers.google.cloud.operators.bigquery_to_bigquery
* [AIP-21] Move operators.bigquery_to_gcs providers.google.cloud.operators.bigquery_to_gcs
* [AIP-21] Move operators.bigquery_to_mysql providers.google.cloud.operators.bigquery_to_mysql
* [AIP-21] Move operators.cassandra_to_gcs providers.google.cloud.operators.cassandra_to_gcs
* [AIP-21] Move operators.gcs_to_bq providers.google.cloud.operators.gcs_to_bigquery
* [AIP-21] Move operators.gcs_to_gcs providers.google.cloud.operators.gcs_to_gcs
* [AIP-21] Move operators.gcs_to_sftp providers.google.cloud.operators.gcs_to_sftp
* [AIP-21] Move operators.local_to_gcs providers.google.cloud.operators.local_to_gcs
* [AIP-21] Move operators.mssql_to_gcs providers.google.cloud.operators.mssql_to_gcs
* [AIP-21] Move operators.mysql_to_gcs providers.google.cloud.operators.mysql_to_gcs
* [AIP-21] Move operators.postgres_to_gcs providers.google.cloud.operators.postgres_to_gcs
* [AIP-21] Move operators.sql_to_gcs providers.google.cloud.operators.sql_to_gcs
* Update docs
* [AIP-21] Move gcp.hooks.base providers.google.cloud.hooks.base
* Move system tests to providers package
* Fix deprecation message
* Remove gcp.{operators,hooks,sensors}
* Move gcp.example_dags to providers.google.cloud.example_dags
* Move gcp.utils to providers.google.cloud.utils
* Remove airflow.gcp
* Fix import paths in tests
* Remove reference to airflow.gcp
* Fix utils path in test_mlengine_utils.py
2020-01-30 21:59:38 +01:00
b568f745a9
[AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator ( #5970 )
...
LatestOnlyOperator is a special case of a BranchOperator, thus it should inherit from it.
This fixes an issue where the skipping behaviour of LatestOnlyOperator is inconsistent with other operators,
by forcefully skipping all downstream tasks recursively ignoring trigger rules.
2020-01-27 14:10:59 +05:30
a3eb58f56b
[AIRFLOW-6473] Show conf in response of dag_state cli command ( #7186 )
2020-01-18 09:54:00 +01:00
8604c43f83
[AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch ( #7183 )
2020-01-15 21:03:56 +00:00
b38766e834
[AIRFLOW-XXXX] Fix typo in UPDATING.md ( #7182 )
2020-01-15 21:39:10 +01:00
5d874911c4
[AIRFLOW-XXXX] Move UPDATING changes into correct versions ( #7166 )
2020-01-15 20:08:49 +00:00
7c8bf94b3d
[AIRFLOW-6534] - BigQuery - move methods from BigQueryBaseCursor to BigQueryHook ( #7131 )
...
* [AIRFLOW-6534] Add backward relation, update tests
* Move create_empty_table method, simplify BigQueryCreateEmptyTableOperator
* Move create_empty_dataset method, simplify BigQueryCreateEmptyDatasetOperator
* Move get_dataset_tables method, simplify BigQueryGetDatasetTablesOperator
* Move delete_dataset method, simplify BigQueryDeleteDatasetOperator
* Move create_external_table method, simplify BigQueryCreateExternalTableOperator
* Move patch_table method
* Move insert_all method
* Move update_dataset method, simplify BigQueryUpdateDatasetOperator
* Move patch_dataset method, simplify BigQueryPatchDatasetOperator
* Move get_dataset_tables_list method
* Move get_datasets_list method
* Move get_dataset method, simplify BigQueryGetDatasetOperator
* Move run_grant_dataset_view_access
* Move run_table_upsert method
* Move run_table_delete method, simplify BigQueryDeleteTableOperator
* Move get_tabledata method, simplify TestBigQueryGetDataOperator
* Move get_schema method
* Move poll_job_complete method
* Move cancel_query method
* Move run_with_configuration method
* Move run_load method
* Move run_copy method
* Move run_extract method
* Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
2020-01-14 09:58:10 +01:00
6b1986ec58
[AIRFLOW-6107] [AIP-21] Rename GCP container operators ( #7154 )
2020-01-14 09:51:36 +01:00
f4d3e5e545
[AIRFLOW-6102] [AIP-21] Rename Dataproc operators ( #7151 )
2020-01-13 16:22:48 +01:00
e7bf8ecb48
[AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors ( #7125 )
...
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules:
* GoogleCloudStorageHook
* GoogleCloudStorageDeleteOperator
* GoogleCloudStorageDownloadOperator
* GoogleCloudStorageListOperator
* GoogleCloudStorageCreateBucketOperator
* GoogleCloudStorageBucketCreateAclEntryOperator
* GoogleCloudStorageObjectCreateAclEntryOperator
* GoogleCloudStorageObjectSensor
* GoogleCloudStorageObjectUpdatedSensor
* GoogleCloudStoragePrefixSensor
* GoogleCloudStorageUploadSessionCompleteSensor
2020-01-13 13:37:15 +01:00
8ad6160950
[AIRFLOW-6111] [AIP-21] Rename GCP spanner operator and hook ( #7004 )
2020-01-08 12:26:49 +01:00
4f8592ae8f
[AIRFLOW-6118] [AIP-21] Rename Pubsub operators and hook ( #7046 )
...
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules
* adde deprecation warnings to the contrib modules
* fixed tests
* updated UPDATING.md
2020-01-08 12:08:38 +01:00
78d8fe6944
[AIRFLOW-6245] Add custom waiters for AWS batch jobs ( #6811 )
...
- add AwsBatchWaiters
- the waiters are based on botocore, but not yet
available for AWS batch services
- refactor AwsBatchOperator:
- use an optional waiters object to wait for
batch job status indicators
- split execute into submit_job and monitor_job
- use job_id with an optional init-parameter;
discard jobId and jobName (already has job_name)
- inherit from AwsBatchClient
- add notes to UPDATING.md
- extract class for AwsBatchClient
- move responsibility for batch API calls and
response parsing to this client
- move responsibility for default wait and
polling to this client
- rename BatchProtocol to AwsBatchProtocol [AIP-21]
- test backward compatibility
- add PROTOCOLS to tests/test_core_to_contrib.py
- add notes to UPDATING.md
- split up polling for job status into steps:
- poll for a JobExists
- poll for a JobRunning
- poll for a JobComplete
- use random jitter for wait-polling delays for
high concurrency job polling
- modify the exponential backoff delay for the
existing polling functions
- revise and update unit tests
2020-01-08 11:52:28 +01:00
5cc81d0512
[AIRFLOW-6116] [AIP-21] Rename MLEngine operators ( #7021 )
2020-01-07 14:51:10 +01:00
cf3c4b34f7
[AIRFLOW-6113] [AIP-21] Rename GCP transfer operator ( #7000 )
...
changes in howto refs in cloud_storage_transfer_service.rst
2020-01-07 01:25:34 +01:00
24fa9380cf
[AIRFLOW-5621] - Failure callback is not triggered when marked Failed on UI ( #7025 )
2020-01-06 12:26:37 +01:00
848022abdd
[AIRFLOW-6442] BigQuery hook - standardize handling http exceptions ( #7028 )
2020-01-03 22:25:01 +01:00
20299473f1
[AIRFLOW-6115] [AIP-21] Rename GCP vision operators ( #7020 )
...
ignred unused imports in gcp_vision_operator and fixed test_core_to_contrib
2020-01-03 22:19:49 +01:00
e79d8d2238
[AIRFLOW-6112] [AIP-21] Rename GCP SQL operators and hooks ( #7006 )
...
fixed tests
2020-01-03 09:42:45 +01:00
e9e0203ebc
[AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers ( #6950 )
...
There is a hidden cyclic dependency between baseoperator and helpers module.
It's hidden by local import but it is detected when baseoperator/helpers are
removed from pylint_todo.txt (and it's really there).
The dependency comes from BaseOperator using helpers and two helpers methods
(chain and cross_downstream) using BaseOperator. This can be solved by
converting the chain and cross_downstream methods to be static methods in
BaseOperator class.
2019-12-31 17:39:13 +01:00
235a87d2ee
[AIRFLOW-6108] [AIP-21] Rename GCP DLP operators ( #6966 )
2019-12-31 16:46:00 +01:00
95087af140
[AIRFLOW-6110] [AIP-21] Rename natural_language service ( #6968 )
2019-12-31 10:05:51 +01:00
7176a830fd
[AIRFLOW-6396] Use tempfile.TemporaryDirectory instead of custom one ( #6957 )
2019-12-31 09:55:14 +01:00
3b5b9560b5
[AIRFLOW-XXX] Add tips for writing a note in UPDATIND.md ( #6960 )
2019-12-30 02:32:44 +00:00
8e037f31f6
[AIRFLOW-6318] Change python3 as Dataflow default interpreter ( #6945 )
2019-12-29 12:37:45 +01:00
2ae10e98ed
[AIRFLOW-XXX] Remove trailing whitespaces from UPDATING.md ( #6940 )
2019-12-28 18:53:18 +01:00
046b36fab7
[AIRFLOW-6332] Extract logging options to new section ( #6887 )
...
* [AIRFLOW-6332] Extract logging options to new section
2019-12-25 11:21:28 +01:00
8734dfa220
[AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor ( #6877 )
...
* [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
* fixup! fixup! fixup! fixup! [AIRFLOW-6324] Hide celery commands when executor is not CeleryExecutor
2019-12-24 10:18:26 +01:00
19939c92f8
[AIRFLOW-6105] [AIP-21] Rename Bigtable operators ( #6862 )
2019-12-21 14:17:26 +01:00
f2a703a8a2
[AIRFLOW-6100] [AIP-21] Rename bigquery service ( #6854 )
2019-12-19 15:52:10 +01:00
57f7e49ffb
[AIRFLOW-6101] [AIP-21] Rename dataflow service ( #6852 )
2019-12-19 15:51:50 +01:00
Jiajie Zhong
Kaxil Naik
Ping Zhang
Kamil Breguła
Ash Berlin-Taylor
Matt Buell
Saurabh
Bas Harenslak
Cyril Shcherbin
Cedrik Neumann
tooptoop4
Matthew Bowden
TobKed
Michał Słowikowski
Darren Weber
Bharath Palaksha
Jarek Potiuk
Tomek Urbaszek
Xinbin Huang