97b6cc7619
Add note in Updating.md about the removel of DagRun.ID_PREFIX ( #8949 )
2020-05-21 12:17:48 +01:00
499493c5c5
[AIRFLOW-6586] Improvements to gcs sensor ( #7197 )
...
* [AIRFLOW-6586] Improvements to gcs sensor
refactors GoogleCloudStorageUploadSessionCompleteSensor to use set instead of number of objects
add poke mode only decorator
assert that poke_mode_only applied to child of BaseSensorOperator
refactor tests
remove assert
[AIRFLOW-6586] Improvements to gcs sensor
refactors GoogleCloudStorageUploadSessionCompleteSensor to use set instead of number of objects
add poke mode only decorator
assert that poke_mode_only applied to child of BaseSensorOperator
remove assert
fix static checks
add back inadvertently remove requirements
pre-commit
fix typo
* gix gcs sensor unit test
* move poke_mode_only to base_sensor_operator module
* add sensor / poke_mode_only docs
* fix ci check add sensor how-to docs
* Update airflow/providers/google/cloud/sensors/gcs.py
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
* Update airflow/sensors/base_sensor_operator.py
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
* Update airflow/sensors/base_sensor_operator.py
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* simplify class decorator
* remove type hint
* add note to UPDATING.md
* remove unecessary declaration of class member
* Fix to kwargs in UPDATING.md
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-05-19 23:14:28 +02:00
8b54919711
Refactor BigQuery hook methods to use python library ( #8631 )
...
* Refactor create_external_table
* Refactor patch_table and add update_table
* Refactor run_grant_dataset_view_access
* Refactor run_upsert_table
* Refactor insert_all
* Refactor delete_table
* Refactor list_rows
* Fix types
* Fix test
* Refactor poll_job_complete
* Refactor cancel_query
* Refactor run_with_configuration
* Refactor run_* methods
* Fix self.project_id issue
* Fixup run_table_upsert
2020-05-12 17:02:33 +02:00
1fb9f0722a
Synchronize extras between airflow and providers ( #8819 )
2020-05-11 19:25:15 +02:00
2ec0130099
[AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream ( #7735 )
...
Previously, tasks that were in SUCCESS or SKIPPED state satisfy the
depends_on_past check, but only tasks that were in the SUCCESS state
satisfy the wait_for_downstream check. The inconsistency in behavior
made the API less intuitive to users.
2020-05-11 15:22:31 +01:00
b37ce294b9
Patch Pool.DEFAULT_POOL_NAME in BaseOperator ( #8587 )
...
Co-authored-by: Vishesh Jain <visheshj@twitter.com>
2020-05-08 14:09:30 +05:30
b185b364df
Reduce response payload size of /dag_stats and /task_stats ( #8633 )
...
Their response format is like {"example_dag_id": [{"state": "success", "dag_id": "example_dag_id"}, ...], ...}
The dag_id is already used as the "key", but still repeatedly appear in each element,
which makes the response payload size unnecessarily bigger
2020-04-30 23:20:01 +02:00
c1fb28230f
Refactor BigQueryHook dataset operations ( #8477 )
...
* Refactor database methods of BigQueryHook
* fixup! Refactor database methods of BigQueryHook
* Apply suggestions from code review
Co-Authored-By: Michał Słowikowski <michalslowikowski00@gmail.com>
* fixup! fixup! Refactor database methods of BigQueryHook
* fixup! fixup! fixup! Refactor database methods of BigQueryHook
* Pass credentials to BQ client
Co-authored-by: Michał Słowikowski <michalslowikowski00@gmail.com>
2020-04-28 14:19:30 +02:00
5a864f0e45
User-friendly error messages when the configuration is incorrect ( #8463 )
...
* Clearer error messages when the configuration is incorrect
2020-04-27 09:37:07 +02:00
57c8c05839
Use python client in BQ hook create_empty_table/dataset and table_exists ( #8377 )
...
* Use python client in BQ hook create_empty_table method
* Refactor table_exists and create_empty_dataset
* Add note in UPDATING
2020-04-22 09:36:19 +02:00
b88ca51a34
Fix typo in UPDATING.md ( #8493 )
2020-04-21 17:42:58 +01:00
297ad30885
Fix Snowflake hook conn id ( #8423 )
...
Do not set manually to ensure the base `DbApiHook` correctly interprets
arguments based on `conn_name_attr` and `default_conn_name`.
2020-04-20 17:33:33 +01:00
1fd9ed3840
Add mypy plugin for decorators. ( #8145 )
...
* Preserve types for decorators that don't change signatures
* Augment types for decorators that change signatures
* Drop redundant type checks
2020-04-13 09:50:14 +02:00
c9c336c3fd
Add Changelog and Updating note for 1.10.10 ( #8235 )
2020-04-10 08:58:58 +01:00
6c79b097d6
[AIRFLOW-5662] Reduce DB queries used to emit pool usage metrics ( #6342 )
...
* [AIRFLOW-5662] fix incorrect naming and batch db call for scheduler metrics
* report used slots value for used slots metrics (was sending occupied_slots)
* move global scheduler metrics calculation and reporting outside of pool loop
* optimize pool slots metrics report into constant time
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
2020-04-09 12:57:02 +08:00
2503857904
Allow setting Airflow Variable values to empty string ('') ( #8021 )
2020-03-31 18:12:17 +01:00
8a02402576
Rename CloudBaseHook to GoogleBaseHook and move it to google.common ( #8011 )
2020-03-31 14:35:55 +02:00
cb0bf4a142
Remove sql like function in base_hook ( #7901 )
2020-03-30 08:49:33 +02:00
f0c65e3210
[AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation ( #7464 )
...
The documentation for the `none_failed` trigger rule
(https://airflow.apache.org/docs/stable/concepts.html#trigger-rules )
describes its behavior as "all parents have not failed (`failed` or
`upstream_failed`) i.e. all parents have succeeded or been skipped."
With that definition in mind, there is no reason that the check for
`none_failed` should ever have to check for skipped upstream tasks or
set the current task to `skipped`. The current behavior causes the rule
to skip if all upstream tasks have skipped, which is more than a little
confusing. This fixes the behavior to be consistent with the documentation.
Co-authored-by: root <jcroteau@liveramp.com>
2020-03-26 21:17:51 +00:00
9546192836
bumping simple-salesforce to 1.0.0 ( #7857 )
2020-03-26 16:52:40 +01:00
4bde99f132
Make airflow/providers pylint compatible ( #7802 )
2020-03-23 10:17:49 +01:00
9575706509
[AIRFLOW-6860] Default ignore_first_depends_on_past to True ( #7610 )
2020-03-20 11:15:24 +00:00
ff3700d2fc
[AIRFLOW-6959] Use NULL as dag.description default value ( #7593 )
2020-03-20 09:00:56 +01:00
d0cf42162b
[AIRFLOW-XXXX] Fix typo in UPDATING.md ( #7751 )
2020-03-17 18:52:08 +00:00
137896f326
[AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op ( #7685 )
...
* [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
* fixup! fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
2020-03-12 08:26:04 +01:00
2ea9278f76
[AIRFLOW-6860] Default ignore_first_depends_on_past to True ( #7490 )
2020-03-02 14:06:40 -08:00
f85cb8709b
[AIRFLOW-6472] Correct short and long option in cli ( #7148 )
...
* [AIRFLOW-6472] Correct short and long option in cli
* Make Airflow cli short options be single character only
* Make Airflow cli long options be kebab-case style
* Add test for airflow/bin
* Update UPDATING.md
2020-03-01 17:06:58 +02:00
37a8f6a910
[AIRFLOW-6939] Executor configuration via import path ( #7563 )
2020-02-27 16:04:06 +01:00
3320e432a1
[AIRFLOW-6817] Lazy-load `airflow.DAG` to keep user-facing API untouched ( #7517 )
...
`from airflow import DAG` is _such_ a common pattern that it's worth
making sure this stays working.
Since we are now on Python 3 we can use PEP-562 to have officially
supported lazy-loading of attributes on a module.
Since the example_dags will be referred to/copied by users I have
updated them all back to import from the airflow module, but have left
any internal uses untouched.
2020-02-24 14:55:52 +00:00
f0a4aa3e8d
[AIRFLOW-XXXX] Fix grammar in UPDATING.md ( #7509 )
2020-02-23 05:51:50 +01:00
4d03e33c11
[AIRFLOW-6817] remove imports from `airflow/__init__.py`, replaced implicit imports with explicit imports, added entry to `UPDATING.MD` - squashed/rebased ( #7456 )
2020-02-22 08:21:19 +01:00
6cd37da0fd
[AIRFLOW-6821] Success callback not called when task marked as success from UI ( #7447 )
2020-02-20 23:52:39 +00:00
4c91cf87d7
[AIRFLOW-6765] Add CLI "airflow dags test" command ( #7426 )
...
* Add DAG return type
* Add airflow dag test CLI command for running a complete DAG with CLI
* Added new command note to updating.md
* Add airflow dags test example to testing.rst
* Fix rst inline code mistake
2020-02-16 17:56:25 +01:00
58c3542ed2
[AIRFLOW-5231] Fix S3Hook.delete_objects method ( #7375 )
2020-02-12 16:44:08 +01:00
9e09fa3c94
[AIRFLOW-XXXX] Add Changelog & Updating.md section for 1.10.9 ( #7385 )
2020-02-08 16:06:15 +05:30
54768c899c
[AIRFLOW-XXXX] Add section for 1.10.8 in Updating.md ( #7384 )
2020-02-08 12:40:07 +05:30
a02265382d
[AIRFLOW-XXXX] Move LatestOnlyOperator change to Master ( #7350 )
2020-02-03 23:21:42 +05:30
486054bc48
[AIRFLOW-6711] Drop plugin support for stat_name_handler ( #7335 )
2020-02-03 09:28:06 +01:00
97a429f9d0
[AIRFLOW-6714] Remove magic comments about UTF-8 ( #7338 )
2020-02-02 22:18:19 +01:00
373c6aa4a2
[AIRFLOW-6682] Move GCP classes to providers package ( #7295 )
...
* [AIP-21] Move gcp.hooks.automl providers.google.cloud.hooks.automl
* [AIP-21] Move gcp.hooks.bigquery providers.google.cloud.hooks.bigquery
* [AIP-21] Move gcp.hooks.bigquery_dts providers.google.cloud.hooks.bigquery_dts
* [AIP-21] Move gcp.hooks.bigtable providers.google.cloud.hooks.bigtable
* [AIP-21] Move gcp.hooks.cloud_build providers.google.cloud.hooks.cloud_build
* [AIP-21] Move gcp.hooks.cloud_memorystore providers.google.cloud.hooks.cloud_memorystore
* [AIP-21] Move gcp.hooks.cloud_sql providers.google.cloud.hooks.cloud_sql
* [AIP-21] Move gcp.hooks.cloud_storage_transfer_service providers.google.cloud.hooks.cloud_storage_transfer_service
* [AIP-21] Move gcp.hooks.compute providers.google.cloud.hooks.compute
* [AIP-21] Move gcp.hooks.dataflow providers.google.cloud.hooks.dataflow
* [AIP-21] Move gcp.hooks.datastore providers.google.cloud.hooks.datastore
* [AIP-21] Move gcp.hooks.dlp providers.google.cloud.hooks.dlp
* [AIP-21] Move gcp.hooks.functions providers.google.cloud.hooks.functions
* [AIP-21] Move gcp.hooks.gcs providers.google.cloud.hooks.gcs
* [AIP-21] Move gcp.hooks.kms providers.google.cloud.hooks.kms
* [AIP-21] Move gcp.hooks.kubernetes_engine providers.google.cloud.hooks.kubernetes_engine
* [AIP-21] Move gcp.hooks.mlengine providers.google.cloud.hooks.mlengine
* [AIP-21] Move gcp.hooks.spanner providers.google.cloud.hooks.spanner
* [AIP-21] Move gcp.hooks.speech_to_text providers.google.cloud.hooks.speech_to_text
* [AIP-21] Move gcp.hooks.text_to_speech providers.google.cloud.hooks.text_to_speech
* [AIP-21] Move gcp.hooks.translate providers.google.cloud.hooks.translate
* [AIP-21] Move gcp.hooks.discovery_api providers.google.cloud.hooks.discovery_api
* [AIP-21] Move gcp.hooks.video_intelligence providers.google.cloud.hooks.video_intelligence
* [AIP-21] Move gcp.sensors.bigquery providers.google.cloud.sensors.bigquery
* [AIP-21] Move gcp.sensors.bigquery_dts providers.google.cloud.sensors.bigquery_dts
* [AIP-21] Move gcp.sensors.bigtable providers.google.cloud.sensors.bigtable
* [AIP-21] Move gcp.sensors.cloud_storage_transfer_service providers.google.cloud.sensors.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.automl providers.google.cloud.operators.automl
* [AIP-21] Move gcp.operators.bigquery providers.google.cloud.operators.bigquery
* [AIP-21] Move gcp.operators.bigquery_dts providers.google.cloud.operators.bigquery_dts
* [AIP-21] Move gcp.operators.bigtable providers.google.cloud.operators.bigtable
* [AIP-21] Move gcp.operators.cloud_build providers.google.cloud.operators.cloud_build
* [AIP-21] Move gcp.operators.cloud_memorystore providers.google.cloud.operators.cloud_memorystore
* [AIP-21] Move gcp.operators.cloud_sql providers.google.cloud.operators.cloud_sql
* [AIP-21] Move gcp.operators.cloud_storage_transfer_service providers.google.cloud.operators.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.compute providers.google.cloud.operators.compute
* [AIP-21] Move gcp.operators.datastore providers.google.cloud.operators.datastore
* [AIP-21] Move gcp.operators.dlp providers.google.cloud.operators.dlp
* [AIP-21] Move gcp.operators.functions providers.google.cloud.operators.functions
* [AIP-21] Move gcp.operators.gcs providers.google.cloud.operators.gcs
* [AIP-21] Move gcp.operators.kubernetes_engine providers.google.cloud.operators.kubernetes_engine
* [AIP-21] Move gcp.operators.mlengine providers.google.cloud.operators.mlengine
* [AIP-21] Move gcp.operators.spanner providers.google.cloud.operators.spanner
* [AIP-21] Move gcp.operators.speech_to_text providers.google.cloud.operators.speech_to_text
* [AIP-21] Move gcp.operators.tasks providers.google.cloud.operators.tasks
* [AIP-21] Move gcp.operators.text_to_speech providers.google.cloud.operators.text_to_speech
* [AIP-21] Move gcp.operators.translate providers.google.cloud.operators.translate
* [AIP-21] Move gcp.operators.translate_speech providers.google.cloud.operators.translate_speech
* [AIP-21] Move gcp.operators.video_intelligence providers.google.cloud.operators.video_intelligence
* [AIP-21] Move operators.adls_to_gcs providers.google.cloud.operators.adls_to_gcs
* [AIP-21] Move operators.bigquery_to_bigquery providers.google.cloud.operators.bigquery_to_bigquery
* [AIP-21] Move operators.bigquery_to_gcs providers.google.cloud.operators.bigquery_to_gcs
* [AIP-21] Move operators.bigquery_to_mysql providers.google.cloud.operators.bigquery_to_mysql
* [AIP-21] Move operators.cassandra_to_gcs providers.google.cloud.operators.cassandra_to_gcs
* [AIP-21] Move operators.gcs_to_bq providers.google.cloud.operators.gcs_to_bigquery
* [AIP-21] Move operators.gcs_to_gcs providers.google.cloud.operators.gcs_to_gcs
* [AIP-21] Move operators.gcs_to_sftp providers.google.cloud.operators.gcs_to_sftp
* [AIP-21] Move operators.local_to_gcs providers.google.cloud.operators.local_to_gcs
* [AIP-21] Move operators.mssql_to_gcs providers.google.cloud.operators.mssql_to_gcs
* [AIP-21] Move operators.mysql_to_gcs providers.google.cloud.operators.mysql_to_gcs
* [AIP-21] Move operators.postgres_to_gcs providers.google.cloud.operators.postgres_to_gcs
* [AIP-21] Move operators.sql_to_gcs providers.google.cloud.operators.sql_to_gcs
* Update docs
* [AIP-21] Move gcp.hooks.base providers.google.cloud.hooks.base
* Move system tests to providers package
* Fix deprecation message
* Remove gcp.{operators,hooks,sensors}
* Move gcp.example_dags to providers.google.cloud.example_dags
* Move gcp.utils to providers.google.cloud.utils
* Remove airflow.gcp
* Fix import paths in tests
* Remove reference to airflow.gcp
* Fix utils path in test_mlengine_utils.py
2020-01-30 21:59:38 +01:00
b568f745a9
[AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator ( #5970 )
...
LatestOnlyOperator is a special case of a BranchOperator, thus it should inherit from it.
This fixes an issue where the skipping behaviour of LatestOnlyOperator is inconsistent with other operators,
by forcefully skipping all downstream tasks recursively ignoring trigger rules.
2020-01-27 14:10:59 +05:30
a3eb58f56b
[AIRFLOW-6473] Show conf in response of dag_state cli command ( #7186 )
2020-01-18 09:54:00 +01:00
8604c43f83
[AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch ( #7183 )
2020-01-15 21:03:56 +00:00
b38766e834
[AIRFLOW-XXXX] Fix typo in UPDATING.md ( #7182 )
2020-01-15 21:39:10 +01:00
5d874911c4
[AIRFLOW-XXXX] Move UPDATING changes into correct versions ( #7166 )
2020-01-15 20:08:49 +00:00
7c8bf94b3d
[AIRFLOW-6534] - BigQuery - move methods from BigQueryBaseCursor to BigQueryHook ( #7131 )
...
* [AIRFLOW-6534] Add backward relation, update tests
* Move create_empty_table method, simplify BigQueryCreateEmptyTableOperator
* Move create_empty_dataset method, simplify BigQueryCreateEmptyDatasetOperator
* Move get_dataset_tables method, simplify BigQueryGetDatasetTablesOperator
* Move delete_dataset method, simplify BigQueryDeleteDatasetOperator
* Move create_external_table method, simplify BigQueryCreateExternalTableOperator
* Move patch_table method
* Move insert_all method
* Move update_dataset method, simplify BigQueryUpdateDatasetOperator
* Move patch_dataset method, simplify BigQueryPatchDatasetOperator
* Move get_dataset_tables_list method
* Move get_datasets_list method
* Move get_dataset method, simplify BigQueryGetDatasetOperator
* Move run_grant_dataset_view_access
* Move run_table_upsert method
* Move run_table_delete method, simplify BigQueryDeleteTableOperator
* Move get_tabledata method, simplify TestBigQueryGetDataOperator
* Move get_schema method
* Move poll_job_complete method
* Move cancel_query method
* Move run_with_configuration method
* Move run_load method
* Move run_copy method
* Move run_extract method
* Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
* fixup! fixup! fixup! Move run_query method, simplify BigQueryExecuteQueryOperator
2020-01-14 09:58:10 +01:00
6b1986ec58
[AIRFLOW-6107] [AIP-21] Rename GCP container operators ( #7154 )
2020-01-14 09:51:36 +01:00
f4d3e5e545
[AIRFLOW-6102] [AIP-21] Rename Dataproc operators ( #7151 )
2020-01-13 16:22:48 +01:00
e7bf8ecb48
[AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors ( #7125 )
...
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules:
* GoogleCloudStorageHook
* GoogleCloudStorageDeleteOperator
* GoogleCloudStorageDownloadOperator
* GoogleCloudStorageListOperator
* GoogleCloudStorageCreateBucketOperator
* GoogleCloudStorageBucketCreateAclEntryOperator
* GoogleCloudStorageObjectCreateAclEntryOperator
* GoogleCloudStorageObjectSensor
* GoogleCloudStorageObjectUpdatedSensor
* GoogleCloudStoragePrefixSensor
* GoogleCloudStorageUploadSessionCompleteSensor
2020-01-13 13:37:15 +01:00
8ad6160950
[AIRFLOW-6111] [AIP-21] Rename GCP spanner operator and hook ( #7004 )
2020-01-08 12:26:49 +01:00
Kaxil Naik
Jacob Ferriero
Tomek Urbaszek
Jarek Potiuk
Teddy Hartanto
vshshjn7
Xiaodong DENG
Kamil Breguła
Daniel Iñigo
Daniel Huang
Joshua Carp
QP Hou
Jiajie Zhong
yuqian90
eladkal
Ping Zhang
Ash Berlin-Taylor
Matt Buell
Saurabh
Bas Harenslak
Cyril Shcherbin
Cedrik Neumann
tooptoop4
Matthew Bowden
TobKed
Michał Słowikowski