Граф коммитов

129 Коммитов

Автор SHA1 Сообщение Дата
Jiajie Zhong 0740dafa0a
[AIRFLOW-7066] Use sphinx syntax in concepts.rst () 2020-03-15 13:08:32 +01:00
dstandish e31e9ddd23
[AIRFLOW-5705] Add secrets backend and support for AWS SSM () 2020-03-14 11:48:10 +00:00
Kaxil Naik 137896f326
[AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op ()
* [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator

* fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator

* fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator

* fixup! fixup! fixup! [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Operator
2020-03-12 08:26:04 +01:00
marwan116 f5da1143aa
[AIRFLOW-XXXX] Fix typo from upstream to downstream ()
This is how it used to be:
Each DAG Run will contain a task_1 Task Instance and a task_2 Task instance. Both Task Instances will
have ``execution_date`` equal to the DAG Run's ``execution_date``, and each task_2 will be *upstream* of
(depends on) its task_1.

if task_2 depends on task_1 this means task_2 is set downstream of tastk_1  - wondering if I am missing something here - or misreading it
2020-03-02 18:00:53 +01:00
Ryan Hamilton 2cc8d20fcf
[AIRFLOW-XXXX] Fix typos in docs directory () 2020-02-27 13:33:53 +00:00
Kamil Breguła 1e576f1234
[AIRFLOW-6680] Last changes for AIP-21 ()
* Move postgres tests from test_operators to test_postgres

* Move myqsl tests from test_operators to test_mysql

* Move myqsl tests from test_operators to test_mysql

* Move myqsl_to_hive tests from test_operators

* Move GCP system tests to providers.google.cloud

* Move PythonVirtualenvOperator to test_python

* Move GenericTransfer tests to test_generic_transfer

* Move MSSQL tests to providers.microsoft.mssql.operators

* Remove duplicate test_to_gcs_to_s3

* [AIP-21] Move operators.presto_to_mysql operators.mysql.operators.presto_to_mysql

* Move Cassandra hook tests to test_cassandra

* Improve strcutre of cassandra senssor testss

* Move TestHiveCli to hooks.test_hive
2020-02-02 20:14:49 +01:00
Kamil Breguła 9a04013b0e
[AIRFLOW-6646][AIP-21] Move protocols classes to providers package ()
* [AIP-21] Move contrib.hooks.ftp_hook providers.ftp.hooks.ftp

* [AIP-21] Move contrib.hooks.grpc_hook providers.grpc.hooks.grpc

* [AIP-21] Move contrib.hooks.imap_hook providers.imap.hooks.imap

* [AIP-21] Move contrib.hooks.ssh_hook providers.ssh.hooks.ssh

* [AIP-21] Move contrib.hooks.winrm_hook providers.microsoft.winrm.hooks.winrm

* [AIP-21] Move contrib.operators.grpc_operator providers.grpc.operators.grpc

* [AIP-21] Move contrib.operators.ssh_operator providers.ssh.operators.ssh

* [AIP-21] Move contrib.operators.winrm_operator providers.microsoft.winrm.operators.winrm

* [AIP-21] Move contrib.sensors.imap_attachment_sensor providers.imap.sensors.imap_attachment

* [AIP-21] Move hooks.http_hook providers.http.hooks.http

* [AIP-21] Move hooks.jdbc_hook providers.jdbc.hooks.jdbc

* [AIP-21] Move contrib.sensors.ftp_sensor providers.ftp.sensors.ftp

* [AIP-21] Move operators.email_operator providers.email.operators.email

* [AIP-21] Move operators.http_operator providers.http.operators.http

* [AIP-21] Move operators.jdbc_operator providers.jdbc.operators.jdbc

* [AIP-21] Move sensors.http_sensor providers.http.sensors.http

* Update docs
2020-01-27 14:36:27 +01:00
Cedrik Neumann b568f745a9 [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator ()
LatestOnlyOperator is a special case of a BranchOperator, thus it should inherit from it.
This fixes an issue where the skipping behaviour of LatestOnlyOperator is inconsistent with other operators,
by forcefully skipping all downstream tasks recursively ignoring trigger rules.
2020-01-27 14:10:59 +05:30
Kamil Breguła c42a375e79
[AIRFLOW-6644][AIP-21] Move service classes to providers package ()
* [AIP-21] Move contrib.hooks.cloudant_hook providers.cloudant.hooks.cloudant

* [AIP-21] Move contrib.hooks.databricks_hook providers.databricks.hooks.databricks

* [AIP-21] Move contrib.hooks.datadog_hook providers.datadog.hooks.datadog

* [AIP-21] Move contrib.hooks.dingding_hook providers.dingding.hooks.dingding

* [AIP-21] Move contrib.hooks.discord_webhook_hook providers.ddiscord.hooks.discord_webhook

* [AIP-21] Move contrib.hooks.gdrive_hook providers.google.suite.hooks.drive

* [AIP-21] Move contrib.hooks.jenkins_hook providers.jenking.hooks.jenkins

* [AIP-21] Move contrib.hooks.opsgenie_alert_hook providers.opsgenie.hooks.opsgenie_alert

* [AIP-21] Move contrib.hooks.pagerduty_hook providers.pagerduty.hooks.pagerduty

* [AIP-21] Move contrib.hooks.qubole_check_hook providers.qubole.hooks.qubole_check

* [AIP-21] Move contrib.hooks.qubole_hook providers.qubole.hooks.qubole

* [AIP-21] Move contrib.hooks.salesforce_hook providers.salesforce.hooks.salesforce

* [AIP-21] Move contrib.hooks.segment_hook providers.segment.hooks.segment

* [AIP-21] Move contrib.hooks.snowflake_hook providers.snowflake.hooks.snowflake

* [AIP-21] Move contrib.hooks.vertica_hook providers.vertica.hooks.vertica

* [AIP-21] Move gcp.hooks.gsheets providers.google.suite.hooks.sheets

* [AIP-21] Move hooks.slack_hook providers.slack.hooks.slack

* [AIP-21] Move hooks.zendesk_hook providers.zendesk.hooks.zendesk

* [AIP-21] Move contrib.operators.databricks_operator providers.databricks.operators.databricks

* [AIP-21] Move contrib.operators.dingding_operator providers.dindding.operators.dingding

* [AIP-21] Move contrib.operators.discord_webhook_operator providers.discord.operators.discord_webhook

* [AIP-21] Move contrib.operators.jenkins_job_trigger_operator providers.jenking.operators.jenkins_job_trigger

* [AIP-21] Move contrib.operators.opsgenie_alert_operator providers.opsgenie.operators.opsgenie_alert

* [AIP-21] Move contrib.operators.qubole_check_operator providers.qubole.operators.qubole_check

* [AIP-21] Move contrib.operators.qubole_operator providers.qubole.operators.qubole

* [AIP-21] Move contrib.operators.segment_track_event_operator providers.segment.operators.segment_track_event

* [AIP-21] Move contrib.operators.slack_webhook_operator providers.slack.operators.slack_webhook

* [AIP-21] Move contrib.operators.vertica_operator providers.vertica.operators.vertica

* [AIP-21] Move contrib.sensors.datadog_sensor providers.datadog.sensors.datadog

* [AIP-21] Move contrib.sensors.qubole_sensor providers.qubole.sensors.qubole

* [AIP-21] Move operators.slack_operator providers.slack.operators.slack

* Update docs
2020-01-27 01:33:15 +01:00
Kamil Breguła 059eda05f8
[AIRFLOW-6610] Move software classes to providers package ()
* [AIP-21] Move contrib.hooks.mongo_hook providers.mongo.hooks.mongo

* [AIP-21] Move contrib.hooks.openfaas_hook providers.openfass.hooks.openfaas

* [AIP-21] Move contrib.hooks.redis_hook providers.redis.hooks.redis

* [AIP-21] Move contrib.operators.docker_swarm_operator providers.docker.operators.docker_swarm

* [AIP-21] Move contrib.operators.redis_publish_operator providers.redis.operators.redis_publish

* [AIP-21] Move contrib.operators.kubernetes_pod_operator providers.cncf.kubernetes.operators.kubernetes_pod

* [AIP-21] Move contrib.sensors.bash_sensor sensors.bash

* [AIP-21] Move contrib.sensors.celery_queue_sensor providers.celery.sensors.celery_queue

* [AIP-21] Move contrib.sensors.mongo_sensor providers.mongo.sensors.mongo

* [AIP-21] Move contrib.sensors.python_sensor sensors.python

* [AIP-21] Move contrib.sensors.redis_key_sensor providers.redis.sensors.redis_key

* [AIP-21] Move contrib.sensors.redis_pub_sub_sensor providers.redis.sensors.redis_pub_sub

* [AIP-21] Move hooks.docker_hook providers.docker.hooks.docker

* [AIP-21] Move hooks.mssql_hook providers.microsoft.mssql.hooks.mssql

* [AIP-21] Move hooks.mysql_hook providers.mysql.hooks.mysql

* [AIP-21] Move hooks.oracle_hook providers.oracle.hooks.oracle

* [AIP-21] Move hooks.postgres_hook providers.postgres.hooks.postgres

* [AIP-21] Move hooks.presto_hook providers.presto.hooks.presto

* [AIP-21] Move hooks.samba_hook providers.samba.hooks.samba

* [AIP-21] Move hooks.sqlite_hook providers.sqlite.hooks.sqlite

* [AIP-21] Move operators.bash_operator operators.bash

* [AIP-21] Move operators.docker_operator providers.docker.operators.docker

* [AIP-21] Move operators.mssql_operator providers.microsoft.mssql.operators.mssql

* [AIP-21] Move operators.mysql_operator providers.mssql.operators.mysql

* [AIP-21] Move operators.oracle_operator providers.oracle.operators.oracle

* [AIP-21] Move operators.papermill_operator providers.papermill.operators.papermill

* [AIP-21] Move operators.postgres_operator providers.postgres.operators.postgres

* [AIP-21] Move operators.presto_check_operator providers.presto.operators.presto_check

* [AIP-21] Move operators.python_operator operators.python

* [AIP-21] Move operators.sqlite_operator providers.sqlite.operators.sqlite

* Update docs
2020-01-21 18:00:52 +01:00
Kamil Breguła c319e81cae
[AIRFLOW-6572] Move AWS classes to providers.amazon.aws package ()
* [AIP-21] Move contrib.hooks.aws_glue_catalog_hook airflow.contrib.hooks.aws_glue_catalog_hook

* [AIP-21] Move contrib.hooks.aws_logs_hook airflow.contrib.hooks.aws_logs_hook

* [AIP-21] Move contrib.hooks.emr_hook airflow.contrib.hooks.emr_hook

* [AIP-21] Move contrib.operators.ecs_operator airflow.contrib.operators.ecs_operator

* [AIP-21] Move contrib.operators.emr_add_steps_operator airflow.contrib.operators.emr_add_steps_operator

* [AIP-21] Move contrib.operators.emr_create_job_flow_operator airflow.contrib.operators.emr_create_job_flow_operator

* [AIP-21] Move contrib.operators.emr_terminate_job_flow_operator airflow.contrib.operators.emr_terminate_job_flow_operator

* [AIP-21] Move contrib.operators.s3_copy_object_operator airflow.contrib.operators.s3_copy_object_operator

* [AIP-21] Move contrib.operators.s3_delete_objects_operator airflow.contrib.operators.s3_delete_objects_operator

* [AIP-21] Move contrib.operators.s3_list_operator airflow.contrib.operators.s3_list_operator

* [AIP-21] Move contrib.operators.sagemaker_base_operator airflow.contrib.operators.sagemaker_base_operator

* [AIP-21] Move contrib.operators.sagemaker_endpoint_config_operator airflow.contrib.operators.sagemaker_endpoint_config_operator

* [AIP-21] Move contrib.operators.sagemaker_endpoint_operator airflow.contrib.operators.sagemaker_endpoint_operator

* [AIP-21] Move contrib.operators.sagemaker_model_operator airflow.contrib.operators.sagemaker_model_operator

* [AIP-21] Move contrib.operators.sagemaker_training_operator airflow.contrib.operators.sagemaker_training_operator

* [AIP-21] Move contrib.operators.sagemaker_transform_operator airflow.contrib.operators.sagemaker_transform_operator

* [AIP-21] Move contrib.operators.sagemaker_tuning_operator airflow.contrib.operators.sagemaker_tuning_operator

* [AIP-21] Move contrib.sensors.aws_glue_catalog_partition_sensor airflow.contrib.sensors.aws_glue_catalog_partition_sensor

* [AIP-21] Move contrib.sensors.emr_base_sensor airflow.contrib.sensors.emr_base_sensor

* [AIP-21] Move contrib.sensors.emr_job_flow_sensor airflow.contrib.sensors.emr_job_flow_sensor

* [AIP-21] Move contrib.sensors.emr_step_sensor airflow.contrib.sensors.emr_step_sensor

* [AIP-21] Move contrib.sensors.sagemaker_base_sensor airflow.contrib.sensors.sagemaker_base_sensor

* [AIP-21] Move contrib.sensors.sagemaker_endpoint_sensor airflow.contrib.sensors.sagemaker_endpoint_sensor

* [AIP-21] Move contrib.sensors.sagemaker_training_sensor airflow.contrib.sensors.sagemaker_training_sensor

* [AIP-21] Move contrib.sensors.sagemaker_transform_sensor airflow.contrib.sensors.sagemaker_transform_sensor

* [AIP-21] Move contrib.sensors.sagemaker_tuning_sensor airflow.contrib.sensors.sagemaker_tuning_sensor

* [AIP-21] Move operators.s3_file_transform_operator airflow.operators.s3_file_transform_operator

* [AIP-21] Move sensors.s3_key_sensor airflow.sensors.s3_key_sensor

* [AIP-21] Move sensors.s3_prefix_sensor airflow.sensors.s3_prefix_sensor

* [AIP-21] Move contrib.hooks.sagemaker_hook providers.amazon.aws.hooks.sagemaker
2020-01-17 09:39:04 +01:00
Kamil Breguła 44769dc172 [AIRFLOW-XXXX] Move email configuration from the concept page ()
* [AIRFLOW-XXXX] Move email configration from the concept page

* fixup! [AIRFLOW-XXXX] Move email configration from the concept page
2020-01-16 17:20:32 +01:00
Kamil Breguła 0481b9a957
[AIRFLOW-6539][AIP-21] Move Apache classes to providers.apache package ()
* [AIP-21] Move contrib.hooks.pinot_hook providers.apache.pinot.hooks.pinot

* [AIP-21] Move contrib.hooks.spark_jdbc_hook providers.apache.spark.hooks.spark_jdbc

* [AIP-21] Move contrib.hooks.spark_jdbc_script providers.apache.spark.hooks.spark_jdbc_script

* [AIP-21] Move contrib.hooks.spark_sql_hook providers.apache.spark.hooks.spark_sql

* [AIP-21] Move contrib.hooks.spark_submit_hook providers.apache.spark.hooks.spark_submit

* [AIP-21] Move contrib.hooks.sqoop_hook providers.apache.sqoop.hooks.sqoop

* [AIP-21] Move contrib.operators.druid_operator providers.apache.druid.operators.druid

* [AIP-21] Move contrib.operators.spark_jdbc_operator providers.apache.spark.operators.spark_jdbc

* [AIP-21] Move contrib.operators.spark_sql_operator providers.apache.spark.operators.spark_sql

* [AIP-21] Move contrib.operators.spark_submit_operator providers.apache.spark.operators.spark_submit

* [AIP-21] Move contrib.operators.sqoop_operator providers.apache.sqoop.operators.sqoop

* [AIP-21] Move contrib.sensors.hdfs_sensor providers.apache.hdfs.sensors.hdfs

* [AIP-21] Move hooks.druid_hook providers.apache.druid.hooks.druid

* [AIP-21] Move hooks.hdfs_hook providers.apache.hdfs.hooks.hdfs

* [AIP-21] Move hooks.hive_hooks providers.apache.hive.hooks.hive

* [AIP-21] Move hooks.pig_hook providers.apache.pig.hooks.pig

* [AIP-21] Move hooks.webhdfs_hook providers.apache.hdfs.hooks.webhdfs

* [AIP-21] Move operators.druid_check_operator providers.apache.druid.operators.druid_check

* [AIP-21] Move operators.hive_operator providers.apache.hive.operators.hive

* [AIP-21] Move operators.hive_stats_operator providers.apache.hive.operators.hive_stats

* [AIP-21] Move operators.pig_operator providers.apache.pig.operators.pig

* [AIP-21] Move sensors.hive_partition_sensor providers.apache.hive.sensors.hive_partition

* [AIP-21] Move sensors.metastore_partition_sensor providers.apache.hive.sensors.metastore_partition

* [AIP-21] Move sensors.named_hive_partition_sensor providers.apache.hive.sensors.named_hive_partition

* [AIP-21] Move sensors.web_hdfs_sensor providers.apache.hdfs.sensors.web_hdfs

* Update docs

* [AIP-21] Move sensors.hdfs_sensor providers.apache.hdfs.sensors.hdfs
2020-01-12 15:51:30 +01:00
Gabriel Eckers a676f159d4 [AIRFLOW-XXXX] Fix reference in concepts doc ()
Correcting reference in Concepts -> Cluster Policy doc from airflow_setting.py to airflow_local_settings.py
2020-01-10 21:05:06 +00:00
Kamil Breguła 5fb64b9f2a [AIRFLOW-6479] Update celery command calls () 2020-01-08 17:02:14 +01:00
Loic Benoit 033e8638f7 [AIRFLOW-XXXX] Add missing relation to op4 () 2020-01-06 18:10:53 +01:00
Bjorn Olsen c232b49be7 [AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date () 2020-01-06 08:03:23 +01:00
Bharat Kashyap e7401730a2 [AIRFLOW-XXXX] Update operation chaining documentation () 2020-01-03 13:14:53 +01:00
Jarek Potiuk e9e0203ebc [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers ()
There is a hidden cyclic dependency between baseoperator and helpers module.
It's hidden by local import but it is detected when baseoperator/helpers are
removed from pylint_todo.txt (and it's really there).

The dependency comes from BaseOperator using helpers and two helpers methods
(chain and cross_downstream) using BaseOperator. This can be solved by
converting the chain and cross_downstream methods to be static methods in
BaseOperator class.
2019-12-31 17:39:13 +01:00
Kaxil Naik bd319e8a86
[AIRFLOW-XXXX] Add Documentation for check_slas flag () 2019-12-30 23:08:17 +00:00
Tomek Urbaszek 34292a5eeb [AIRFLOW-6383] Add no trailing-whitespace pre-commit hook () 2019-12-28 19:34:35 +01:00
Jiajie Zhong 65ef8f3f81 [AIRFLOW-6313] Unify example or doc dag owner ()
We already change Airflow DAG default owner to 'airflow'
in https://github.com/apache/airflow/pull/4151 but some
of our example DAGs and docs are still using
owner = 'Airflow', This patch to unify them
2019-12-20 14:09:59 +01:00
pradeepbhadani ebeaef34ba [AIRFLOW-XXX] Add link to XCom section in concepts.rst ()
Add link to XCom section in concepts.rst
2019-12-12 15:13:48 +01:00
kaverisharma09 6882d355b9 [AIRFLOW-XXX] Add task lifecycle diagram to documentation () 2019-12-12 10:07:18 +01:00
Daphna Shezaf 1accd4907e Clarified a grammatically incorrect sentence () 2019-11-26 12:23:08 +01:00
Kartik Khare c7c0a53a96 [AIRFLOW-XXX] GSoD: How to make DAGs production ready () 2019-11-25 15:34:23 +00:00
Kartik Khare c04ee1375b [AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc () 2019-10-26 00:20:19 +01:00
Steve Taggart 6c7c58c5d3 [AIRFLOW-XXX] Improve wording () 2019-10-22 18:21:55 -07:00
Frank Cash dde65a609a [AIRFLOW-XXXX] Include code tags for sample file names () 2019-10-18 00:49:11 +02:00
Kamil Breguła 99a7191e55 [AIRFLOW-XXX] Improve linking in docs () 2019-10-04 20:07:44 +02:00
Kaxil Naik cfd8d605d9
[AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook () 2019-09-21 20:47:02 +01:00
Géraud d567f9ab8d [AIRFLOW-3871] Operators template fields can now render fields inside objects () 2019-09-18 08:44:12 +01:00
Fokko Driesprong dd175fa8db [AIRFLOW-5390] Remove provide context () 2019-09-10 15:17:03 +02:00
Matthew Bowden 891a90665c [AIRFLOW-XXX] Add information about default pool to docs () 2019-09-05 16:54:14 +01:00
Géraud 3d14a51cdb [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration () 2019-08-29 20:06:17 +01:00
Jarek Potiuk e405be0141 [AIRFLOW-5234] Rst files have consistent, auto-added license 2019-08-18 19:51:02 -04:00
Kamil Breguła 84b65d835e
[AIRFLOW-XXX] Group references in one section () 2019-08-14 23:57:50 +02:00
Joshua Carp 30defe130d [AIRFLOW-3998] Use nested commands in cli. () 2019-07-19 08:40:14 +01:00
Gordon Ball 3891be4247 [AIRFLOW-4456] Add sub-classable BaseBranchOperator ()
This adds an abstract BaseBranchOperator, which implementors can use to
create concrete operators for branching workflows fully encapsulated in
a class, rather than needing to pass an external callable (or
self-reference) to BranchPythonOperator. Also adds tests for same.

It also moves the logic of skipping non-selected branches into
airflow.models.SkipMixin:skip_all_except, and modifies
BranchPythonOperator and BaseBranchOperator to use this mixin instead of
implementing the logic themselves.
2019-06-24 16:01:54 +01:00
Jiajie Zhong 75bec88f04 [AIRFLOW-3958] Support list tasks as upstream in chain () 2019-06-21 23:36:32 +05:30
Ghislain Antony Vaillant f1734e57dd [AIRFLOW-XXX] Add missing word in concepts documentation () 2019-06-07 08:47:16 +05:30
Joshua Carp 759b475da0 [AIRFLOW-4720] Allow comments in .airflowignore files. () 2019-06-02 09:18:25 +02:00
Michal Baumgartner dc0b52719e [AIRFLOW-XXX] Fix S3FileTransformOp reference typo () 2019-05-31 23:00:55 +08:00
wderose 9d07369bae [AIRFLOW-4566] Document sla & sla_miss_callback task params () 2019-05-23 17:43:02 -07:00
al ec631954b8 [AIRFLOW-4232] Add `none_skipped` trigger rule ()
Downstream tasks should run as long as their parents are in
`success`, `failed`, or `upstream_failed` states.
2019-04-06 09:17:26 +01:00
Kamil Breguła 99c8a6f125 [AIRFLOW-3971] Add Google Cloud Natural Language operators () 2019-04-05 22:36:50 +01:00
Chao-Han Tsai 44ae78b253 [AIRFLOW-XXX] fix check docs failure on CI () 2019-03-29 09:59:30 +00:00
Kamil Breguła e27950a75c [AIRFLOW-3811] automatic generation of API Reference in docs ()
Co-authored-by: Jarek Potiuk <jarek.potiuk@polidea.com>
2019-03-26 18:06:14 +00:00
Kristian Yrjölä 781a82f638 [AIRFLOW-3997] Extend Variable.get so it can return None when var not found ()
This will not change existing regular functions in the `Variable` class. If
variable `foo` doesn't exist:

```
foo = Variable.get("foo")
-> KeyError
```

For passing `default_var=None` to get, `None` is returned instead:
```
foo = Variable.get("foo", default_var=None)
if foo is None:
    handle_missing_foo()
```
2019-03-13 16:14:28 +00:00
Kamil Breguła 02fc327347 [AIRFLOW-XXX] Split guide for operators to multiple files () 2019-03-06 20:11:35 +01:00