Updated taskflow api doc to show dependency with sensor (#13968)

* Updated taskflow api doc to show dependency with sensor
Updated the taskflow api tutorial document to show how to setup a
dependency to a python-based decorated task from a classic
FileSensor task.

(cherry picked from commit df11a1d7dc)
This commit is contained in:
Vikram Koka 2021-01-29 11:30:35 -08:00 коммит произвёл Kaxil Naik
Родитель 9a075be9d1
Коммит c62f118e5a
1 изменённых файлов: 34 добавлений и 0 удалений

Просмотреть файл

@ -171,6 +171,40 @@ is automatically set to true.
Note, If you manually set the ``multiple_outputs`` parameter the inference is disabled and
the parameter value is used.
Adding dependencies to decorated tasks from regular tasks
---------------------------------------------------------
The above tutorial shows how to create dependencies between python-based tasks. However, it is
quite possible while writing a DAG to have some pre-existing tasks such as :class:`~airflow.operators.bash.BashOperator` or :class:`~airflow.sensors.filesystem.FileSensor`
based tasks which need to be run first before a python-based task is run.
Building this dependency is shown in the code below:
.. code-block:: python
@task()
def extract_from_file():
"""
#### Extract from file task
A simple Extract task to get data ready for the rest of the data
pipeline, by reading the data from a file into a pandas dataframe
"""
order_data_file = '/tmp/order_data.csv'
order_data_df = pd.read_csv(order_data_file)
file_task = FileSensor(task_id='check_file', filepath='/tmp/order_data.csv')
order_data = extract_from_file()
file_task >> order_data
In the above code block, a new python-based task is defined as ``extract_from_file`` which
reads the data from a known file location.
In the main DAG, a new ``FileSensor`` task is defined to check for this file. Please note
that this is a Sensor task which waits for the file.
Finally, a dependency between this Sensor task and the python-based task is specified.
What's Next?
------------