incubator-airflow/docs/code.rst

125 строки
4.2 KiB
ReStructuredText
Исходник Обычный вид История

2014-10-13 09:05:34 +04:00
Code / API
==========
Operators
---------
Operators allow for generation of certain types of tasks that become nodes in
the DAG when instantiated. All operators derive from BaseOperator and
inherit many attributes and methods that way. Refer to the BaseOperator
documentation for more details.
2015-04-05 09:27:00 +03:00
There are 3 main types of operators:
- Operators that performs an **action**, or tell another system to
2015-04-05 09:27:00 +03:00
perform an action
- **Transfer** operators move data from one system to another
- **Sensors** are a certain type of operator that will keep running until a
certain criterion is met. Examples include a specific file landing in HDFS or
2015-04-21 17:41:17 +03:00
S3, a partition appearing in Hive, or a specific time of the day. Sensors
2015-04-05 09:27:00 +03:00
are derived from ``BaseSensorOperator`` and run a poke
method at a specified ``poke_interval`` until it returns ``True``.
.. automodule:: airflow.operators
:show-inheritance:
2015-04-21 17:41:17 +03:00
:members:
2015-03-08 01:24:11 +03:00
BashOperator,
2015-06-26 03:48:03 +03:00
BranchPythonOperator,
2015-03-08 01:24:11 +03:00
DummyOperator,
EmailOperator,
ExternalTaskSensor,
HdfsSensor,
Hive2SambaOperator,
HiveOperator,
HivePartitionSensor,
2015-06-03 00:14:11 +03:00
HiveToMySqlTransfer,
2015-07-08 16:15:30 +03:00
SimpleHttpOperator,
HttpSensor,
2015-03-12 02:48:15 +03:00
MySqlOperator,
2015-04-05 09:27:00 +03:00
MySqlToHiveTransfer,
2015-04-21 17:41:17 +03:00
PostgresOperator,
2015-02-03 09:09:49 +03:00
PrestoCheckOperator,
PrestoIntervalCheckOperator,
PrestoValueCheckOperator,
2015-03-08 01:24:11 +03:00
PythonOperator,
2015-06-03 00:14:11 +03:00
S3KeySensor,
2015-04-21 17:41:17 +03:00
S3ToHiveTransfer,
2015-03-08 01:24:11 +03:00
SqlSensor,
SubDagOperator,
TimeSensor
Macros
---------
2015-02-03 09:09:49 +03:00
Here's a list of variables and macros that can be used in templates
Default Variables
'''''''''''''''''
The Airflow engine passes a few variables by default that are accessible
in all templates
================================= ====================================
Variable Description
================================= ====================================
``{{ ds }}`` the execution date as ``YYYY-MM-DD``
``{{ yesterday_ds }}`` yesterday's date as ``YYYY-MM-DD``
``{{ tomorrow_ds }}`` tomorrow's date as ``YYYY-MM-DD``
``{{ ds }}`` the execution date as ``YYYY-MM-DD``
2015-02-03 09:09:49 +03:00
``{{ execution_date }}`` the execution_date, (datateime.datetime)
``{{ dag }}`` the DAG object
``{{ task }}`` the Task object
``{{ macros }}`` a reference to the macros package, described bellow
``{{ task_instance }}`` the task_instance object
``{{ ds_nodash }}`` the execution date as ``YYYYMMDD``
``{{ end_date }}`` same as ``{{ ds }}``
``{{ lastest_date }}`` same as ``{{ ds }}``
``{{ ti }}`` same as ``{{ task_instance }}``
``{{ params }}`` a reference to the user defined params dictionary
``{{ task_instance_key_str }}`` a unique, human readable key to the task instance
formatted ``{dag_id}_{task_id}_{ds}``
================================= ====================================
Note that you can access the object's attributes and methods with simple
2015-04-21 17:41:17 +03:00
dot notation. Here are some examples of what is possible:
2015-02-03 09:09:49 +03:00
``{{ task.owner }}``, ``{{ task.task_id }}``, ``{{ ti.hostname }}``, ...
Refer to the models documentation for more information on the objects'
2015-02-03 09:09:49 +03:00
attributes and methods.
Macros
''''''
These macros live under the ``macros`` namespace in your templates.
.. automodule:: airflow.macros
2015-04-21 17:41:17 +03:00
:show-inheritance:
2015-02-03 09:09:49 +03:00
:members:
.. automodule:: airflow.macros.hive
2015-04-21 17:41:17 +03:00
:show-inheritance:
:members:
.. _models_ref:
2014-10-13 09:05:34 +04:00
Models
------
Models are built on top of th SQLAlchemy ORM Base class, and instances are
2014-10-13 09:05:34 +04:00
persisted in the database.
2015-01-17 02:55:11 +03:00
.. automodule:: airflow.models
2014-10-20 20:40:43 +04:00
:show-inheritance:
2015-01-17 19:26:11 +03:00
:members: DAG, BaseOperator, TaskInstance, DagBag, Connection
Hooks
-----
.. automodule:: airflow.hooks
:show-inheritance:
2015-07-08 16:15:30 +03:00
:members: MySqlHook, PostgresHook, PrestoHook, HiveCliHook, HiveServer2Hook, HiveMetastoreHook, HttpHook, S3Hook, SqliteHook
2014-10-13 09:05:34 +04:00
Executors
---------
2015-04-21 17:41:17 +03:00
Executors are the mechanism by which task instances get run.
2015-01-18 02:27:01 +03:00
2015-01-17 02:55:11 +03:00
.. automodule:: airflow.executors
2015-04-21 17:41:17 +03:00
:show-inheritance:
2015-01-17 19:26:11 +03:00
:members: LocalExecutor, CeleryExecutor, SequentialExecutor