2015-06-11 09:09:22 +03:00
|
|
|
Plugins
|
|
|
|
=======
|
|
|
|
|
|
|
|
Airflow has a simple plugin manager built-in that can integrate external
|
2015-06-29 01:57:13 +03:00
|
|
|
features to its core by simply dropping files in your
|
2015-06-11 09:09:22 +03:00
|
|
|
``$AIRFLOW_HOME/plugins`` folder.
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
The python modules in the ``plugins`` folder get imported,
|
|
|
|
and **hooks**, **operators**, **macros**, **executors** and web **views**
|
2015-06-11 09:09:22 +03:00
|
|
|
get integrated to Airflow's main collections and become available for use.
|
|
|
|
|
|
|
|
What for?
|
|
|
|
---------
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
Airflow offers a generic toolbox for working with data. Different
|
2015-06-11 09:09:22 +03:00
|
|
|
organizations have different stacks and different needs. Using Airflow
|
|
|
|
plugins can be a way for companies to customize their Airflow installation
|
|
|
|
to reflect their ecosystem.
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
Plugins can be used as an easy way to write, share and activate new sets of
|
2015-06-11 09:09:22 +03:00
|
|
|
features.
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
There's also a need for a set of more complex applications to interact with
|
2015-06-11 09:09:22 +03:00
|
|
|
different flavors of data and metadata.
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
|
|
* A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /...)
|
|
|
|
* An anomaly detection framework, allowing people to collect metrics, set thresholds and alerts
|
|
|
|
* An auditing tool, helping understand who accesses what
|
|
|
|
* A config-driven SLA monitoring tool, allowing you to set monitored tables and at what time
|
2015-06-29 01:57:13 +03:00
|
|
|
they should land, alert people, and expose visualizations of outages
|
2015-06-11 09:09:22 +03:00
|
|
|
* ...
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
Why build on top of Airflow?
|
|
|
|
----------------------------
|
2015-06-11 09:09:22 +03:00
|
|
|
|
|
|
|
Airflow has many components that can be reused when building an application:
|
|
|
|
|
|
|
|
* A web server you can use to render your views
|
|
|
|
* A metadata database to store your models
|
2015-06-29 01:57:13 +03:00
|
|
|
* Access to your databases, and knowledge of how to connect to them
|
|
|
|
* An array of workers that your application can push workload to
|
2015-06-11 09:09:22 +03:00
|
|
|
* Airflow is deployed, you can just piggy back on it's deployment logistics
|
|
|
|
* Basic charting capabilities, underlying libraries and abstractions
|
|
|
|
|
|
|
|
|
2015-06-18 00:31:05 +03:00
|
|
|
Interface
|
|
|
|
---------
|
|
|
|
|
2015-06-29 01:57:13 +03:00
|
|
|
To create a plugin you will need to derive the
|
2015-06-18 00:31:05 +03:00
|
|
|
``airflow.plugins_manager.AirflowPlugin`` class and reference the objects
|
|
|
|
you want to plug into Airflow. Here's what the class you need to derive
|
2015-06-29 01:57:13 +03:00
|
|
|
looks like:
|
2015-06-18 00:31:05 +03:00
|
|
|
|
|
|
|
|
|
|
|
.. code:: python
|
|
|
|
|
|
|
|
class AirflowPlugin(object):
|
|
|
|
# The name of your plugin (str)
|
|
|
|
name = None
|
|
|
|
# A list of class(es) derived from BaseOperator
|
|
|
|
operators = []
|
|
|
|
# A list of class(es) derived from BaseHook
|
|
|
|
hooks = []
|
|
|
|
# A list of class(es) derived from BaseExecutor
|
|
|
|
executors = []
|
|
|
|
# A list of references to inject into the macros namespace
|
|
|
|
macros = []
|
2015-06-29 01:57:13 +03:00
|
|
|
# A list of objects created from a class derived
|
2015-06-18 00:31:05 +03:00
|
|
|
# from flask_admin.BaseView
|
|
|
|
admin_views = []
|
|
|
|
# A list of Blueprint object created from flask.Blueprint
|
|
|
|
flask_blueprints = []
|
2016-02-25 22:06:59 +03:00
|
|
|
# A list of menu links (flask_admin.base.MenuLink)
|
2015-06-18 00:31:05 +03:00
|
|
|
menu_links = []
|
|
|
|
|
|
|
|
|
2015-06-11 09:09:22 +03:00
|
|
|
Example
|
|
|
|
-------
|
|
|
|
|
2016-02-02 00:25:56 +03:00
|
|
|
The code below defines a plugin that injects a set of dummy object
|
2015-06-29 01:57:13 +03:00
|
|
|
definitions in Airflow.
|
2015-06-11 09:09:22 +03:00
|
|
|
|
|
|
|
.. code:: python
|
2015-06-29 01:57:13 +03:00
|
|
|
|
2015-06-18 00:31:05 +03:00
|
|
|
# This is the class you derive to create a plugin
|
|
|
|
from airflow.plugins_manager import AirflowPlugin
|
|
|
|
|
|
|
|
from flask import Blueprint
|
2016-02-25 22:06:59 +03:00
|
|
|
from flask_admin import BaseView, expose
|
|
|
|
from flask_admin.base import MenuLink
|
2015-06-11 09:09:22 +03:00
|
|
|
|
|
|
|
# Importing base classes that we need to derive
|
|
|
|
from airflow.hooks.base_hook import BaseHook
|
|
|
|
from airflow.models import BaseOperator
|
|
|
|
from airflow.executors.base_executor import BaseExecutor
|
|
|
|
|
|
|
|
# Will show up under airflow.hooks.PluginHook
|
|
|
|
class PluginHook(BaseHook):
|
|
|
|
pass
|
|
|
|
|
|
|
|
# Will show up under airflow.operators.PluginOperator
|
|
|
|
class PluginOperator(BaseOperator):
|
|
|
|
pass
|
|
|
|
|
|
|
|
# Will show up under airflow.executors.PluginExecutor
|
|
|
|
class PluginExecutor(BaseExecutor):
|
|
|
|
pass
|
|
|
|
|
2015-06-18 00:31:05 +03:00
|
|
|
# Creating a flask admin BaseView
|
|
|
|
class TestView(BaseView):
|
2015-06-11 09:09:22 +03:00
|
|
|
@expose('/')
|
2015-06-18 00:31:05 +03:00
|
|
|
def test(self):
|
|
|
|
return self.render("test_plugin/test.html", content="Hello galaxy!")
|
|
|
|
v = TestView(category="Test Plugin", name="Test View")
|
|
|
|
|
|
|
|
# Creating a flask blueprint to intergrate the templates and static folder
|
|
|
|
bp = Blueprint(
|
|
|
|
"test_plugin", __name__,
|
|
|
|
template_folder='templates',
|
|
|
|
static_folder='static',
|
|
|
|
static_url_path='/static/test_plugin')
|
|
|
|
|
|
|
|
|
|
|
|
ml = MenuLink(
|
|
|
|
category='Test Plugin',
|
|
|
|
name='Test Menu Link',
|
|
|
|
url='http://pythonhosted.org/airflow/')
|
|
|
|
|
|
|
|
# Defining the plugin class
|
|
|
|
class AirflowTestPlugin(AirflowPlugin):
|
|
|
|
name = "test_plugin"
|
|
|
|
operators = [PluginOperator]
|
|
|
|
flask_blueprints = [bp]
|
|
|
|
hooks = [PluginHook]
|
|
|
|
executors = [PluginExecutor]
|
|
|
|
admin_views = [v]
|
|
|
|
menu_links = [ml]
|