This reverts commit d4655c506e as it causes doc test warnings/failures.
This commit is contained in:
Ash Berlin-Taylor
GitHub
Родитель
d4655c506e
Коммит
c159e8e391
|
|
@ -84,7 +84,7 @@ you need to have set up an Airflow development environment (see below). Also
|
|||
install the `doc` extra.
|
||||
|
||||
```
|
||||
pip install -e '.[doc]'
|
||||
pip install -e .[doc]
|
||||
```
|
||||
|
||||
Generate and serve the documentation by running:
|
||||
|
|
@ -116,7 +116,7 @@ There are three ways to setup an Apache Airflow development environment.
|
|||
cd $AIRFLOW_HOME
|
||||
virtualenv env
|
||||
source env/bin/activate
|
||||
pip install -e '.[devel]'
|
||||
pip install -e .[devel]
|
||||
```
|
||||
|
||||
2. Using a Docker container
|
||||
|
|
@ -129,7 +129,7 @@ There are three ways to setup an Apache Airflow development environment.
|
|||
|
||||
# Install Airflow with all the required dependencies,
|
||||
# including the devel which will provide the development tools
|
||||
pip install -e '.[hdfs,hive,druid,devel]'
|
||||
pip install -e ".[hdfs,hive,druid,devel]"
|
||||
|
||||
# Init the database
|
||||
airflow initdb
|
||||
|
|
|
|||
|
|
@ -129,15 +129,15 @@ has changed to follow style.
|
|||
|
||||
If you want to install integration for Microsoft Azure, then instead of
|
||||
```
|
||||
pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'
|
||||
pip install apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]
|
||||
```
|
||||
you should execute `pip install 'apache-airflow[azure]'`
|
||||
you should execute `pip install apache-airflow[azure]`
|
||||
|
||||
If you want to install integration for Amazon Web Services, then instead of
|
||||
`pip install 'apache-airflow[s3,emr]'`, you should execute `pip install 'apache-airflow[aws]'`
|
||||
`pip install apache-airflow[s3,emr]`, you should execute `pip install apache-airflow[aws]`
|
||||
|
||||
If you want to install integration for Google Cloud Platform, then instead of
|
||||
`pip install 'apache-airflow[gcp_api]'`, you should execute `pip install 'apache-airflow[gcp]'`.
|
||||
`pip install apache-airflow[gcp_api]`, you should execute `pip install apache-airflow[gcp]`.
|
||||
The old way will work until the release of Airflow 2.1.
|
||||
|
||||
### Changes in Google Cloud Platform related operators
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ log = LoggingMixin().log
|
|||
|
||||
try:
|
||||
# Kubernetes is optional, so not available in vanilla Airflow
|
||||
# pip install 'apache-airflow[kubernetes]'
|
||||
# pip install apache-airflow[kubernetes]
|
||||
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
|
||||
|
||||
args = {
|
||||
|
|
@ -64,4 +64,4 @@ try:
|
|||
except ImportError as e:
|
||||
log.warn("Could not import KubernetesPodOperator: " + str(e))
|
||||
log.warn("Install kubernetes dependencies with: "
|
||||
" pip install 'apache-airflow[kubernetes]'")
|
||||
" pip install apache-airflow[kubernetes]")
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ def send_email(to, subject, html_content, files=None, dryrun=False, cc=None,
|
|||
|
||||
To use this plugin:
|
||||
0. include sendgrid subpackage as part of your Airflow installation, e.g.,
|
||||
pip install 'apache-airflow[sendgrid]'
|
||||
pip install apache-airflow[sendgrid]
|
||||
1. update [email] backend in airflow.cfg, i.e.,
|
||||
[email]
|
||||
email_backend = airflow.contrib.utils.sendgrid.send_email
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ If ``crypto`` package was not installed initially, it means that your Fernet key
|
|||
|
||||
You can still enable encryption for passwords within connections by following below steps:
|
||||
|
||||
#. Install crypto package ``pip install 'apache-airflow[crypto]'``
|
||||
#. Install crypto package ``pip install apache-airflow[crypto]``
|
||||
#. Generate fernet_key, using this code snippet below. ``fernet_key`` must be a base64-encoded 32-byte key:
|
||||
|
||||
.. code:: python
|
||||
|
|
|
|||
|
|
@ -122,7 +122,7 @@ example:
|
|||
remote_base_log_folder = gs://my-bucket/path/to/logs
|
||||
remote_log_conn_id = MyGCSConn
|
||||
|
||||
#. Install the ``gcp`` package first, like so: ``pip install 'apache-airflow[gcp]'``.
|
||||
#. Install the ``gcp`` package first, like so: ``pip install apache-airflow[gcp]``.
|
||||
#. Make sure a Google Cloud Platform connection hook has been defined in Airflow. The hook should have read and write access to the Google Cloud Storage bucket defined above in ``remote_base_log_folder``.
|
||||
#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
|
||||
#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ You can also install Airflow with support for extra features like ``gcp`` or ``p
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install 'apache-airflow[postgres,gcp]'
|
||||
pip install apache-airflow[postgres,gcp]
|
||||
|
||||
Extra Packages
|
||||
''''''''''''''
|
||||
|
|
@ -50,74 +50,74 @@ Here's the list of the subpackages and what they enable:
|
|||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| subpackage | install command | enables |
|
||||
+=====================+===================================================+======================================================================+
|
||||
| all | ``pip install 'apache-airflow[all]'`` | All Airflow features known to man |
|
||||
| all | ``pip install apache-airflow[all]`` | All Airflow features known to man |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` | All databases integrations |
|
||||
| all_dbs | ``pip install apache-airflow[all_dbs]`` | All databases integrations |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| async | ``pip install 'apache-airflow[async]'`` | Async worker classes for Gunicorn |
|
||||
| async | ``pip install apache-airflow[async]`` | Async worker classes for Gunicorn |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| azure | ``pip install 'apache-airflow[azure]'`` | Microsoft Azure |
|
||||
| azure | ``pip install apache-airflow[azure]`` | Microsoft Azure |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| aws | ``pip install 'apache-airflow[aws]'`` | Amazon Web Services |
|
||||
| aws | ``pip install apache-airflow[aws]`` | Amazon Web Services |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| celery | ``pip install 'apache-airflow[celery]'`` | CeleryExecutor |
|
||||
| celery | ``pip install apache-airflow[celery]`` | CeleryExecutor |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| cloudant | ``pip install 'apache-airflow[cloudant]'`` | Cloudant hook |
|
||||
| cloudant | ``pip install apache-airflow[cloudant]`` | Cloudant hook |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| crypto | ``pip install 'apache-airflow[crypto]'`` | Encrypt connection passwords in metadata db |
|
||||
| crypto | ``pip install apache-airflow[crypto]`` | Encrypt connection passwords in metadata db |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| devel | ``pip install 'apache-airflow[devel]'`` | Minimum dev tools requirements |
|
||||
| devel | ``pip install apache-airflow[devel]`` | Minimum dev tools requirements |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` | Airflow + dependencies on the Hadoop stack |
|
||||
| devel_hadoop | ``pip install apache-airflow[devel_hadoop]`` | Airflow + dependencies on the Hadoop stack |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| druid | ``pip install 'apache-airflow[druid]'`` | Druid related operators & hooks |
|
||||
| druid | ``pip install apache-airflow[druid]`` | Druid related operators & hooks |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| gcp | ``pip install 'apache-airflow[gcp]'`` | Google Cloud Platform |
|
||||
| gcp | ``pip install apache-airflow[gcp]`` | Google Cloud Platform |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| github_enterprise | ``pip install 'apache-airflow[github_enterprise]'`` | GitHub Enterprise auth backend |
|
||||
| github_enterprise | ``pip install apache-airflow[github_enterprise]`` | GitHub Enterprise auth backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
|
||||
| google_auth | ``pip install apache-airflow[google_auth]`` | Google auth backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| hdfs | ``pip install 'apache-airflow[hdfs]'`` | HDFS hooks and operators |
|
||||
| hdfs | ``pip install apache-airflow[hdfs]`` | HDFS hooks and operators |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| hive | ``pip install 'apache-airflow[hive]'`` | All Hive related operators |
|
||||
| hive | ``pip install apache-airflow[hive]`` | All Hive related operators |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| jdbc | ``pip install 'apache-airflow[jdbc]'`` | JDBC hooks and operators |
|
||||
| jdbc | ``pip install apache-airflow[jdbc]`` | JDBC hooks and operators |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized Hadoop |
|
||||
| kerberos | ``pip install apache-airflow[kerberos]`` | Kerberos integration for Kerberized Hadoop |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| kubernetes | ``pip install 'apache-airflow[kubernetes]'`` | Kubernetes Executor and operator |
|
||||
| kubernetes | ``pip install apache-airflow[kubernetes]`` | Kubernetes Executor and operator |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| ldap | ``pip install 'apache-airflow[ldap]'`` | LDAP authentication for users |
|
||||
| ldap | ``pip install apache-airflow[ldap]`` | LDAP authentication for users |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| mssql | ``pip install 'apache-airflow[mssql]'`` | Microsoft SQL Server operators and hook, |
|
||||
| mssql | ``pip install apache-airflow[mssql]`` | Microsoft SQL Server operators and hook, |
|
||||
| | | support as an Airflow backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| mysql | ``pip install 'apache-airflow[mysql]'`` | MySQL operators and hook, support as an Airflow |
|
||||
| mysql | ``pip install apache-airflow[mysql]`` | MySQL operators and hook, support as an Airflow |
|
||||
| | | backend. The version of MySQL server has to be |
|
||||
| | | 5.6.4+. The exact version upper bound depends |
|
||||
| | | on version of ``mysqlclient`` package. For |
|
||||
| | | example, ``mysqlclient`` 1.3.12 can only be |
|
||||
| | | used with MySQL server 5.6.4 through 5.7. |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| password | ``pip install 'apache-airflow[password]'`` | Password authentication for users |
|
||||
| password | ``pip install apache-airflow[password]`` | Password authentication for users |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| postgres | ``pip install 'apache-airflow[postgres]'`` | PostgreSQL operators and hook, support as an |
|
||||
| postgres | ``pip install apache-airflow[postgres]`` | PostgreSQL operators and hook, support as an |
|
||||
| | | Airflow backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| qds | ``pip install 'apache-airflow[qds]'`` | Enable QDS (Qubole Data Service) support |
|
||||
| qds | ``pip install apache-airflow[qds]`` | Enable QDS (Qubole Data Service) support |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| rabbitmq | ``pip install 'apache-airflow[rabbitmq]'`` | RabbitMQ support as a Celery backend |
|
||||
| rabbitmq | ``pip install apache-airflow[rabbitmq]`` | RabbitMQ support as a Celery backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| redis | ``pip install 'apache-airflow[redis]'`` | Redis hooks and sensors |
|
||||
| redis | ``pip install apache-airflow[redis]`` | Redis hooks and sensors |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| samba | ``pip install 'apache-airflow[samba]'`` | :class:`airflow.operators.hive_to_samba_operator.Hive2SambaOperator` |
|
||||
| samba | ``pip install apache-airflow[samba]`` | :class:`airflow.operators.hive_to_samba_operator.Hive2SambaOperator` |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| slack | ``pip install 'apache-airflow[slack]'`` | :class:`airflow.operators.slack_operator.SlackAPIOperator` |
|
||||
| slack | ``pip install apache-airflow[slack]`` | :class:`airflow.operators.slack_operator.SlackAPIOperator` |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| ssh | ``pip install 'apache-airflow[ssh]'`` | SSH hooks and Operator |
|
||||
| ssh | ``pip install apache-airflow[ssh]`` | SSH hooks and Operator |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
| vertica | ``pip install 'apache-airflow[vertica]'`` | Vertica hook support as an Airflow backend |
|
||||
| vertica | ``pip install apache-airflow[vertica]`` | Vertica hook support as an Airflow backend |
|
||||
+---------------------+---------------------------------------------------+----------------------------------------------------------------------+
|
||||
|
||||
Initiating Airflow Database
|
||||
|
|
|
|||
|
|
@ -241,7 +241,7 @@ To use kerberos authentication, you must install Airflow with the `kerberos` ext
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install 'apache-airflow[kerberos]'
|
||||
pip install apache-airflow[kerberos]
|
||||
|
||||
OAuth Authentication
|
||||
--------------------
|
||||
|
|
@ -274,7 +274,7 @@ To use GHE authentication, you must install Airflow with the `github_enterprise`
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install 'apache-airflow[github_enterprise]'
|
||||
pip install apache-airflow[github_enterprise]
|
||||
|
||||
Setting up GHE Authentication
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -322,7 +322,7 @@ To use Google authentication, you must install Airflow with the `google_auth` ex
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install 'apache-airflow[google_auth]'
|
||||
pip install apache-airflow[google_auth]
|
||||
|
||||
Setting up Google Authentication
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче