incubator-airflow/BREEZE.rst

 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

.. image:: images/AirflowBreeze_logo.png
    :align: center
    :alt: Airflow Breeze Logo

.. contents:: :local:

Airflow Breeze CI Environment
=============================

Airflow Breeze is an easy-to-use development environment using
`Docker Compose <https://docs.docker.com/compose/>`_.
The environment is available for local use and is also used in Airflow's CI tests.

We called it *Airflow Breeze* as **It's a Breeze to develop Airflow**.

The advantages and disadvantages of using the Breeze environment vs. other ways of testing Airflow
are described in `CONTRIBUTING.rst <CONTRIBUTING.rst#integration-test-development-environment>`_.

Here is a short 10-minute video about Airflow Breeze (note that it shows an old version of Breeze. Some
of the points in the video are not valid anymore. The video will be updated shortly with more up-to-date
version):

.. image:: http://img.youtube.com/vi/ffKFHV6f3PQ/0.jpg
   :width: 480px
   :height: 360px
   :scale: 100 %
   :alt: Airflow Breeze Simplified Development Workflow
   :align: center
   :target: http://www.youtube.com/watch?v=ffKFHV6f3PQ

Prerequisites
=============

Docker Community Edition
------------------------

- **Version**: Install the latest stable Docker Community Edition and add it to the PATH.
- **Permissions**: Configure to run the ``docker`` commands directly and not only via root user.
  Your user should be in the ``docker`` group.
  See `Docker installation guide <https://docs.docker.com/install/>`_ for details.
- **Disk space**: On macOS, increase your available disk space before starting to work with
  the environment. At least 128 GB of free disk space is recommended. You can also get by with a
  smaller space but make sure to clean up the Docker disk space periodically.
  See also `Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>`_ for details
  on increasing disk space available for Docker on Mac.
- **Docker problems**: Sometimes it is not obvious that space is an issue when you run into
  a problem with Docker. If you see a weird behaviour, try ``breeze cleanup-image`` command.
  Also see `pruning <https://docs.docker.com/config/pruning/>`_ instructions from Docker.

Here is an example configuration with more than 200GB disk space for Docker:

.. image:: images/disk_space_osx.png
    :align: left
    :alt: Disk space OSX

Docker Compose
--------------

- **Version**: Install the latest stable Docker Compose and add it to the PATH.
  See `Docker Compose Installation Guide <https://docs.docker.com/compose/install/>`_ for details.

- **Permissions**: Configure to run the ``docker-compose`` command.

Docker in WSL
-------------

- **WSL installation** :
    `WSL Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.

- **Docker installation** :
    You should install docker in WSL.
    follow `Docker Installtion Guide <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`_
    only docker-ce without docker-ce-cli containerd.io.
- **Docker setting** :
    You should expose Docker daemon,

.. image:: images/docker_expose_daemon.png
    :align: left
    :alt: Docker expose daemon

and set env variable DOCKER_HOST.

.. code-block:: bash

    echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc

- **WSL problems** :
  There is a mounting problem in docker because docker could not recognize ``/mnt/c``, ``/mnt/d`` driver path.
  run this command in Windows Version 18.03+ and reboot Windows

.. code-block:: bash

    printf '[automount]\nroot = /\n options = "metadata"\n' >> /etc/wsl.conf

Docker Images Used by Breeze
----------------------------

For all development tasks, unit tests, integration tests, and static code checks, we use the
**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
This Docker image contains a lot of test-related packages (size of ~1GB).
Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v1-10-test-python3.6-ci``).
The image is built using the `<Dockerfile.ci>`_ Dockerfile.

For testing production image, the **Production image** is used and maintained on the DockerHub in the
```apache/airflow`` repository. This Docker image contains only size-optimised Airflow with selected
extras and dependencies. Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>``
(for example, ``apache/airflow:master-python3.6`` or ``apache/airflow:v1-10-test-python3.6``).

More information about the images can be found in `<IMAGES.rst>`_.

By default CI images are used unless ``--production-image`` flag is used.

Before you run tests, enter the environment or run local static checks, the necessary local images should be
pulled and built from Docker Hub. This happens automatically for the test environment but you need to
manually trigger it for static checks as described in `Building the images <#building-the-images>`_
and `Pulling the latest images <#pulling-the-latest-images>`_.
The static checks will fail and inform what to do if the image is not yet built.

Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
time. But for subsequent source code changes, no wait time is expected.
However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
that may take more time though it is highly optimized to only rebuild what is needed.

In most cases, rebuilding an image requires network connectivity (for example, to download new
dependencies). If you work offline and do not want to rebuild the images when needed, you can set the
``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
`Default behaviour for user interaction <#default-behaviour-for-user-interaction>`_ section.

See the `Troubleshooting section <#troubleshooting>`_ for steps you can make to clean the environment.

Getopt and gstat
----------------

* For Linux, run ``apt install util-linux coreutils`` or an equivalent if your system is not Debian-based.
* For macOS, install GNU ``getopt`` and ``gstat`` utilities to get Airflow Breeze running.

  Run ``brew install gnu-getopt coreutils`` and then follow instructions to link the gnu-getopt version to
  become the first on the PATH. Make sure to re-login after you make the suggested changes.

**Examples:**

If you use bash, run this command and re-login:

.. code-block:: bash

    echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.bash_profile
    . ~/.bash_profile


If you use zsh, run this command and re-login:

.. code-block:: bash

    echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zprofile
    . ~/.zprofile


Memory
------

Minimum 4GB RAM is required to run the full Breeze environment.

On macOS, 2GB of RAM are available for your Docker containers by default, but more memory is recommended
(4GB should be comfortable). For details see
`Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_.

Airflow Directory Structure inside Docker
-----------------------------------------

When you are in the CI container, the following directories are used:

.. code-block:: text

  /opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
  /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
      airflow.db - sqlite database in case sqlite is used;
      dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
      logs - logs from Airflow executions;
      unittest.cfg - unit test configuration generated when entering the environment;
      webserver_config.py - webserver configuration generated when running Airflow in the container.

Note that when running in your local environment, the ``/root/airflow/logs`` folder is actually mounted
from your ``logs`` directory in the Airflow sources, so all logs created in the container are automatically
visible in the host as well. Every time you enter the container, the ``logs`` directory is
cleaned so that logs do not accumulate.

When you are in the production container, the following directories are used:

.. code-block:: text

  /opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
  /root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
      airflow.db - sqlite database in case sqlite is used;
      dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
      logs - logs from Airflow executions;
      unittest.cfg - unit test configuration generated when entering the environment;
      webserver_config.py - webserver configuration generated when running Airflow in the container.

Note that when running in your local environment, the ``/root/airflow/logs`` folder is actually mounted
from your ``logs`` directory in the Airflow sources, so all logs created in the container are automatically
visible in the host as well. Every time you enter the container, the ``logs`` directory is
cleaned so that logs do not accumulate.

Using the Airflow Breeze Environment
=====================================

Airflow Breeze is a bash script serving as a "swiss-army-knife" of Airflow testing. Under the
hood it uses other scripts that you can also run manually if you have problem with running the Breeze
environment.

Breeze script allows performing the following tasks:

Manage environments - CI (default) or Production - if ``--production-image`` flag is specified:

    * Build docker images with ``breeze build-image`` command
    * Enter interactive shell when no command are specified (default behaviour)
    * Join running interactive shell with ``breeze exec`` command
    * Start Kind Kubernetes cluster for Kubernetes tests if ``--start-kind-cluster`` flag is specified
    * Stop running interactive environment with ``breeze stop`` command
    * Restart running interactive environment with ``breeze restart`` command
    * Optionally reset database if specified as extra ``--db-reset`` flag
    * Optionally start integrations (separate images) if specified as extra ``--integration`` flags (only CI)

Interact with CI environment:

    * Run test target specified with ``breeze test-target`` command
    * Execute arbitrary command in the test environment with ``breeze execute-command`` command
    * Execute arbitrary docker-compose command with ``breeze docker-compose`` command

Run static checks:

    * Run static checks - either for currently staged change or for all files with
      ``breeze static-check`` or ``breeze static-check-all-files`` command

Build documentation:

    * Build documentation with ``breeze build-docs`` command

Set up local development environment:

    * Setup local virtualenv with ``breeze setup-virtualenv`` command
    * Setup autocomplete for itself with ``breeze setup-autocomplete`` command


Note that the below environment interaction is by default with the CI image. If you want to use production
image for those commands you need to add ``--production-image`` flag.

Note that you also should not run both (CI and production) environments simultaneously, as they are using
the same docker-compose configuration which for example contain the link to the database, port mapping, etc.

Entering Breeze CI environment
------------------------------

You enter the Breeze test environment by running the ``./breeze`` script. You can run it with
the ``help`` command to see the list of available options. See `Breeze Command-Line Interface Reference`_
for details.

.. code-block:: bash

  ./breeze

The First time you run Breeze, it pulls and builds a local version of Docker images.
It pulls the latest Airflow CI images from `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_
and uses them to build your local Docker images. Note that the first run (per python) might take up to 10
minutes on a fast connection to start. Subsequent runs should be much faster.

Once you enter the environment, you are dropped into bash shell of the Airflow container and you can
run tests immediately.

You can `set up autocomplete <#setting-up-autocomplete>`_ for commands and add the
checked-out Airflow repository to your PATH to run Breeze without the ``./`` and from any directory.


When you enter the Breeze environment, automatically an environment file is sourced from
``files/airflow-breeze-config/variables.env``. The ``files`` folder from your local sources is
automatically mounted to the container under ``/files`` path and you can put there any files you want
to make available for the Breeze container.

Launching multiple terminals
----------------------------

Often if you want to run full airflow in the Breeze environment you need to launch multiple terminals and
run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in separate terminals.

This can be achieved either via ``tmux`` or via exec-ing into the running container from the host. Tmux
is installed inside the container and you can launch it with ``tmux`` command. Tmux provides you with the
capability of creating multiple virtual terminals and multiplex between them. More about ``tmux`` can be
found at `tmux github wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts
that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.

Another - slightly easier - way is to exec into Breeze terminal from the host's terminal. Often you can
have multiple terminals in the host (Linux/MacOS/WSL2 on Windows) and you can simply use those terminals
to enter the running container. It's as easy as launching ``breeze exec`` while you already started the
Breeze environment. You will be dropped into bash and environment variables will be read in the same
way as when you enter the environment. You can do it multiple times and open as many terminals as you need.

Stopping Interactive environment
--------------------------------

After starting up, the environment runs in the background and takes precious memory.
You can always stop it via:

.. code-block:: bash

   ./breeze stop

Restarting Breeze environment
-----------------------------

You can also  restart the environment and enter it via:

.. code-block:: bash

   ./breeze restart

Choosing a Breeze Environment
-----------------------------

You can use additional ``breeze`` flags to customize your environment. For example, you can specify a Python
version to use, backend and a container environment for testing. With Breeze, you can recreate the same
environments as we have in matrix builds in the CI.

For example, you can choose to run Python 3.6 tests with MySQL as backend and in the Docker environment as
follows:

.. code-block:: bash

    ./breeze --python 3.6 --backend mysql

The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the
``breeze`` script, it could use the values that were used previously. This way you do not have to specify
them when you run the script. You can delete the ``.build/`` directory in case you want to restore the
default settings.

The defaults when you run the Breeze environment are Python 3.6, Sqlite, and Docker.

Launching Breeze Integrations
-----------------------------

When Breeze starts, it can start additional integrations. Those are additional docker containers
that are started in the same docker-compose command. Those are required by some of the tests
as described in `TESTING.rst <TESTING.rst#airflow-integration-tests>`_.

By default Breeze starts only airflow container without any integration enabled. If you selected
``postgres`` or ``mysql`` backend, the container for the selected backend is also started (but only the one
that is selected). You can start the additional integrations by passing ``--integration`` flag
with appropriate integration name when starting Breeze. You can specify several ``--integration`` flags
to start more than one integration at a time.
Finally you can specify ``--integration all`` to start all integrations.

Once integration is started, it will continue to run until the environment is stopped with
``breeze stop`` command. or restarted via ``breeze restart`` command

Note that running integrations uses significant resources - CPU and memory.

Cleaning the Environment
------------------------

You may need to clean up your Docker environment occasionally. The images are quite big
(1.5GB for both images needed for static code analysis and CI tests) and, if you often rebuild/update
them, you may end up with some unused image data.

To clean up the Docker environment:

1. Stop Breeze with ``./breeze stop``.

2. Run the ``docker system prune`` command.

3. Run ``docker images --all`` and ``docker ps --all`` to verify that your Docker is clean.

   Both commands should return an empty list of images and containers respectively.

If you run into disk space errors, consider pruning your Docker images with the ``docker system prune --all``
command. You may need to restart the Docker Engine before running this command.

In case of disk space errors on macOS, increase the disk space available for Docker. See
`Prerequisites <#prerequisites>`_ for details.

Running Arbitrary Commands in the Breeze Environment
-------------------------------------------------------

To run other commands/executables inside the Breeze Docker-based environment, use the
``./breeze execute-command`` command. To add arguments, specify them
together with the command surrounded with either ``"`` or ``'``, or pass them after ``--`` as extra arguments.

.. code-block:: bash

     ./breeze execute-command "ls -la"

.. code-block:: bash

     ./breeze execute-command ls -- --la


Running Docker Compose Commands
-------------------------------

To run Docker Compose commands (such as ``help``, ``pull``, etc), use the
``docker-compose`` command. To add extra arguments, specify them
after ``--`` as extra arguments.

.. code-block:: bash

     ./breeze docker-compose pull -- --ignore-pull-failures


Mounting Local Sources to Breeze
--------------------------------

Important sources of Airflow are mounted inside the ``airflow`` container that you enter.
This means that you can continue editing your changes on the host in your favourite IDE and have them
visible in the Docker immediately and ready to test without rebuilding images. You can disable mounting
by specifying ``--skip-mounting-local-sources`` flag when running Breeze. In this case you will have sources
embedded in the container and changes to these sources will not be persistent.


After you run Breeze for the first time, you will have empty directory ``files`` in your source code,
which will be mapped to ``/files`` in your Docker container. You can pass there any files you need to
configure and run Docker. They will not be removed between Docker runs.

By default ``/files/dags`` folder is mounted from your local ``<AIRFLOW_SOURCES>/files/dags`` and this is
the directory used by airflow scheduler and webserver to scan dags for. You can use it to test your dags
from local sources in Airflow. If you wish to add local DAGs that can be run by Breeze.

Adding/Modifying Dependencies
-----------------------------

If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or
add javascript dependencies in ``package.json``, you can either add dependencies temporarily for a single
Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files.

Installing Dependencies for a Single Breeze Session
...................................................

You can install dependencies inside the container using ``sudo apt install``, ``pip install`` or
``yarn install`` (in ``airflow/www`` folder) respectively. This is useful if you want to test something
quickly while you are in the container. However, these changes are not retained: they disappear once you
exit the container (except for the node.js dependencies if your sources are mounted to the container).
Therefore, if you want to retain a new dependency, follow the second option described below.

Adding Dependencies Permanently
...............................

You can add dependencies to the ``Dockerfile.ci``, ``setup.py`` or ``package.json`` and rebuild the image.
This should happen automatically if you modify any of these files.
After you exit the container and re-run ``breeze``, Breeze detects changes in dependencies,
asks you to confirm rebuilding the image and proceeds with rebuilding if you confirm (or skip it
if you do not confirm). After rebuilding is done, Breeze drops you to shell. You may also use the
``build-image`` command to only rebuild CI image and not to go into shell.

Changing apt Dependencies in the Dockerfile.ci
..............................................

During development, changing dependencies in ``apt-get`` closer to the top of the ``Dockerfile.ci``
invalidates cache for most of the image. It takes long time for Breeze to rebuild the image.
So, it is a recommended practice to add new dependencies initially closer to the end
of the ``Dockerfile.ci``. This way dependencies will be added incrementally.

Before merge, these dependencies should be moved to the appropriate ``apt-get install`` command,
which is already in the ``Dockerfile.ci``.

Port Forwarding
---------------

When you run Airflow Breeze, the following ports are automatically forwarded:

* 28080 -> forwarded to Airflow webserver -> airflow:8080
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database  -> mysql:3306

You can connect to these ports/databases using:

* Webserver: ``http://127.0.0.1:28080``
* Postgres: ``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow``
* Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``

Start the webserver manually with the ``airflow webserver`` command if you want to connect
to the webserver. You can use ``tmux`` to multiply terminals. You may need to create a user prior to
running the webserver in order to log in. This can be done with the following command:

.. code-block:: bash

    airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname foo --lastname bar

For databases, you need to run ``airflow db reset`` at least once (or run some tests) after you started
Airflow Breeze to get the database/tables created. You can connect to databases with IDE or any other
database client:

.. image:: images/database_view.png
    :align: center
    :alt: Database view

You can change the used host port numbers by setting appropriate environment variables:

* ``WEBSERVER_HOST_PORT``
* ``POSTGRES_HOST_PORT``
* ``MYSQL_HOST_PORT``

If you set these variables, next time when you enter the environment the new ports should be in effect.

Setting Up Autocompletion
-------------------------

The ``breeze`` command comes with a built-in bash/zsh autocomplete option for its options. When you start typing
the command, you can use <TAB> to show all the available switches and get autocompletion on typical
values of parameters that you can use.

You can set up the autocomplete option automatically by running:

.. code-block:: bash

   ./breeze setup-autocomplete

You get the autocompletion working when you re-enter the shell.

Zsh autocompletion is currently limited to only autocomplete options. Bash autocompletion also completes
options values (for example, Python version or static check name).

Setting Defaults for User Interaction
--------------------------------------

Sometimes during the build, you are asked whether to perform an action, skip it, or quit. This happens
when rebuilding or removing an image - actions that take a lot of time and could be potentially destructive.

For automation scripts, you can export one of the three variables to control the default
interaction behaviour:

.. code-block::

  export FORCE_ANSWER_TO_QUESTIONS="yes"

If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are automatically rebuilt when needed.
Images are deleted without asking.

.. code-block::

  export FORCE_ANSWER_TO_QUESTIONS="no"

If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used even if rebuilding is needed.
This is useful when you work offline. Deleting images is aborted.

.. code-block::

  export FORCE_ANSWER_TO_QUESTIONS="quit"

If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is aborted. Deleting images is aborted.

If more than one variable is set, ``yes`` takes precedence over ``no``, which takes precedence over ``quit``.

Building the Documentation
--------------------------

To build documentation in Breeze, use the ``build-docs`` command:

.. code-block:: bash

     ./breeze build-docs

Results of the build can be found in the ``docs/_build`` folder.

Often errors during documentation generation come from the docstrings of auto-api generated classes.
During the docs building auto-api generated files are stored in the ``docs/_api`` folder. This helps you
easily identify the location the problems with documentation originated from.

Using Your Host IDE
===================

You can set up your host IDE (for example, IntelliJ's PyCharm/Idea) to work with Breeze
and benefit from all the features provided by your IDE, such as local and remote debugging,
autocompletion, documentation support, etc.

To use your host IDE with Breeze:

1. Create a local virtual environment as follows:

   ``mkvirtualenv <ENV_NAME> --python=python<VERSION>``

   You can use any of the following wrappers to create and manage your virtual environemnts:
   `pyenv <https://github.com/pyenv/pyenv>`_, `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_,
   or `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_.

   Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6, 3.7)
   and switch between them with the ``workon`` command.

2. Use the ``workon`` command to enter the Breeze environment.

3. Initialize the created local virtualenv:

   ``./breeze initialize-local-virtualenv``

4. Select the virtualenv you created as the project's default virtualenv in your IDE.

Note that you can also use the local virtualenv for Airflow development without Breeze.
This is a lightweight solution that has its own limitations.

More details on using the local virtualenv are available in the `LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`_.

Running static checks in Breeze
===============================

The Breeze environment is also used to run some of the static checks as described in
`STATIC_CODE_CHECKS.rst <STATIC_CODE_CHECKS.rst>`_.


Running Tests in Breeze
=======================

As soon as you enter the Breeze environment, you can run Airflow unit tests via the ``pytest`` command.

For supported CI test suites, types of unit tests, and other tests, see `TESTING.rst <TESTING.rst>`_.

Breeze Command-Line Interface Reference
=======================================

Airflow Breeze Syntax
---------------------

This is the current syntax for  `./breeze <./breeze>`_:

 .. START BREEZE HELP MARKER

.. code-block:: text


  ####################################################################################################

  Usage: breeze [FLAGS] [COMMAND] -- <EXTRA_ARGS>

  By default the script enters IT environment and drops you to bash shell, but you can choose one
  of the commands to run specific actions instead. Add --help after each command to see details:

  Commands without arguments:

    shell                                    [Default] Enters interactive shell in the container
    build-docs                               Builds documentation in the container
    build-image                              Builds CI or Production docker image
    cleanup-image                            Cleans up the container image created
    exec                                     Execs into running breeze container in new terminal
    generate-requirements                    Generates pinned requirements for pip dependencies
    initialize-local-virtualenv              Initializes local virtualenv
    setup-autocomplete                       Sets up autocomplete for breeze
    stop                                     Stops the docker-compose environment
    restart                                  Stops the docker-compose environment including DB cleanup
    toggle-suppress-cheatsheet               Toggles on/off cheatsheet
    toggle-suppress-asciiart                 Toggles on/off asciiart

  Commands with arguments:

    docker-compose                <ARG>      Executes specified docker-compose command
    execute-command               <ARG>      Executes specified command in the container
    static-check                  <ARG>      Performs selected static check for changed files
    static-check-all-files        <ARG>      Performs selected static check for all files
    test-target                   <ARG>      Runs selected test target in the container

  Help commands:

    flags                                    Shows all breeze's flags
    help                                     Shows this help message
    help-all                                 Shows detailed help for all commands and flags

  ####################################################################################################

  Detailed usage

  ####################################################################################################


  Detailed usage for command: shell

  breeze [FLAGS] shell -- <EXTRA_ARGS>

        This is default subcommand if no subcommand is used.

        Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
        workers, interact with the database, run DAGs etc. It is the default command if no command
        is selected. The shell is executed in the container and in case integrations are chosen,
        the integrations will be started as separated docker containers - under the docker-compose
        supervision. Local sources are by default mounted to within the container so you can edit
        them locally and run tests immediately in the container. Several folders ('files', 'dist')
        are also mounted so that you can exchange files between the host and container.

        The 'files/airflow-breeze-config/variables.env' file can contain additional variables
        and setup. This file is automatically sourced when you enter the container. Database
        and webserver ports are forwarded to appropriate database/webserver so that you can
        connect to it from your host environment.

  Flags:

  Run 'breeze flags' to see all applicable flags.


  ####################################################################################################


  Detailed usage for command: build-docs

  breeze [FLAGS] build-docs -- <EXTRA_ARGS>

        Builds Airflow documentation. The documentation is build inside docker container - to
        maintain the same build environment for everyone. Appropriate sources are mapped from
        the host to the container so that latest sources are used. The folders where documentation
        is generated ('docs/build') are also mounted to the container - this way results of
        the documentation build is available in the host.


  ####################################################################################################


  Detailed usage for command: build-image

  breeze [FLAGS] build-image -- <EXTRA_ARGS>

        Builds docker image (CI or production) without entering the container. You can pass
        additional options to this command, such as '--force-build-image',
        '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
        You can also pass '--production-image' flag to build production image rather than CI image.

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  -a, --install-airflow-version <INSTALL_AIRFLOW_VERSION>
          If specified, installs Airflow directly from PIP released version. This happens at
          image building time in production image and at container entering time for CI image. One of:

                 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2 master v1-10-test

  -t, --install-airflow-reference <INSTALL_AIRFLOW_REFERENCE>
          If specified, installs Airflow directly from reference in GitHub. This happens at
          image building time in production image and at container entering time for CI image.

  -I, --production-image
          Use production image for entering the environment and builds (not for tests).

  -F, --force-build-images
          Forces building of the local docker images. The images are rebuilt
          automatically for the first time or when changes are detected in
          package-related files, but you can force it using this flag.

  -P, --force-pull-images
          Forces pulling of images from DockerHub before building to populate cache. The
          images are pulled by default only for the first time you run the
          environment, later the locally build images are used as cache.

  -E, --extras
          Extras to pass to build images The default are different for CI and production images:

          CI image:
                 devel_ci

          Production image:
                 async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                 ssh,statsd,virtualenv

  -C, --force-clean-images
          Force build images with cache disabled. This will remove the pulled or build images
          and start building images from scratch. This might take a long time.

  -L, --use-local-cache
          Uses local cache to build images. No pulled images will be used, but results of local
          builds in the Docker cache are used instead.

  -u, --push-images
          After building - uploads the images to DockerHub
          It is useful in case you use your own DockerHub user to store images and you want
          to build them locally. Note that you need to use 'docker login' before you upload images.

  -D, --dockerhub-user
          DockerHub user used to pull, push and build images. Default: apache.

  -H, --dockerhub-repo
          DockerHub repository used to pull, push, build images. Default: airflow.

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.


  ####################################################################################################


  Detailed usage for command: cleanup-image

  breeze [FLAGS] cleanup-image -- <EXTRA_ARGS>

        Removes the breeze-related images created in your local docker image cache. This will
        not reclaim space in docker cache. You need to 'docker system prune' (optionally
        with --all) to reclaim that space.

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  -I, --production-image
          Use production image for entering the environment and builds (not for tests).

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.


  ####################################################################################################


  Detailed usage for command: exec

  breeze [FLAGS] exec -- <EXTRA_ARGS>

        Execs into interactive shell to an already running container. The container mus be started
        already by breeze shell command. If you are not familiar with tmux, this is the best
        way to run multiple processes in the same container at the same time for example scheduler,
        webserver, workers, database console and interactive terminal.


  ####################################################################################################


  Detailed usage for command: generate-requirements

  breeze [FLAGS] generate-requirements -- <EXTRA_ARGS>

        Generates pinned requirements from setup.py. Those requirements are generated in requirements
        directory - separately for different python version. Those requirements are used to run
        CI builds as well as run repeatable production image builds. You can use those requirements
        to predictably install released Airflow versions. You should run it always after you update
        setup.py.

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.


  ####################################################################################################


  Detailed usage for command: initialize-local-virtualenv

  breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS>

        Initializes locally created virtualenv installing all dependencies of Airflow
        taking into account the frozen requirements from requirements folder.
        This local virtualenv can be used to aid autocompletion and IDE support as
        well as run unit tests directly from the IDE. You need to have virtualenv
        activated before running this command.

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7


  ####################################################################################################


  Detailed usage for command: setup-autocomplete

  breeze [FLAGS] setup-autocomplete -- <EXTRA_ARGS>

        Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
        shell and when typing breeze command <TAB> will provide autocomplete for
        parameters and values.


  ####################################################################################################


  Detailed usage for command: stop

  breeze [FLAGS] stop -- <EXTRA_ARGS>

        Brings down running docker compose environment. When you start the environment, the docker
        containers will continue running so that startup time is shorter. But they take quite a lot of
        memory and CPU. This command stops all running containers from the environment.


  ####################################################################################################


  Detailed usage for command: restart

  breeze [FLAGS] restart -- <EXTRA_ARGS>

        Restarts running docker compose environment. When you restart the environment, the docker
        containers will be restarted. That includes cleaning up the databases. This is
        especially useful if you switch between different versions of Airflow.

  Flags:

  Run 'breeze flags' to see all applicable flags.


  ####################################################################################################


  Detailed usage for command: toggle-suppress-cheatsheet

  breeze [FLAGS] toggle-suppress-cheatsheet -- <EXTRA_ARGS>

        Toggles on/off cheatsheet displayed before starting bash shell.


  ####################################################################################################


  Detailed usage for command: toggle-suppress-asciiart

  breeze [FLAGS] toggle-suppress-asciiart -- <EXTRA_ARGS>

        Toggles on/off asciiart displayed before starting bash shell.


  ####################################################################################################


  Detailed usage for command: docker-compose

  breeze [FLAGS] docker-compose <DOCKER_COMPOSE_COMMAND> -- <EXTRA_ARGS>

        Run docker-compose command instead of entering the environment. Use 'help' as command
        to see available commands. The <EXTRA_ARGS> passed after -- are treated
        as additional options passed to docker-compose. For example

        'breeze docker-compose pull -- --ignore-pull-failures'

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  -b, --backend <BACKEND>
          Backend to use for tests - it determines which database is used.
          One of:

                 sqlite mysql postgres

          Default: sqlite

  --postgres-version <POSTGRES_VERSION>
          Postgres version used. One of:

                 9.6 10

  --mysql-version <MYSQL_VERSION>
          Mysql version used. One of:

                 5.7 8

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.


  ####################################################################################################


  Detailed usage for command: execute-command

  breeze [FLAGS] execute-command -- <EXTRA_ARGS>

        Run chosen command instead of entering the environment. The command is run using
        'bash -c "<command with args>" if you need to pass arguments to your command, you need
        to pass them together with command surrounded with " or '. Alternatively you can
        pass arguments as <EXTRA_ARGS> passed after --. For example:

        'breeze execute-command "ls -la"' or
        'breeze execute-command ls -- --la'

  Flags:

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  -b, --backend <BACKEND>
          Backend to use for tests - it determines which database is used.
          One of:

                 sqlite mysql postgres

          Default: sqlite

  --postgres-version <POSTGRES_VERSION>
          Postgres version used. One of:

                 9.6 10

  --mysql-version <MYSQL_VERSION>
          Mysql version used. One of:

                 5.7 8

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.


  ####################################################################################################


  Detailed usage for command: static-check

  breeze [FLAGS] static-check <STATIC_CHECK> -- <EXTRA_ARGS>

        Run selected static checks for currently changed files. You should specify static check that
        you would like to run or 'all' to run all checks. One of:

                 all all-but-pylint airflow-config-yaml base-operator bat-tests build
                 build-providers-dependencies check-apache-license check-executables-have-shebangs
                 check-hooks-apply check-integrations check-merge-conflict check-xml
                 consistent-pylint daysago-import-check debug-statements detect-private-key doctoc
                 end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs
                 incorrect-use-of-LoggingMixin insert-license isort lint-dockerfile mixed-line-ending
                 mypy provide-create-sessions pydevd pylint pylint-tests python-no-log-warn
                 rst-backticks setup-order shellcheck stylelint trailing-whitespace
                 update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
                 yamllint

        You can pass extra arguments including options to to the pre-commit framework as
        <EXTRA_ARGS> passed after --. For example:

        'breeze static-check mypy' or
        'breeze static-check mypy -- --files tests/core.py'

        You can see all the options by adding --help EXTRA_ARG:

        'breeze static-check mypy -- --help'


  ####################################################################################################


  Detailed usage for command: static-check-all-files

  breeze [FLAGS] static-check-all-files <STATIC_CHECK> -- <EXTRA_ARGS>

        Run selected static checks for all applicable files. You should specify static check that
        you would like to run or 'all' to run all checks. One of:

                 all all-but-pylint airflow-config-yaml base-operator bat-tests build
                 build-providers-dependencies check-apache-license check-executables-have-shebangs
                 check-hooks-apply check-integrations check-merge-conflict check-xml
                 consistent-pylint daysago-import-check debug-statements detect-private-key doctoc
                 end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs
                 incorrect-use-of-LoggingMixin insert-license isort lint-dockerfile mixed-line-ending
                 mypy provide-create-sessions pydevd pylint pylint-tests python-no-log-warn
                 rst-backticks setup-order shellcheck stylelint trailing-whitespace
                 update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
                 yamllint

        You can pass extra arguments including options to the pre-commit framework as
        <EXTRA_ARGS> passed after --. For example:

        'breeze static-check-all-files mypy' or
        'breeze static-check-all-files mypy -- --verbose'

        You can see all the options by adding --help EXTRA_ARG:

        'breeze static-check-all-files mypy -- --help'


  ####################################################################################################


  Detailed usage for command: test-target

  breeze [FLAGS] test-target <TEST_TARGET> -- <EXTRA_ARGS>

        Run the specified unit test target. There might be multiple
        targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
        as additional options passed to pytest. For example:

        'breeze test-target tests/test_core.py -- --logging-level=DEBUG'

  Flags:

  Run 'breeze flags' to see all applicable flags.


  ####################################################################################################


  Detailed usage for command: flags

  breeze [FLAGS] flags -- <EXTRA_ARGS>

        Explains in detail all the flags that can be used with breeze.


  ####################################################################################################


  Detailed usage for command: help

  breeze [FLAGS] help -- <EXTRA_ARGS>

        Shows this help message.


  ####################################################################################################


  Detailed usage for command: help-all

  breeze [FLAGS] help-all -- <EXTRA_ARGS>

        Shows detailed help for all commands and flags.


  ####################################################################################################


  ####################################################################################################

  Summary of all flags supported by Breeze:

  ****************************************************************************************************
   Choose Airflow variant

  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
          Python version used for the image. This is always major/minor version.
          One of:

                 3.6 3.7

  ****************************************************************************************************
   Choose backend to run for Airflow

  -b, --backend <BACKEND>
          Backend to use for tests - it determines which database is used.
          One of:

                 sqlite mysql postgres

          Default: sqlite

  --postgres-version <POSTGRES_VERSION>
          Postgres version used. One of:

                 9.6 10

  --mysql-version <MYSQL_VERSION>
          Mysql version used. One of:

                 5.7 8

  ****************************************************************************************************
   Enable production image

  -I, --production-image
          Use production image for entering the environment and builds (not for tests).

  ****************************************************************************************************
   Additional actions executed while entering breeze

  -d, --db-reset
          Resets the database at entry to the environment. It will drop all the tables
          and data and recreate the DB from scratch even if 'restart' command was not used.
          Combined with 'restart' command it enters the environment in the state that is
          ready to start Airflow webserver/scheduler/worker. Without the switch, the database
          does not have any tables and you need to run reset db manually.

  -i, --integration <INTEGRATION>
          Integration to start during tests - it determines which integrations are started
          for integration tests. There can be more than one integration started, or all to
          }
          start all integrations. Selected integrations are not saved for future execution.
          One of:

                 cassandra kerberos mongo openldap presto rabbitmq redis

  ****************************************************************************************************
   Manage Kind kubernetes cluster (optional)

  Action for the cluster : only one of the --kind-cluster-* flags can be used at a time:

  -s, --kind-cluster-start
          Starts KinD Kubernetes cluster after entering the environment. The cluster is started using
          Kubernetes Mode selected and Kubernetes version specified via --kubernetes-mode and
          --kubernetes-version flags.

  -x, --kind-cluster-stop
          Stops KinD Kubernetes cluster if one has already been created. By default, if you do not
          stop environment, the Kubernetes cluster created for testing is continuously running and
          when you start Kubernetes testing again it will be reused. You can force deletion and
          recreation of such cluster with this flag.

  -r, --kind-cluster-recreate

          Recreates KinD Kubernetes cluster if one has already been created. By default, if you do
          not stop environment, the Kubernetes cluster created for testing is continuously running
          and when you start Kubernetes testing again it will be reused. You can force deletion and
          recreation of such cluster with this flag.

  Kubernetes mode/version flags:

  -K, --kubernetes-mode <KUBERNETES_MODE>
          Kubernetes mode - only used in case one of --kind-cluster-* commands is used.
          One of:

                 persistent_mode git_mode

          Default: git_mode

  -V, --kubernetes-version <KUBERNETES_VERSION>
          Kubernetes version - only used in case one of --kind-cluster-* commands is used.
          One of:

                 v1.15.3 v1.16.2

          Default: v1.15.3

  ****************************************************************************************************
   Manage mounting local files

  -l, --skip-mounting-local-sources
          Skips mounting local volume with sources - you get exactly what is in the
          docker image rather than your current local sources of Airflow.

  ****************************************************************************************************
   Assume answers to questions

  -y, --assume-yes
          Assume 'yes' answer to all questions.

  -n, --assume-no
          Assume 'no' answer to all questions.

  -q, --assume-quit
          Assume 'quit' answer to all questions.

  ****************************************************************************************************
   Choose different Airflow version to install or run

  -a, --install-airflow-version <INSTALL_AIRFLOW_VERSION>
          If specified, installs Airflow directly from PIP released version. This happens at
          image building time in production image and at container entering time for CI image. One of:

                 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2 master v1-10-test

  -t, --install-airflow-reference <INSTALL_AIRFLOW_REFERENCE>
          If specified, installs Airflow directly from reference in GitHub. This happens at
          image building time in production image and at container entering time for CI image.

  ****************************************************************************************************
   Credentials

  -f, --forward-credentials
          Forwards host credentials to docker container. Use with care as it will make
          your credentials available to everything you install in Docker.

  ****************************************************************************************************
   Flags for building Docker images (both CI and production)

  -F, --force-build-images
          Forces building of the local docker images. The images are rebuilt
          automatically for the first time or when changes are detected in
          package-related files, but you can force it using this flag.

  -P, --force-pull-images
          Forces pulling of images from DockerHub before building to populate cache. The
          images are pulled by default only for the first time you run the
          environment, later the locally build images are used as cache.

  -E, --extras
          Extras to pass to build images The default are different for CI and production images:

          CI image:
                 devel_ci

          Production image:
                 async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                 ssh,statsd,virtualenv

  -C, --force-clean-images
          Force build images with cache disabled. This will remove the pulled or build images
          and start building images from scratch. This might take a long time.

  -L, --use-local-cache
          Uses local cache to build images. No pulled images will be used, but results of local
          builds in the Docker cache are used instead.

  ****************************************************************************************************
   Flags for pushing Docker images (both CI and production)

  -u, --push-images
          After building - uploads the images to DockerHub
          It is useful in case you use your own DockerHub user to store images and you want
          to build them locally. Note that you need to use 'docker login' before you upload images.

  -D, --dockerhub-user
          DockerHub user used to pull, push and build images. Default: apache.

  -H, --dockerhub-repo
          DockerHub repository used to pull, push, build images. Default: airflow.

  ****************************************************************************************************
   Increase verbosity of the scripts

  -v, --verbose
          Show verbose information about executed commands (enabled by default for running test).
          Note that you can further increase verbosity and see all the commands executed by breeze
          by running 'export VERBOSE_COMMANDS="true"' before running breeze.

 .. END BREEZE HELP MARKER

Convenience Scripts
-------------------

Once you run ``./breeze`` you can also execute various actions via generated convenience scripts:

.. code-block::

   Enter the environment          : ./.build/cmd_run
   Run command in the environment : ./.build/cmd_run "[command with args]" [bash options]
   Run tests in the environment   : ./.build/test_run [test-target] [pytest options]
   Run Docker compose command     : ./.build/dc [help/pull/...] [docker-compose options]

Troubleshooting
===============

If you are having problems with the Breeze environment, try the steps below. After each step you
can check whether your problem is fixed.

1. If you are on macOS, check if you have enough disk space for Docker.
2. Restart Breeze with ``./breeze restart``.
3. Delete the ``.build`` directory and run ``./breeze build-image --force-pull-images``.
4. Clean up Docker images via ``breeze cleanup-image`` command.
5. Restart your Docker Engine and try again.
6. Restart your machine and try again.
7. Re-install Docker CE and try again.

In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":

.. code-block::

        export VERBOSE_COMMANDS="true"


Then run the failed command, copy-and-paste the output from your terminal to the
`Airflow Slack <https://apache-airflow-slack.herokuapp.com/>`_  #airflow-breeze channel and
describe your problem.

Fixing File/Directory Ownership
-------------------------------

On Linux, there is a problem with propagating ownership of created files (a known Docker problem). The
files and directories created in the container are not owned by the host user (but by the root user in our
case). This may prevent you from switching branches, for example, if files owned by the root user are
created within your sources. In case you are on a Linux host and have some files in your sources created
y the root user, you can fix the ownership of those files by running this script:

.. code-block::

  ./scripts/ci/ci_fix_ownership.sh