2019-08-27 21:39:36 +03:00
|
|
|
|
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
|
or more contributor license agreements. See the NOTICE file
|
|
|
|
|
distributed with this work for additional information
|
|
|
|
|
regarding copyright ownership. The ASF licenses this file
|
|
|
|
|
to you under the Apache License, Version 2.0 (the
|
|
|
|
|
"License"); you may not use this file except in compliance
|
|
|
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
|
|
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
|
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
|
|
|
software distributed under the License is distributed on an
|
|
|
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
|
|
|
KIND, either express or implied. See the License for the
|
|
|
|
|
specific language governing permissions and limitations
|
|
|
|
|
under the License.
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<img src="images/AirflowBreeze_logo.png"
|
|
|
|
|
alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
|
|
|
|
|
</div>
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-09-05 11:28:48 +03:00
|
|
|
|
.. contents:: :local:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Airflow Breeze CI environment
|
2019-12-04 15:15:02 +03:00
|
|
|
|
=============================
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Airflow Breeze is an easy-to-use development and test environment using
|
2019-10-26 14:31:51 +03:00
|
|
|
|
`Docker Compose <https://docs.docker.com/compose/>`_.
|
2020-01-13 22:47:37 +03:00
|
|
|
|
The environment is available for local use and is also used in Airflow's CI tests.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
We called it *Airflow Breeze* as **It's a Breeze to contribute to Airflow**.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
The advantages and disadvantages of using the Breeze environment vs. other ways of testing Airflow
|
|
|
|
|
are described in `CONTRIBUTING.rst <CONTRIBUTING.rst#integration-test-development-environment>`_.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Watch the video below about Airflow Breeze. It explains the motivation for Breeze
|
|
|
|
|
and screencasts all its uses.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-09-05 11:28:48 +03:00
|
|
|
|
Prerequisites
|
|
|
|
|
=============
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Docker Community Edition
|
|
|
|
|
------------------------
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
- **Version**: Install the latest stable Docker Community Edition and add it to the PATH.
|
|
|
|
|
- **Permissions**: Configure to run the ``docker`` commands directly and not only via root user.
|
|
|
|
|
Your user should be in the ``docker`` group.
|
|
|
|
|
See `Docker installation guide <https://docs.docker.com/install/>`_ for details.
|
|
|
|
|
- **Disk space**: On macOS, increase your available disk space before starting to work with
|
|
|
|
|
the environment. At least 128 GB of free disk space is recommended. You can also get by with a
|
|
|
|
|
smaller space but make sure to clean up the Docker disk space periodically.
|
|
|
|
|
See also `Docker for Mac - Space <https://docs.docker.com/docker-for-mac/space>`_ for details
|
|
|
|
|
on increasing disk space available for Docker on Mac.
|
|
|
|
|
- **Docker problems**: Sometimes it is not obvious that space is an issue when you run into
|
2020-04-02 20:52:11 +03:00
|
|
|
|
a problem with Docker. If you see a weird behaviour, try ``breeze cleanup-image`` command.
|
|
|
|
|
Also see `pruning <https://docs.docker.com/config/pruning/>`_ instructions from Docker.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-01-21 13:44:00 +03:00
|
|
|
|
Here is an example configuration with more than 200GB disk space for Docker:
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<img src="images/disk_space_osx.png" width="640"
|
|
|
|
|
alt="Disk space MacOS">
|
|
|
|
|
</div>
|
2020-01-21 13:44:00 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Docker Compose
|
|
|
|
|
--------------
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
- **Version**: Install the latest stable Docker Compose and add it to the PATH.
|
|
|
|
|
See `Docker Compose Installation Guide <https://docs.docker.com/compose/install/>`_ for details.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
- **Permissions**: Configure permission to run the ``docker-compose`` command.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-01 10:04:31 +03:00
|
|
|
|
Docker in WSL 2
|
|
|
|
|
---------------
|
2020-03-01 19:38:07 +03:00
|
|
|
|
|
2020-06-01 10:04:31 +03:00
|
|
|
|
- **WSL 2 installation** :
|
|
|
|
|
Install WSL 2 and a Linux Distro (e.g. Ubuntu) see
|
|
|
|
|
`WSL 2 Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.
|
|
|
|
|
|
|
|
|
|
- **Docker Desktop installation** :
|
|
|
|
|
Install Docker Desktop for Windows. For Windows Home follow the
|
|
|
|
|
`Docker Windows Home Installation Guide <https://docs.docker.com/docker-for-windows/install-windows-home>`_.
|
|
|
|
|
For Windows Pro, Enterprise, or Education follow the
|
|
|
|
|
`Docker Windows Installation Guide <https://docs.docker.com/docker-for-windows/install/>`_.
|
2020-03-01 19:38:07 +03:00
|
|
|
|
|
|
|
|
|
- **Docker setting** :
|
2020-06-01 10:04:31 +03:00
|
|
|
|
WSL integration needs to be enabled
|
2020-03-01 19:38:07 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<img src="images/docker_wsl_integration.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Docker WSL2 integration">
|
|
|
|
|
</div>
|
2020-06-01 10:04:31 +03:00
|
|
|
|
|
|
|
|
|
- **WSL 2 Filesystem Performance** :
|
|
|
|
|
Accessing the host Windows filesystem incurs a performance penalty,
|
|
|
|
|
it is therefore recommended to do development on the Linux filesystem.
|
|
|
|
|
E.g. Run ``cd ~`` and create a development folder in your Linux distro home
|
|
|
|
|
and git pull the Airflow repo there.
|
|
|
|
|
|
|
|
|
|
- **WSL 2 Memory Usage** :
|
2020-06-04 20:12:09 +03:00
|
|
|
|
WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after
|
|
|
|
|
development you can:
|
|
|
|
|
|
2020-06-01 10:04:31 +03:00
|
|
|
|
* On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
|
|
|
|
|
* If no longer using Docker you can quit Docker Desktop
|
|
|
|
|
(right click system try icon and select "Quit Docker Desktop")
|
|
|
|
|
* If no longer using WSL you can shut it down on the Windows Host
|
|
|
|
|
with the following command: ``wsl --shutdown``
|
|
|
|
|
|
|
|
|
|
- **Developing in WSL 2** :
|
|
|
|
|
You can use all the standard Linux command line utilities to develop on WSL 2.
|
|
|
|
|
Further VS Code supports developing in Windows but remotely executing in WSL.
|
|
|
|
|
If VS Code is installed on the Windows host system then in the WSL Linux Distro
|
|
|
|
|
you can run ``code .`` in the root directory of you Airflow repo to launch VS Code.
|
2020-03-01 19:38:07 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Getopt and gstat
|
|
|
|
|
----------------
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-12-04 15:15:02 +03:00
|
|
|
|
* For Linux, run ``apt install util-linux coreutils`` or an equivalent if your system is not Debian-based.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
* For macOS, install GNU ``getopt`` and ``gstat`` utilities to get Airflow Breeze running.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Run ``brew install gnu-getopt coreutils`` and then follow instructions to link the gnu-getopt version to
|
|
|
|
|
become the first on the PATH. Make sure to re-login after you make the suggested changes.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-12-04 15:15:02 +03:00
|
|
|
|
**Examples:**
|
|
|
|
|
|
|
|
|
|
If you use bash, run this command and re-login:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.bash_profile
|
|
|
|
|
. ~/.bash_profile
|
|
|
|
|
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2019-12-04 15:15:02 +03:00
|
|
|
|
If you use zsh, run this command and re-login:
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zprofile
|
|
|
|
|
. ~/.zprofile
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2019-09-05 11:28:48 +03:00
|
|
|
|
Memory
|
|
|
|
|
------
|
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Minimum 4GB RAM is required to run the full Breeze environment.
|
2019-09-05 11:28:48 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
On macOS, 2GB of RAM are available for your Docker containers by default, but more memory is recommended
|
2019-09-05 11:28:48 +03:00
|
|
|
|
(4GB should be comfortable). For details see
|
2019-10-26 14:31:51 +03:00
|
|
|
|
`Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_.
|
2019-09-05 11:28:48 +03:00
|
|
|
|
|
2020-09-13 17:42:30 +03:00
|
|
|
|
On Windows WSL 2 expect the Linux Distro and Docker containers to use 7 - 8 GB of RAM.
|
2020-06-01 10:04:31 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Cleaning the environment
|
|
|
|
|
------------------------
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
You may need to clean up your Docker environment occasionally. The images are quite big
|
|
|
|
|
(1.5GB for both images needed for static code analysis and CI tests) and, if you often rebuild/update
|
|
|
|
|
them, you may end up with some unused image data.
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
To clean up the Docker environment:
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
1. Stop Breeze with ``./breeze stop``.
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
2. Run the ``docker system prune`` command.
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
3. Run ``docker images --all`` and ``docker ps --all`` to verify that your Docker is clean.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Both commands should return an empty list of images and containers respectively.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
If you run into disk space errors, consider pruning your Docker images with the ``docker system prune --all``
|
|
|
|
|
command. You may need to restart the Docker Engine before running this command.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
In case of disk space errors on macOS, increase the disk space available for Docker. See
|
|
|
|
|
`Prerequisites <#prerequisites>`_ for details.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Installation
|
|
|
|
|
============
|
|
|
|
|
|
|
|
|
|
Installation is as easy as checking out Airflow repository and running Breeze command.
|
|
|
|
|
You enter the Breeze test environment by running the ``./breeze`` script. You can run it with
|
|
|
|
|
the ``help`` command to see the list of available options. See `Breeze Command-Line Interface Reference`_
|
|
|
|
|
for details.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze
|
|
|
|
|
|
|
|
|
|
The First time you run Breeze, it pulls and builds a local version of Docker images.
|
|
|
|
|
It pulls the latest Airflow CI images from `Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_
|
|
|
|
|
and uses them to build your local Docker images. Note that the first run (per python) might take up to 10
|
|
|
|
|
minutes on a fast connection to start. Subsequent runs should be much faster.
|
|
|
|
|
|
|
|
|
|
Once you enter the environment, you are dropped into bash shell of the Airflow container and you can
|
|
|
|
|
run tests immediately.
|
|
|
|
|
|
|
|
|
|
To use the full potential of breeze you should set up autocomplete and you can
|
|
|
|
|
add the checked-out Airflow repository to your PATH to run Breeze without the ``./`` and from any directory.
|
|
|
|
|
|
|
|
|
|
The ``breeze`` command comes with a built-in bash/zsh autocomplete setup command. After installing, when you
|
|
|
|
|
start typing the command, you can use <TAB> to show all the available switches and get
|
|
|
|
|
auto-completion on typical values of parameters that you can use.
|
|
|
|
|
|
|
|
|
|
You should set up the autocomplete option automatically by running:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze setup-autocomplete
|
|
|
|
|
|
|
|
|
|
You get the auto-completion working when you re-enter the shell.
|
|
|
|
|
|
|
|
|
|
When you enter the Breeze environment, automatically an environment file is sourced from
|
|
|
|
|
``files/airflow-breeze-config/variables.env``. The ``files`` folder from your local sources is
|
|
|
|
|
automatically mounted to the container under ``/files`` path and you can put there any files you want
|
|
|
|
|
to make available for the Breeze container.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=78">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_installation.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Installation">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Running tests in the CI interactive environment
|
|
|
|
|
===============================================
|
|
|
|
|
|
|
|
|
|
Breeze helps with running tests in the same environment/way as CI tests are run. You can run various
|
|
|
|
|
types of tests while you enter Breeze CI interactive environment - this is described in detail
|
|
|
|
|
in `<TESTING.rst>`_
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=262">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_running_tests.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Running tests">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Choosing different Breeze environment configuration
|
|
|
|
|
===================================================
|
|
|
|
|
|
|
|
|
|
You can use additional ``breeze`` flags to choose your environment. You can specify a Python
|
|
|
|
|
version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same
|
|
|
|
|
environments as we have in matrix builds in the CI.
|
|
|
|
|
|
|
|
|
|
For example, you can choose to run Python 3.6 tests with MySQL as backend and in the Docker environment as
|
|
|
|
|
follows:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
./breeze --python 3.6 --backend mysql
|
|
|
|
|
|
|
|
|
|
The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the
|
|
|
|
|
``breeze`` script, it could use the values that were used previously. This way you do not have to specify
|
|
|
|
|
them when you run the script. You can delete the ``.build/`` directory in case you want to restore the
|
|
|
|
|
default settings.
|
|
|
|
|
|
|
|
|
|
The defaults when you run the Breeze environment are Python 3.6 version and SQLite database.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=389">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_select_backend_python.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Selecting Python and Backend version">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Troubleshooting
|
|
|
|
|
===============
|
|
|
|
|
|
|
|
|
|
If you are having problems with the Breeze environment, try the steps below. After each step you
|
|
|
|
|
can check whether your problem is fixed.
|
|
|
|
|
|
|
|
|
|
1. If you are on macOS, check if you have enough disk space for Docker.
|
|
|
|
|
2. Restart Breeze with ``./breeze restart``.
|
|
|
|
|
3. Delete the ``.build`` directory and run ``./breeze build-image --force-pull-images``.
|
|
|
|
|
4. Clean up Docker images via ``breeze cleanup-image`` command.
|
|
|
|
|
5. Restart your Docker Engine and try again.
|
|
|
|
|
6. Restart your machine and try again.
|
|
|
|
|
7. Re-install Docker CE and try again.
|
|
|
|
|
|
|
|
|
|
In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":
|
|
|
|
|
|
|
|
|
|
.. code-block::
|
|
|
|
|
|
|
|
|
|
export VERBOSE_COMMANDS="true"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Then run the failed command, copy-and-paste the output from your terminal to the
|
2020-09-22 12:33:49 +03:00
|
|
|
|
`Airflow Slack <https://s.apache.org/airflow-slack>`_ #airflow-breeze channel and
|
2020-07-11 22:21:19 +03:00
|
|
|
|
describe your problem.
|
|
|
|
|
|
2020-09-13 17:43:01 +03:00
|
|
|
|
Other uses of the Airflow Breeze environment
|
|
|
|
|
============================================
|
2019-10-26 14:31:51 +03:00
|
|
|
|
|
|
|
|
|
Airflow Breeze is a bash script serving as a "swiss-army-knife" of Airflow testing. Under the
|
|
|
|
|
hood it uses other scripts that you can also run manually if you have problem with running the Breeze
|
|
|
|
|
environment.
|
|
|
|
|
|
|
|
|
|
Breeze script allows performing the following tasks:
|
|
|
|
|
|
2020-07-01 15:50:30 +03:00
|
|
|
|
Managing CI environment:
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-01 15:50:30 +03:00
|
|
|
|
* Build CI docker image with ``breeze build-image`` command
|
|
|
|
|
* Enter interactive shell in CI container when ``shell`` (or no command) is specified
|
2020-04-02 20:52:11 +03:00
|
|
|
|
* Join running interactive shell with ``breeze exec`` command
|
|
|
|
|
* Stop running interactive environment with ``breeze stop`` command
|
|
|
|
|
* Restart running interactive environment with ``breeze restart`` command
|
2020-07-01 15:50:30 +03:00
|
|
|
|
* Run test specified with ``breeze tests`` command
|
2020-07-20 15:36:03 +03:00
|
|
|
|
* Generate constraints with ``breeze generate-constraints`` command
|
2020-07-01 15:50:30 +03:00
|
|
|
|
* Execute arbitrary command in the test environment with ``breeze shell`` command
|
|
|
|
|
* Execute arbitrary docker-compose command with ``breeze docker-compose`` command
|
|
|
|
|
* Push docker images with ``breeze push-image`` command (require committer's rights to push images)
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-07-01 15:50:30 +03:00
|
|
|
|
You can optionally reset database if specified as extra ``--db-reset`` flag and for CI image you can also
|
|
|
|
|
start integrations (separate Docker images) if specified as extra ``--integration`` flags. You can also
|
|
|
|
|
chose which backend database should be used with ``--backend`` flag and python version with ``--python`` flag.
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-10-08 18:47:56 +03:00
|
|
|
|
You can also have breeze launch Airflow automatically ``breeze start-airflow``, this will drop you in a
|
2020-09-12 00:26:56 +03:00
|
|
|
|
tmux session with three panes (one to monitor the scheduler, one for the webserver and one with a shell
|
|
|
|
|
for additional commands.
|
|
|
|
|
|
2020-07-01 15:50:30 +03:00
|
|
|
|
Managing Prod environment (with ``--production-image`` flag):
|
|
|
|
|
|
|
|
|
|
* Build CI docker image with ``breeze build-image`` command
|
|
|
|
|
* Enter interactive shell in PROD container when ``shell`` (or no command) is specified
|
|
|
|
|
* Join running interactive shell with ``breeze exec`` command
|
|
|
|
|
* Stop running interactive environment with ``breeze stop`` command
|
|
|
|
|
* Restart running interactive environment with ``breeze restart`` command
|
|
|
|
|
* Execute arbitrary command in the test environment with ``breeze shell`` command
|
2020-04-02 20:52:11 +03:00
|
|
|
|
* Execute arbitrary docker-compose command with ``breeze docker-compose`` command
|
2020-07-01 15:50:30 +03:00
|
|
|
|
* Push docker images with ``breeze push-image`` command (require committer's rights to push images)
|
|
|
|
|
|
|
|
|
|
You can optionally reset database if specified as extra ``--db-reset`` flag. You can also
|
|
|
|
|
chose which backend database should be used with ``--backend`` flag and python version with ``--python`` flag.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Manage and Interact with Kubernetes tests environment:
|
|
|
|
|
|
|
|
|
|
* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``breeze kind-cluster`` commands
|
|
|
|
|
* Run Kubernetes tests specified with ``breeze kind-cluster tests`` command
|
|
|
|
|
* Enter the interactive kubernetes test environment with ``breeze kind-cluster shell`` command
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
|
|
|
|
Run static checks:
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
* Run static checks - either for currently staged change or for all files with
|
2020-07-01 17:02:24 +03:00
|
|
|
|
``breeze static-check`` command
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
|
|
|
|
Build documentation:
|
|
|
|
|
|
|
|
|
|
* Build documentation with ``breeze build-docs`` command
|
|
|
|
|
|
|
|
|
|
Set up local development environment:
|
|
|
|
|
|
|
|
|
|
* Setup local virtualenv with ``breeze setup-virtualenv`` command
|
|
|
|
|
* Setup autocomplete for itself with ``breeze setup-autocomplete`` command
|
|
|
|
|
|
2020-10-18 17:39:44 +03:00
|
|
|
|
Database volumes in Breeze
|
|
|
|
|
--------------------------
|
|
|
|
|
|
|
|
|
|
Breeze keeps data for all it's integration in named docker volumes. Each backend and integration
|
|
|
|
|
keeps data in their own volume. Those volumes are persisted until ``./breeze stop`` command or
|
|
|
|
|
``./breeze restart`` command is run. You can also preserve the volumes by adding flag
|
|
|
|
|
``--preserve-volumes`` when you run either of those commands. Then, next time when you start
|
|
|
|
|
``Breeze``, it will have the data pre-populated. You can always delete the volumes by
|
2020-10-18 21:45:25 +03:00
|
|
|
|
running ``./breeze stop`` without the ``--preserve-volumes`` flag.
|
2020-10-18 17:39:44 +03:00
|
|
|
|
|
2020-03-07 22:18:51 +03:00
|
|
|
|
Launching multiple terminals
|
2020-07-11 22:21:19 +03:00
|
|
|
|
----------------------------
|
2020-03-07 22:18:51 +03:00
|
|
|
|
|
|
|
|
|
Often if you want to run full airflow in the Breeze environment you need to launch multiple terminals and
|
|
|
|
|
run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in separate terminals.
|
|
|
|
|
|
|
|
|
|
This can be achieved either via ``tmux`` or via exec-ing into the running container from the host. Tmux
|
2020-03-31 18:44:14 +03:00
|
|
|
|
is installed inside the container and you can launch it with ``tmux`` command. Tmux provides you with the
|
2020-03-07 22:18:51 +03:00
|
|
|
|
capability of creating multiple virtual terminals and multiplex between them. More about ``tmux`` can be
|
2020-10-21 15:32:41 +03:00
|
|
|
|
found at `tmux GitHub wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts
|
2020-03-07 22:18:51 +03:00
|
|
|
|
that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=824">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_using_tmux.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Using tmux">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Another way is to exec into Breeze terminal from the host's terminal. Often you can
|
2020-03-07 22:18:51 +03:00
|
|
|
|
have multiple terminals in the host (Linux/MacOS/WSL2 on Windows) and you can simply use those terminals
|
2020-03-31 18:44:14 +03:00
|
|
|
|
to enter the running container. It's as easy as launching ``breeze exec`` while you already started the
|
2020-03-07 22:18:51 +03:00
|
|
|
|
Breeze environment. You will be dropped into bash and environment variables will be read in the same
|
|
|
|
|
way as when you enter the environment. You can do it multiple times and open as many terminals as you need.
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=978">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_using_exec.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Using tmux">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
CLIs for cloud providers
|
2020-07-11 22:21:19 +03:00
|
|
|
|
------------------------
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
|
|
|
|
|
CLIs are not installed when you build or pull the image - they will be downloaded as docker images
|
|
|
|
|
the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
|
|
|
|
|
it is downloaded, it will stay until you remove the downloaded images from your host container.
|
|
|
|
|
|
|
|
|
|
For each of those CLI credentials are taken (automatically) from the credentials you have defined in
|
|
|
|
|
your ${HOME} directory on host.
|
|
|
|
|
|
|
|
|
|
Those tools also have host Airflow source directory mounted in /opt/airflow path
|
|
|
|
|
so you can directly transfer files to/from your airflow host sources.
|
|
|
|
|
|
|
|
|
|
Those are currently installed CLIs (they are available as aliases to the docker commands):
|
|
|
|
|
|
|
|
|
|
+-----------------------+----------+-------------------------------------------------+-------------------+
|
|
|
|
|
| Cloud Provider | CLI tool | Docker image | Configuration dir |
|
|
|
|
|
+=======================+==========+=================================================+===================+
|
|
|
|
|
| Amazon Web Services | aws | amazon/aws-cli:latest | .aws |
|
|
|
|
|
+-----------------------+----------+-------------------------------------------------+-------------------+
|
|
|
|
|
| Microsoft Azure | az | mcr.microsoft.com/azure-cli:latest | .azure |
|
|
|
|
|
+-----------------------+----------+-------------------------------------------------+-------------------+
|
2020-08-30 00:36:52 +03:00
|
|
|
|
| Google Cloud | bq | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
|
2020-07-01 17:02:24 +03:00
|
|
|
|
| +----------+-------------------------------------------------+-------------------+
|
|
|
|
|
| | gcloud | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
|
|
|
|
|
| +----------+-------------------------------------------------+-------------------+
|
|
|
|
|
| | gsutil | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud |
|
|
|
|
|
+-----------------------+----------+-------------------------------------------------+-------------------+
|
|
|
|
|
|
|
|
|
|
For each of the CLIs we have also an accompanying ``*-update`` alias (for example ``aws-update``) which
|
2020-08-30 00:36:52 +03:00
|
|
|
|
will pull the latest image for the tool. Note that all Google Cloud tools are served by one
|
2020-07-01 17:02:24 +03:00
|
|
|
|
image and they are updated together.
|
|
|
|
|
|
|
|
|
|
Also - in case you run several different Breeze containers in parallel (from different directories,
|
|
|
|
|
with different versions) - they docker images for CLI Cloud Providers tools are shared so if you update it
|
|
|
|
|
for one Breeze container, they will also get updated for all the other containers.
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1072">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_cloud_tools.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Cloud tools">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Launching Breeze integrations
|
|
|
|
|
-----------------------------
|
|
|
|
|
|
|
|
|
|
When Breeze starts, it can start additional integrations. Those are additional docker containers
|
|
|
|
|
that are started in the same docker-compose command. Those are required by some of the tests
|
2020-08-20 10:57:07 +03:00
|
|
|
|
as described in `<TESTING.rst#airflow-integration-tests>`_.
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
By default Breeze starts only airflow container without any integration enabled. If you selected
|
|
|
|
|
``postgres`` or ``mysql`` backend, the container for the selected backend is also started (but only the one
|
|
|
|
|
that is selected). You can start the additional integrations by passing ``--integration`` flag
|
|
|
|
|
with appropriate integration name when starting Breeze. You can specify several ``--integration`` flags
|
|
|
|
|
to start more than one integration at a time.
|
|
|
|
|
Finally you can specify ``--integration all`` to start all integrations.
|
|
|
|
|
|
|
|
|
|
Once integration is started, it will continue to run until the environment is stopped with
|
|
|
|
|
``breeze stop`` command. or restarted via ``breeze restart`` command
|
|
|
|
|
|
|
|
|
|
Note that running integrations uses significant resources - CPU and memory.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1187">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_integrations.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Integrations">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Building CI images
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
With Breeze you can build images that are used by Airflow CI and production ones.
|
|
|
|
|
|
|
|
|
|
For all development tasks, unit tests, integration tests, and static code checks, we use the
|
|
|
|
|
**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
|
|
|
|
|
This Docker image contains a lot of test-related packages (size of ~1GB).
|
|
|
|
|
Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
|
|
|
|
|
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v1-10-test-python3.6-ci``).
|
|
|
|
|
The image is built using the `<Dockerfile.ci>`_ Dockerfile.
|
|
|
|
|
|
|
|
|
|
The CI image is built automatically as needed, however it can be rebuilt manually with
|
|
|
|
|
``build-image`` command. The production
|
|
|
|
|
image should be built manually - but also a variant of this image is built automatically when
|
|
|
|
|
kubernetes tests are executed see `Running Kubernetes tests <#running-kubernetes-tests>`_
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1387">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_build_images.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Building images">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
|
|
|
|
|
time. But for subsequent source code changes, no wait time is expected.
|
|
|
|
|
However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
|
|
|
|
|
that may take more time though it is highly optimized to only rebuild what is needed.
|
|
|
|
|
|
|
|
|
|
Breeze has built in mechanism to check if your local image has not diverged too much from the
|
|
|
|
|
latest image build on CI. This might happen when for example latest patches have been released as new
|
|
|
|
|
Python images or when significant changes are made in the Dockerfile. In such cases, Breeze will
|
|
|
|
|
download the latest images before rebuilding because this is usually faster than rebuilding the image.
|
|
|
|
|
|
|
|
|
|
In most cases, rebuilding an image requires network connectivity (for example, to download new
|
|
|
|
|
dependencies). If you work offline and do not want to rebuild the images when needed, you can set the
|
|
|
|
|
``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
|
|
|
|
|
`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
Preparing packages
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
Breeze can also be used to prepare airflow packages - both "apache-airflow" main package and
|
|
|
|
|
provider packages.
|
|
|
|
|
|
|
|
|
|
You can read more about testing provider packages in
|
|
|
|
|
`TESTING.rst <TESTING.rst#running-tests-with-packages>`_
|
|
|
|
|
|
|
|
|
|
There are several commands that you can run in Breeze to manage and build packages:
|
|
|
|
|
|
|
|
|
|
* preparing Provider Readme files
|
|
|
|
|
* preparing Airflow packages
|
|
|
|
|
* preparing Provider packages
|
|
|
|
|
|
|
|
|
|
Preparing provider readme files is part of the release procedure by the release managers
|
|
|
|
|
and it is described in detail in `dev <dev/README.md>`_ .
|
|
|
|
|
|
|
|
|
|
You can prepare provider packages - by default regular provider packages are prepared, but with
|
|
|
|
|
``--backport`` flag you can prepare backport packages.
|
|
|
|
|
|
|
|
|
|
The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
|
|
|
|
|
before running, so you should run it before generating airflow package below as it will be removed.
|
|
|
|
|
|
|
|
|
|
The below example builds provider packages in the wheel format.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-provider-packages
|
|
|
|
|
|
|
|
|
|
If you run this command without packages, you will prepare all packages, you can however specify
|
|
|
|
|
providers that you would like to build. By default only ``wheel`` packages are prepared,
|
|
|
|
|
but you can change it providing optional --package-format flag.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-provider-packages --package-format=both google amazon
|
|
|
|
|
|
|
|
|
|
You can also prepare backport provider packages, if you specify ``--backport`` flag. You can read more
|
|
|
|
|
about backport packages in `dev <dev/README.md>`_
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-provider-packages --backports --package-format=both google amazon
|
|
|
|
|
|
|
|
|
|
You can see all providers available by running this command:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-provider-packages -- --help
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You can also prepare airflow packages using breeze:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-airflow-packages
|
|
|
|
|
|
|
|
|
|
This prepares airflow .whl package in the dist folder.
|
|
|
|
|
|
|
|
|
|
Again, you can specify optional ``--package-format`` flag to build airflow packages.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze prepare-airflow-packages --package-format=bot
|
|
|
|
|
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Building Production images
|
|
|
|
|
--------------------------
|
|
|
|
|
|
|
|
|
|
The **Production image** is also maintained on the DockerHub in the
|
|
|
|
|
```apache/airflow`` repository. This Docker image (and Dockerfile) contains size-optimised Airflow
|
|
|
|
|
installation with selected extras and dependencies. Its tag follows the pattern of
|
|
|
|
|
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example, ``apache/airflow:master-python3.6``
|
|
|
|
|
or ``apache/airflow:v1-10-test-python3.6``).
|
|
|
|
|
|
|
|
|
|
However in many cases you want to add your own custom version of the image - with added apt dependencies,
|
|
|
|
|
python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
|
2020-09-29 16:30:00 +03:00
|
|
|
|
customized variant of the image that contains everything you need.
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
You can switch to building the production image by adding ``--production-image`` flag to the ``build_image``
|
|
|
|
|
command. Note, that the images can also be build using ``docker build`` command by passing appropriate
|
|
|
|
|
build-args as described in `IMAGES.rst <IMAGES.rst>`_ , but Breeze provides several flags that
|
|
|
|
|
makes it easier to do it. You can see all the flags by running ``./breeze build-image --help``,
|
|
|
|
|
but here typical examples are presented:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --additional-extras "jira"
|
|
|
|
|
|
|
|
|
|
This installs additional ``jira`` extra while installing airflow in the image.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --additional-python-deps "torchio==0.17.10"
|
|
|
|
|
|
|
|
|
|
This install additional pypi dependency - torchio in specified version.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
./breeze build-image --production-image --additional-dev-apt-deps "libasound2-dev" \
|
|
|
|
|
--additional-runtime-apt-deps "libasound2"
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
This install additional apt dependencies - ``libasound2-dev`` in build image and ``libasound`` in the
|
|
|
|
|
final image. Those are development dependencies that might be needed to build and use python packages added
|
|
|
|
|
via the ``--additional-python-deps`` flag. The ``dev`` dependencies are not installed in the final
|
|
|
|
|
production image, they are only installed in the build "segment" of the production image that is used
|
|
|
|
|
as an intermediate step to build the final image. Usually names of the ``dev`` dependencies end with ``-dev``
|
|
|
|
|
suffix and they need to also be paired with corresponding runtime dependency added for the runtime image
|
|
|
|
|
(without -dev).
|
|
|
|
|
|
2020-07-13 20:23:55 +03:00
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --python 3.7 --additional-dev-deps "libasound2-dev" \
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-runtime-apt-deps "libasound2"
|
2020-07-13 20:23:55 +03:00
|
|
|
|
|
|
|
|
|
Same as above but uses python 3.7.
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1496">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_build_images_prod.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Building Production images">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Building Production images for 1.10 Airflow versions
|
|
|
|
|
----------------------------------------------------
|
|
|
|
|
|
|
|
|
|
With Breeze you can also use the master Dockerfile to build custom images for released Airflow versions.
|
|
|
|
|
This works in the same way as building production image from master, but you need to add additional switch
|
|
|
|
|
``--install-airflow-version``. You should pass version of airflow (as released in PyPI). It can be used
|
|
|
|
|
to install both released versions and release candidates. Similarly as in case of master images,
|
|
|
|
|
we can pass additional extras/dependencies to install via the additional flags.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --additional-extras "jira" --install-airflow-version="1.10.11"
|
|
|
|
|
|
|
|
|
|
Builds airflow image with released Airflow version 1.10.11 and additional extra "jira" added.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --install-airflow-version="1.10.11rc2"
|
|
|
|
|
|
|
|
|
|
Builds airflow image with released Airflow version 1.10.11rc2.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You can also build airflow directly from GitHub source code - by providing Git Reference via
|
|
|
|
|
``--install-airflow-reference``. The reference can be a branch name, tag name, or commit hash. This
|
|
|
|
|
is useful mostly for testing.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image --install-airflow-reference="v1-10-test"
|
|
|
|
|
|
|
|
|
|
This Builds airflow image from the current ``v1-10-test`` branch of Airflow.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-image --production-image \
|
|
|
|
|
--install-airflow-reference="0d91fcf725f69e10f0969ca36f9e38e1d74110d0"
|
|
|
|
|
|
|
|
|
|
This Builds airflow image from the ``0d91fcf725f69e10f0969ca36f9e38e1d74110d0`` commit hash on
|
|
|
|
|
GitHub.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1586">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_build_images_released_versions.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Building Production images for 1.10 Airflow versions">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Running static checks
|
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
|
|
You can run static checks via Breeze. You can also run them via pre-commit command but with auto-completion
|
|
|
|
|
Breeze makes it easier to run selective static checks. If you press <TAB> after the static-check and if
|
|
|
|
|
you have auto-complete setup you should see auto-completable list of all checks available.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze static-check mypy
|
|
|
|
|
|
|
|
|
|
The above will run mypy check for currently staged files.
|
|
|
|
|
|
|
|
|
|
You can also add arbitrary pre-commit flag after ``--``
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze static-check mypy -- --all-files
|
|
|
|
|
|
|
|
|
|
The above will run mypy check for all files.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1675">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_static_checks.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Static checks">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
2020-12-16 11:03:48 +03:00
|
|
|
|
If you want ever need to get a list of the files that will be checked (for troubleshooting when playing with the
|
|
|
|
|
``--from-ref`` and ``--to-ref``
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
breeze static-check identity --verbose # currently staged files
|
|
|
|
|
breeze static-check identity --verbose -- --from-ref $(git merge-base master HEAD) --to-ref HEAD # branch updates
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Building the Documentation
|
|
|
|
|
--------------------------
|
|
|
|
|
|
|
|
|
|
To build documentation in Breeze, use the ``build-docs`` command:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze build-docs
|
|
|
|
|
|
|
|
|
|
Results of the build can be found in the ``docs/_build`` folder.
|
|
|
|
|
|
2020-08-20 10:57:07 +03:00
|
|
|
|
The documentation build consists of three steps:
|
|
|
|
|
|
|
|
|
|
* verifying consistency of indexes
|
|
|
|
|
* building documentation
|
|
|
|
|
* spell checking
|
|
|
|
|
|
2020-08-31 13:46:39 +03:00
|
|
|
|
You can choose only one stage of the two by providing ``--spellcheck-only`` or ``--docs-only`` after
|
|
|
|
|
extra ``--`` flag.
|
2020-08-20 10:57:07 +03:00
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-08-31 13:46:39 +03:00
|
|
|
|
./breeze build-docs -- --spellcheck-only
|
2020-08-20 10:57:07 +03:00
|
|
|
|
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Often errors during documentation generation come from the docstrings of auto-api generated classes.
|
|
|
|
|
During the docs building auto-api generated files are stored in the ``docs/_api`` folder. This helps you
|
|
|
|
|
easily identify the location the problems with documentation originated from.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1760">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_build_docs.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Build docs">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
Generating constraints
|
|
|
|
|
----------------------
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
Whenever setup.py gets modified, the CI master job will re-generate constraint files. Those constraint
|
2020-12-30 16:24:21 +03:00
|
|
|
|
files are stored in separated orphan branches: ``constraints-master``, ``constraints-2-0`` and ``constraints-1-10``.
|
2020-07-20 15:36:03 +03:00
|
|
|
|
They are stored separately for each python version. Those are
|
|
|
|
|
constraint files as described in detail in the
|
2020-08-20 10:57:07 +03:00
|
|
|
|
`<CONTRIBUTING.rst#pinned-constraint-files>`_ contributing documentation.
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
In case someone modifies setup.py, the ``CRON`` scheduled CI build automatically upgrades and
|
|
|
|
|
pushes changed to the constraint files, however you can also perform test run of this locally using
|
|
|
|
|
``generate-constraints`` command of Breeze.
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
./breeze generate-constraints --python 3.6
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
./breeze generate-constraints --python 3.7
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
./breeze generate-constraints --python 3.8
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
|
|
|
|
|
and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
|
|
|
|
|
of changes vs the previous constraint files is printed.
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
Using local virtualenv environment in Your Host IDE
|
|
|
|
|
---------------------------------------------------
|
|
|
|
|
|
|
|
|
|
You can set up your host IDE (for example, IntelliJ's PyCharm/Idea) to work with Breeze
|
|
|
|
|
and benefit from all the features provided by your IDE, such as local and remote debugging,
|
|
|
|
|
language auto-completion, documentation support, etc.
|
|
|
|
|
|
|
|
|
|
To use your host IDE with Breeze:
|
|
|
|
|
|
|
|
|
|
1. Create a local virtual environment:
|
|
|
|
|
|
|
|
|
|
You can use any of the following wrappers to create and manage your virtual environments:
|
|
|
|
|
`pyenv <https://github.com/pyenv/pyenv>`_, `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_,
|
|
|
|
|
or `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_.
|
|
|
|
|
|
|
|
|
|
Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6, 3.7)
|
|
|
|
|
|
|
|
|
|
2. Use the right command to activate the virtualenv (``workon`` if you use virtualenvwrapper or
|
|
|
|
|
``pyenv activate`` if you use pyenv.
|
|
|
|
|
|
|
|
|
|
3. Initialize the created local virtualenv:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
./breeze initialize-local-virtualenv --python 3.8
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
4. Select the virtualenv you created as the project's default virtualenv in your IDE.
|
|
|
|
|
|
|
|
|
|
Note that you can also use the local virtualenv for Airflow development without Breeze.
|
|
|
|
|
This is a lightweight solution that has its own limitations.
|
|
|
|
|
|
|
|
|
|
More details on using the local virtualenv are available in the `LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`_.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=1920">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_initialize_virtualenv.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Initialize virtualenv">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Running Kubernetes tests
|
|
|
|
|
------------------------
|
|
|
|
|
|
|
|
|
|
Breeze helps with running Kubernetes tests in the same environment/way as CI tests are run.
|
|
|
|
|
Breeze helps to setup KinD cluster for testing, setting up virtualenv and downloads the right tools
|
|
|
|
|
automatically to run the tests.
|
|
|
|
|
|
|
|
|
|
This is described in detail in `Testing Kubernetes <TESTING.rst#running-tests-with-kubernetes>`_.
|
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=2093">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_kubernetes_tests.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Kubernetes tests">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
Stopping the interactive environment
|
|
|
|
|
------------------------------------
|
2019-09-05 11:28:48 +03:00
|
|
|
|
|
|
|
|
|
After starting up, the environment runs in the background and takes precious memory.
|
|
|
|
|
You can always stop it via:
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2019-12-04 15:15:02 +03:00
|
|
|
|
.. code-block:: bash
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2020-03-04 03:21:11 +03:00
|
|
|
|
./breeze stop
|
2019-09-05 11:28:48 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<a href="https://youtu.be/4MCTXq-oF68?t=2639">
|
|
|
|
|
<img src="images/breeze/overlayed_breeze_stop.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Stop environment">
|
|
|
|
|
</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Internal details of Breeze
|
|
|
|
|
==========================
|
|
|
|
|
|
|
|
|
|
Airflow directory structure inside container
|
|
|
|
|
--------------------------------------------
|
|
|
|
|
|
|
|
|
|
When you are in the CI container, the following directories are used:
|
|
|
|
|
|
|
|
|
|
.. code-block:: text
|
|
|
|
|
|
|
|
|
|
/opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
|
|
|
|
|
/root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
|
|
|
|
|
airflow.db - sqlite database in case sqlite is used;
|
|
|
|
|
dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
|
|
|
|
|
logs - logs from Airflow executions;
|
|
|
|
|
unittest.cfg - unit test configuration generated when entering the environment;
|
|
|
|
|
webserver_config.py - webserver configuration generated when running Airflow in the container.
|
|
|
|
|
|
|
|
|
|
Note that when running in your local environment, the ``/root/airflow/logs`` folder is actually mounted
|
|
|
|
|
from your ``logs`` directory in the Airflow sources, so all logs created in the container are automatically
|
|
|
|
|
visible in the host as well. Every time you enter the container, the ``logs`` directory is
|
|
|
|
|
cleaned so that logs do not accumulate.
|
|
|
|
|
|
|
|
|
|
When you are in the production container, the following directories are used:
|
|
|
|
|
|
|
|
|
|
.. code-block:: text
|
|
|
|
|
|
|
|
|
|
/opt/airflow - Contains sources of Airflow mounted from the host (AIRFLOW_SOURCES).
|
|
|
|
|
/root/airflow - Contains all the "dynamic" Airflow files (AIRFLOW_HOME), such as:
|
|
|
|
|
airflow.db - sqlite database in case sqlite is used;
|
|
|
|
|
dags - folder with non-test dags (test dags are in /opt/airflow/tests/dags);
|
|
|
|
|
logs - logs from Airflow executions;
|
|
|
|
|
unittest.cfg - unit test configuration generated when entering the environment;
|
|
|
|
|
webserver_config.py - webserver configuration generated when running Airflow in the container.
|
|
|
|
|
|
|
|
|
|
Note that when running in your local environment, the ``/root/airflow/logs`` folder is actually mounted
|
|
|
|
|
from your ``logs`` directory in the Airflow sources, so all logs created in the container are automatically
|
|
|
|
|
visible in the host as well. Every time you enter the container, the ``logs`` directory is
|
|
|
|
|
cleaned so that logs do not accumulate.
|
|
|
|
|
|
|
|
|
|
Running Arbitrary commands in the Breeze environment
|
|
|
|
|
----------------------------------------------------
|
|
|
|
|
|
|
|
|
|
To run other commands/executables inside the Breeze Docker-based environment, use the
|
|
|
|
|
``./breeze shell`` command. You should add your command as -c "command" after ``--`` as extra arguments.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze shell -- -c "ls -la"
|
|
|
|
|
|
|
|
|
|
Running "Docker Compose" commands
|
|
|
|
|
---------------------------------
|
|
|
|
|
|
|
|
|
|
To run Docker Compose commands (such as ``help``, ``pull``, etc), use the
|
|
|
|
|
``docker-compose`` command. To add extra arguments, specify them
|
|
|
|
|
after ``--`` as extra arguments.
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze docker-compose pull -- --ignore-pull-failures
|
|
|
|
|
|
2020-04-02 20:52:11 +03:00
|
|
|
|
Restarting Breeze environment
|
2020-07-11 22:21:19 +03:00
|
|
|
|
-----------------------------
|
2020-02-27 12:44:43 +03:00
|
|
|
|
|
|
|
|
|
You can also restart the environment and enter it via:
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
./breeze restart
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Setting default answers for user interaction
|
|
|
|
|
--------------------------------------------
|
|
|
|
|
|
|
|
|
|
Sometimes during the build, you are asked whether to perform an action, skip it, or quit. This happens
|
|
|
|
|
when rebuilding or removing an image - actions that take a lot of time and could be potentially destructive.
|
|
|
|
|
|
|
|
|
|
For automation scripts, you can export one of the three variables to control the default
|
|
|
|
|
interaction behaviour:
|
|
|
|
|
|
|
|
|
|
.. code-block::
|
2020-02-27 12:44:43 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
export FORCE_ANSWER_TO_QUESTIONS="yes"
|
2020-02-27 12:44:43 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are automatically rebuilt when needed.
|
|
|
|
|
Images are deleted without asking.
|
2019-09-05 11:28:48 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. code-block::
|
2019-08-28 10:33:31 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
export FORCE_ANSWER_TO_QUESTIONS="no"
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used even if rebuilding is needed.
|
|
|
|
|
This is useful when you work offline. Deleting images is aborted.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. code-block::
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
export FORCE_ANSWER_TO_QUESTIONS="quit"
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is aborted. Deleting images is aborted.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
If more than one variable is set, ``yes`` takes precedence over ``no``, which takes precedence over ``quit``.
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
Fixing File/Directory Ownership
|
|
|
|
|
-------------------------------
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
On Linux, there is a problem with propagating ownership of created files (a known Docker problem). The
|
|
|
|
|
files and directories created in the container are not owned by the host user (but by the root user in our
|
|
|
|
|
case). This may prevent you from switching branches, for example, if files owned by the root user are
|
|
|
|
|
created within your sources. In case you are on a Linux host and have some files in your sources created
|
|
|
|
|
by the root user, you can fix the ownership of those files by running this script:
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
.. code-block::
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-07-16 19:05:35 +03:00
|
|
|
|
./scripts/ci/tools/ci_fix_ownership.sh
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-07-01 17:02:24 +03:00
|
|
|
|
Mounting Local Sources to Breeze
|
2020-07-11 22:21:19 +03:00
|
|
|
|
--------------------------------
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
Important sources of Airflow are mounted inside the ``airflow`` container that you enter.
|
|
|
|
|
This means that you can continue editing your changes on the host in your favourite IDE and have them
|
|
|
|
|
visible in the Docker immediately and ready to test without rebuilding images. You can disable mounting
|
|
|
|
|
by specifying ``--skip-mounting-local-sources`` flag when running Breeze. In this case you will have sources
|
|
|
|
|
embedded in the container and changes to these sources will not be persistent.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
After you run Breeze for the first time, you will have empty directory ``files`` in your source code,
|
|
|
|
|
which will be mapped to ``/files`` in your Docker container. You can pass there any files you need to
|
|
|
|
|
configure and run Docker. They will not be removed between Docker runs.
|
|
|
|
|
|
|
|
|
|
By default ``/files/dags`` folder is mounted from your local ``<AIRFLOW_SOURCES>/files/dags`` and this is
|
|
|
|
|
the directory used by airflow scheduler and webserver to scan dags for. You can use it to test your dags
|
|
|
|
|
from local sources in Airflow. If you wish to add local DAGs that can be run by Breeze.
|
|
|
|
|
|
|
|
|
|
Port Forwarding
|
2020-07-11 22:21:19 +03:00
|
|
|
|
---------------
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
When you run Airflow Breeze, the following ports are automatically forwarded:
|
|
|
|
|
|
|
|
|
|
* 28080 -> forwarded to Airflow webserver -> airflow:8080
|
2020-10-18 12:46:22 +03:00
|
|
|
|
* 25555 -> forwarded to Flower dashboard -> airflow:5555
|
2020-07-01 17:02:24 +03:00
|
|
|
|
* 25433 -> forwarded to Postgres database -> postgres:5432
|
|
|
|
|
* 23306 -> forwarded to MySQL database -> mysql:3306
|
2020-10-18 12:46:22 +03:00
|
|
|
|
* 26379 -> forwarded to Redis broker -> redis:6379
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
You can connect to these ports/databases using:
|
|
|
|
|
|
|
|
|
|
* Webserver: ``http://127.0.0.1:28080``
|
2020-10-18 12:46:22 +03:00
|
|
|
|
* Flower: ``http://127.0.0.1:25555``
|
2020-07-01 17:02:24 +03:00
|
|
|
|
* Postgres: ``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow``
|
2020-10-18 12:46:22 +03:00
|
|
|
|
* Mysql: ``jdbc:mysql://127.0.0.1:23306/airflow?user=root``
|
|
|
|
|
* Redis: ``redis://127.0.0.1:26379/0```
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
Start the webserver manually with the ``airflow webserver`` command if you want to connect
|
|
|
|
|
to the webserver. You can use ``tmux`` to multiply terminals. You may need to create a user prior to
|
|
|
|
|
running the webserver in order to log in. This can be done with the following command:
|
|
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
|
|
|
|
airflow users create --role Admin --username admin --password admin --email admin@example.com --firstname foo --lastname bar
|
|
|
|
|
|
|
|
|
|
For databases, you need to run ``airflow db reset`` at least once (or run some tests) after you started
|
|
|
|
|
Airflow Breeze to get the database/tables created. You can connect to databases with IDE or any other
|
|
|
|
|
database client:
|
|
|
|
|
|
2020-07-11 22:21:19 +03:00
|
|
|
|
|
|
|
|
|
.. raw:: html
|
|
|
|
|
|
|
|
|
|
<div align="center">
|
|
|
|
|
<img src="images/database_view.png" width="640"
|
|
|
|
|
alt="Airflow Breeze - Database view">
|
|
|
|
|
</div>
|
2020-07-01 17:02:24 +03:00
|
|
|
|
|
|
|
|
|
You can change the used host port numbers by setting appropriate environment variables:
|
|
|
|
|
|
|
|
|
|
* ``WEBSERVER_HOST_PORT``
|
|
|
|
|
* ``POSTGRES_HOST_PORT``
|
|
|
|
|
* ``MYSQL_HOST_PORT``
|
|
|
|
|
|
|
|
|
|
If you set these variables, next time when you enter the environment the new ports should be in effect.
|
|
|
|
|
|
|
|
|
|
Managing Dependencies
|
|
|
|
|
---------------------
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-03-23 10:56:26 +03:00
|
|
|
|
If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or
|
2020-10-10 01:50:31 +03:00
|
|
|
|
add JavaScript dependencies in ``package.json``, you can either add dependencies temporarily for a single
|
2020-03-23 10:56:26 +03:00
|
|
|
|
Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Installing Dependencies for a Single Breeze Session
|
|
|
|
|
...................................................
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
You can install dependencies inside the container using ``sudo apt install``, ``pip install`` or
|
2019-12-20 00:21:41 +03:00
|
|
|
|
``yarn install`` (in ``airflow/www`` folder) respectively. This is useful if you want to test something
|
2019-10-26 14:31:51 +03:00
|
|
|
|
quickly while you are in the container. However, these changes are not retained: they disappear once you
|
2020-01-07 02:08:12 +03:00
|
|
|
|
exit the container (except for the node.js dependencies if your sources are mounted to the container).
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Therefore, if you want to retain a new dependency, follow the second option described below.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Adding Dependencies Permanently
|
|
|
|
|
...............................
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-03-23 10:56:26 +03:00
|
|
|
|
You can add dependencies to the ``Dockerfile.ci``, ``setup.py`` or ``package.json`` and rebuild the image.
|
|
|
|
|
This should happen automatically if you modify any of these files.
|
2019-10-26 14:31:51 +03:00
|
|
|
|
After you exit the container and re-run ``breeze``, Breeze detects changes in dependencies,
|
|
|
|
|
asks you to confirm rebuilding the image and proceeds with rebuilding if you confirm (or skip it
|
2020-02-25 00:31:50 +03:00
|
|
|
|
if you do not confirm). After rebuilding is done, Breeze drops you to shell. You may also use the
|
2020-04-02 20:52:11 +03:00
|
|
|
|
``build-image`` command to only rebuild CI image and not to go into shell.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-01 17:02:24 +03:00
|
|
|
|
Incremental apt Dependencies in the Dockerfile.ci during development
|
|
|
|
|
....................................................................
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-01 17:02:24 +03:00
|
|
|
|
During development, changing dependencies in ``apt-get`` closer to the top of the ``Dockerfile.ci``
|
|
|
|
|
invalidates cache for most of the image. It takes long time for Breeze to rebuild the image.
|
|
|
|
|
So, it is a recommended practice to add new dependencies initially closer to the end
|
|
|
|
|
of the ``Dockerfile.ci``. This way dependencies will be added incrementally.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-07-01 17:02:24 +03:00
|
|
|
|
Before merge, these dependencies should be moved to the appropriate ``apt-get install`` command,
|
|
|
|
|
which is already in the ``Dockerfile.ci``.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Breeze Command-Line Interface Reference
|
|
|
|
|
=======================================
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
Airflow Breeze Syntax
|
|
|
|
|
---------------------
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-10-26 14:31:51 +03:00
|
|
|
|
This is the current syntax for `./breeze <./breeze>`_:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-11-04 12:37:31 +03:00
|
|
|
|
.. START BREEZE HELP MARKER
|
|
|
|
|
|
2019-08-27 21:39:36 +03:00
|
|
|
|
.. code-block:: text
|
|
|
|
|
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
####################################################################################################
|
|
|
|
|
|
2020-09-02 22:58:37 +03:00
|
|
|
|
usage: breeze [FLAGS] [COMMAND] -- <EXTRA_ARGS>
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
2020-09-02 22:55:50 +03:00
|
|
|
|
By default the script enters the CI container and drops you to bash shell, but you can choose
|
|
|
|
|
one of the commands to run specific actions instead.
|
|
|
|
|
|
|
|
|
|
Add --help after each command to see details:
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
|
|
|
|
Commands without arguments:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
shell [Default] Enters interactive shell in the container
|
|
|
|
|
build-docs Builds documentation in the container
|
2020-04-02 20:52:11 +03:00
|
|
|
|
build-image Builds CI or Production docker image
|
|
|
|
|
cleanup-image Cleans up the container image created
|
2020-03-07 22:18:51 +03:00
|
|
|
|
exec Execs into running breeze container in new terminal
|
2020-07-20 15:36:03 +03:00
|
|
|
|
generate-constraints Generates pinned constraint files
|
2020-06-02 11:24:47 +03:00
|
|
|
|
push-image Pushes images to registry
|
2020-02-25 00:31:50 +03:00
|
|
|
|
initialize-local-virtualenv Initializes local virtualenv
|
2020-12-07 01:36:33 +03:00
|
|
|
|
prepare-airflow-packages Prepares airflow packages
|
2020-02-25 00:31:50 +03:00
|
|
|
|
setup-autocomplete Sets up autocomplete for breeze
|
2020-09-27 19:31:50 +03:00
|
|
|
|
start-airflow Starts Scheduler and Webserver and enters the shell
|
2020-04-13 09:51:33 +03:00
|
|
|
|
stop Stops the docker-compose environment
|
|
|
|
|
restart Stops the docker-compose environment including DB cleanup
|
2020-02-25 00:31:50 +03:00
|
|
|
|
toggle-suppress-cheatsheet Toggles on/off cheatsheet
|
|
|
|
|
toggle-suppress-asciiart Toggles on/off asciiart
|
|
|
|
|
|
|
|
|
|
Commands with arguments:
|
|
|
|
|
|
|
|
|
|
docker-compose <ARG> Executes specified docker-compose command
|
2020-06-03 21:58:38 +03:00
|
|
|
|
kind-cluster <ARG> Manages KinD cluster on the host
|
2020-10-12 17:29:48 +03:00
|
|
|
|
prepare-provider-readme <ARG> Prepares provider packages readme files
|
|
|
|
|
prepare-provider-packages <ARG> Prepares provider packages
|
2020-02-25 00:31:50 +03:00
|
|
|
|
static-check <ARG> Performs selected static check for changed files
|
2020-06-16 13:36:46 +03:00
|
|
|
|
tests <ARG> Runs selected tests in the container
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
|
|
|
|
Help commands:
|
|
|
|
|
|
|
|
|
|
flags Shows all breeze's flags
|
|
|
|
|
help Shows this help message
|
|
|
|
|
help-all Shows detailed help for all commands and flags
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
Detailed usage
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
####################################################################################################
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Detailed usage for command: shell
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
breeze shell [FLAGS] [-- <EXTRA_ARGS>]
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
This is default subcommand if no subcommand is used.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
|
2020-02-25 00:31:50 +03:00
|
|
|
|
workers, interact with the database, run DAGs etc. It is the default command if no command
|
|
|
|
|
is selected. The shell is executed in the container and in case integrations are chosen,
|
|
|
|
|
the integrations will be started as separated docker containers - under the docker-compose
|
|
|
|
|
supervision. Local sources are by default mounted to within the container so you can edit
|
|
|
|
|
them locally and run tests immediately in the container. Several folders ('files', 'dist')
|
|
|
|
|
are also mounted so that you can exchange files between the host and container.
|
|
|
|
|
|
|
|
|
|
The 'files/airflow-breeze-config/variables.env' file can contain additional variables
|
|
|
|
|
and setup. This file is automatically sourced when you enter the container. Database
|
|
|
|
|
and webserver ports are forwarded to appropriate database/webserver so that you can
|
|
|
|
|
connect to it from your host environment.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
|
|
|
|
|
especially useful to pass bash options, for example -c to execute command:
|
|
|
|
|
|
|
|
|
|
'breeze shell -- -c "ls -la"'
|
2020-08-19 01:53:04 +03:00
|
|
|
|
'breeze -- -c "ls -la"'
|
|
|
|
|
|
|
|
|
|
For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
|
|
|
|
|
the repository to pull from. For GitHub repository, the --github-repository
|
|
|
|
|
flag can be used for the same purpose. You can also use
|
|
|
|
|
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image
|
|
|
|
|
with specific COMMIT_SHA tag or RUN_ID.
|
|
|
|
|
|
|
|
|
|
'breeze shell \
|
|
|
|
|
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
|
|
|
|
|
'breeze \
|
|
|
|
|
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
|
|
|
|
|
'breeze shell \
|
|
|
|
|
--github-image-id 209845560' - pull/use image with RUN_ID
|
|
|
|
|
'breeze \
|
|
|
|
|
--github-image-id 209845560' - pull/use image with RUN_ID
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: build-docs
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-08-18 20:40:18 +03:00
|
|
|
|
breeze build-docs [-- <EXTRA_ARGS>]
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Builds Airflow documentation. The documentation is build inside docker container - to
|
2020-02-25 00:31:50 +03:00
|
|
|
|
maintain the same build environment for everyone. Appropriate sources are mapped from
|
|
|
|
|
the host to the container so that latest sources are used. The folders where documentation
|
2020-08-18 20:40:18 +03:00
|
|
|
|
is generated ('docs/_build') are also mounted to the container - this way results of
|
2020-02-25 00:31:50 +03:00
|
|
|
|
the documentation build is available in the host.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-12-10 18:57:21 +03:00
|
|
|
|
The possible extra args are: --docs-only, --spellcheck-only, --package-filter, --help
|
2020-08-18 20:40:18 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: build-image
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze build-image [FLAGS]
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-04-02 20:52:11 +03:00
|
|
|
|
Builds docker image (CI or production) without entering the container. You can pass
|
|
|
|
|
additional options to this command, such as '--force-build-image',
|
2020-06-28 18:38:17 +03:00
|
|
|
|
'--force-pull-image', '--python', '--build-cache-local' or '-build-cache-pulled'
|
|
|
|
|
in order to modify build behaviour.
|
|
|
|
|
|
2020-04-02 20:52:11 +03:00
|
|
|
|
You can also pass '--production-image' flag to build production image rather than CI image.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-08-19 01:53:04 +03:00
|
|
|
|
For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
|
|
|
|
|
the repository to pull from. For GitHub repository, the --github-repository
|
|
|
|
|
flag can be used for the same purpose. You can also use
|
|
|
|
|
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image with
|
|
|
|
|
specific COMMIT_SHA tag or RUN_ID.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
If specified, installs Airflow directly from PIP released version. This happens at
|
|
|
|
|
image building time in production image and at container entering time for CI image. One of:
|
|
|
|
|
|
2020-12-30 16:24:21 +03:00
|
|
|
|
2.0.0 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
|
2020-11-09 15:27:10 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
When 'none' is used, you can install airflow from local packages. When building image,
|
|
|
|
|
airflow package should be added to 'docker-context-files' and
|
|
|
|
|
--install-from-docker-context-files flag should be used. When running an image, airflow
|
|
|
|
|
package should be added to dist folder and --install-packages-from-dist flag should be used.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
|
2020-04-13 09:51:33 +03:00
|
|
|
|
If specified, installs Airflow directly from reference in GitHub. This happens at
|
|
|
|
|
image building time in production image and at container entering time for CI image.
|
2020-10-16 11:50:17 +03:00
|
|
|
|
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-10-07 03:00:00 +03:00
|
|
|
|
--no-rbac-ui
|
|
|
|
|
Disables RBAC UI when Airflow 1.10.* is installed.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--install-packages-from-dist
|
|
|
|
|
If specified it will look for packages placed in dist folder and it will install the
|
|
|
|
|
packages after installing Airflow. This is useful for testing provider
|
|
|
|
|
packages.
|
2020-10-10 01:41:11 +03:00
|
|
|
|
|
2020-12-28 22:08:18 +03:00
|
|
|
|
--upgrade-to-newer-dependencies
|
|
|
|
|
Upgrades PIP packages to latest versions available without looking at the constraints.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
-I, --production-image
|
|
|
|
|
Use production image for entering the environment and builds (not for tests).
|
|
|
|
|
|
|
|
|
|
-F, --force-build-images
|
|
|
|
|
Forces building of the local docker images. The images are rebuilt
|
|
|
|
|
automatically for the first time or when changes are detected in
|
|
|
|
|
package-related files, but you can force it using this flag.
|
|
|
|
|
|
|
|
|
|
-P, --force-pull-images
|
|
|
|
|
Forces pulling of images from DockerHub before building to populate cache. The
|
|
|
|
|
images are pulled by default only for the first time you run the
|
|
|
|
|
environment, later the locally build images are used as cache.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
Customization options:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-E, --extras EXTRAS
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Extras to pass to build images The default are different for CI and production images:
|
|
|
|
|
|
|
|
|
|
CI image:
|
|
|
|
|
devel_ci
|
|
|
|
|
|
|
|
|
|
Production image:
|
2020-12-07 01:36:33 +03:00
|
|
|
|
async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,
|
2020-12-28 19:07:00 +03:00
|
|
|
|
http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,
|
2020-12-07 01:36:33 +03:00
|
|
|
|
virtualenv
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--image-tag TAG
|
|
|
|
|
Additional tag in the image.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--skip-installing-airflow-providers-from-sources
|
2020-10-17 12:16:28 +03:00
|
|
|
|
By default 'pip install' in Airflow 2.0 installs only the provider packages that
|
2020-12-07 01:36:33 +03:00
|
|
|
|
are needed by the extras. When you build image during the development (which is
|
|
|
|
|
default in Breeze) all providers are installed by default from sources.
|
|
|
|
|
You can disable it by adding this flag but then you have to install providers from
|
|
|
|
|
wheel packages via --install-packages-from-dist flag.
|
2020-10-17 12:16:28 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--disable-pypi-when-building
|
|
|
|
|
Disable installing Airflow from pypi when building. If you use this flag and want
|
|
|
|
|
to install Airflow, you have to install it from packages placed in
|
|
|
|
|
'docker-context-files' and use --install-from-local-files-when-building flag.
|
2020-10-15 16:19:18 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-extras ADDITIONAL_EXTRAS
|
2020-05-27 18:09:11 +03:00
|
|
|
|
Additional extras to pass to build images The default is no additional extras.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-python-deps ADDITIONAL_PYTHON_DEPS
|
2020-05-27 18:09:11 +03:00
|
|
|
|
Additional python dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--dev-apt-command DEV_APT_COMMAND
|
|
|
|
|
The basic command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
|
|
|
|
|
Additional command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--dev-apt-deps DEV_APT_DEPS
|
|
|
|
|
The basic apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_DEPS
|
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
|
|
|
|
|
Additional environment variables set when adding dev dependencies.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-command RUNTIME_APT_COMMAND
|
|
|
|
|
The basic command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
|
|
|
|
|
Additional command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
The basic apt runtime dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt runtime dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
Additional environment variables set when adding runtime dependencies.
|
|
|
|
|
|
|
|
|
|
Build options:
|
2020-09-27 19:00:03 +03:00
|
|
|
|
|
2020-09-27 19:56:58 +03:00
|
|
|
|
--disable-mysql-client-installation
|
|
|
|
|
Disables installation of the mysql client which might be problematic if you are building
|
|
|
|
|
image in controlled environment. Only valid for production image.
|
|
|
|
|
|
2020-10-10 13:58:09 +03:00
|
|
|
|
--constraints-location
|
|
|
|
|
Url to the constraints file. In case of the production image it can also be a path to the
|
|
|
|
|
constraint file placed in 'docker-context-files' folder, in which case it has to be
|
|
|
|
|
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
|
|
|
|
|
|
|
|
|
|
--disable-pip-cache
|
2020-10-21 15:32:41 +03:00
|
|
|
|
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--install-from-local-files-when-building
|
2020-10-15 16:19:18 +03:00
|
|
|
|
This flag is used during image building. If it is used additionally to installing
|
2020-12-07 01:36:33 +03:00
|
|
|
|
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
|
2020-10-15 16:19:18 +03:00
|
|
|
|
in the 'docker-context-files' folder. The same flag can be used during entering the image in
|
2020-12-07 01:36:33 +03:00
|
|
|
|
the CI image - in this case also the .whl and .tar.gz files will be installed automatically
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
-C, --force-clean-images
|
|
|
|
|
Force build images with cache disabled. This will remove the pulled or build images
|
|
|
|
|
and start building images from scratch. This might take a long time.
|
|
|
|
|
|
2020-08-20 10:57:07 +03:00
|
|
|
|
-r, --skip-rebuild-check
|
|
|
|
|
Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
|
|
|
|
|
|
2020-06-28 18:38:17 +03:00
|
|
|
|
-L, --build-cache-local
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Uses local cache to build images. No pulled images will be used, but results of local
|
2020-06-28 18:38:17 +03:00
|
|
|
|
builds in the Docker cache are used instead. This will take longer than when the pulled
|
|
|
|
|
cache is used for the first time, but subsequent '--build-cache-local' builds will be
|
|
|
|
|
faster as they will use mostly the locally build cache.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the Production image builds.
|
|
|
|
|
|
|
|
|
|
-U, --build-cache-pulled
|
|
|
|
|
Uses images pulled from registry (either DockerHub or GitHub depending on
|
|
|
|
|
--github-registry flag) to build images. The pulled images will be used as cache.
|
|
|
|
|
Those builds are usually faster than when ''--build-cache-local'' with the exception if
|
|
|
|
|
the registry images are not yet updated. The DockerHub images are updated nightly and the
|
|
|
|
|
GitHub images are updated after merges to master so it might be that the images are still
|
|
|
|
|
outdated vs. the latest version of the Dockerfiles you are using. In this case, the
|
|
|
|
|
''--build-cache-local'' might be faster, especially if you iterate and change the
|
|
|
|
|
Dockerfiles yourself.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the CI image builds.
|
|
|
|
|
|
|
|
|
|
-X, --build-cache-disabled
|
|
|
|
|
Disables cache during docker builds. This is useful if you want to make sure you want to
|
|
|
|
|
rebuild everything from scratch.
|
|
|
|
|
|
|
|
|
|
This strategy is used by default for both Production and CI images for the scheduled
|
|
|
|
|
(nightly) builds in CI.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-D, --dockerhub-user DOCKERHUB_USER
|
2020-04-13 09:51:33 +03:00
|
|
|
|
DockerHub user used to pull, push and build images. Default: apache.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-H, --dockerhub-repo DOCKERHUB_REPO
|
2020-04-13 09:51:33 +03:00
|
|
|
|
DockerHub repository used to pull, push, build images. Default: airflow.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-c, --github-registry GITHUB_REGISTRY
|
2020-06-16 13:36:46 +03:00
|
|
|
|
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
|
2020-08-19 01:53:04 +03:00
|
|
|
|
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
|
2020-06-16 13:36:46 +03:00
|
|
|
|
and you need to be committer to push to Apache Airflow' GitHub registry.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-g, --github-repository GITHUB_REPOSITORY
|
2020-08-19 01:53:04 +03:00
|
|
|
|
GitHub repository used to pull, push images when cache is used.
|
|
|
|
|
Default: apache/airflow.
|
|
|
|
|
|
|
|
|
|
If you use this flag, automatically --github-registry flag is enabled.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-s, --github-image-id COMMIT_SHA|RUN_ID
|
2020-08-19 01:53:04 +03:00
|
|
|
|
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
|
|
|
|
|
to be able to easily find the image for particular CI runs. Once you know the
|
|
|
|
|
<RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
|
|
|
|
|
automatically pull and use that image so that you can easily reproduce a problem
|
|
|
|
|
that occurred in CI.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-08-19 01:53:04 +03:00
|
|
|
|
If you use this flag, automatically --github-registry is enabled.
|
|
|
|
|
|
2020-11-27 22:06:44 +03:00
|
|
|
|
|
2020-08-19 01:53:04 +03:00
|
|
|
|
Default: latest.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: cleanup-image
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze cleanup-image [FLAGS]
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Removes the breeze-related images created in your local docker image cache. This will
|
|
|
|
|
not reclaim space in docker cache. You need to 'docker system prune' (optionally
|
|
|
|
|
with --all) to reclaim that space.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
-I, --production-image
|
|
|
|
|
Use production image for entering the environment and builds (not for tests).
|
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: exec
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
breeze exec [-- <EXTRA_ARGS>]
|
2020-03-07 22:18:51 +03:00
|
|
|
|
|
|
|
|
|
Execs into interactive shell to an already running container. The container mus be started
|
|
|
|
|
already by breeze shell command. If you are not familiar with tmux, this is the best
|
|
|
|
|
way to run multiple processes in the same container at the same time for example scheduler,
|
|
|
|
|
webserver, workers, database console and interactive terminal.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
Detailed usage for command: generate-constraints
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
breeze generate-constraints [FLAGS]
|
2020-03-27 17:27:53 +03:00
|
|
|
|
|
2020-07-20 15:36:03 +03:00
|
|
|
|
Generates pinned constraint files from setup.py. Those files are generated in files folder
|
|
|
|
|
- separate files for different python version. Those constraint files when pushed to orphan
|
2020-12-30 16:24:21 +03:00
|
|
|
|
constraints-master, constraints-2-0 and constraints-1-10 branches are used to generate
|
|
|
|
|
repeatable CI builds as well as run repeatable production image builds. You can use those
|
|
|
|
|
constraints to predictably install released Airflow versions. This is mainly used to test
|
|
|
|
|
the constraint generation - constraints are pushed to the orphan branches by a
|
|
|
|
|
successful scheduled CRON job in CI automatically.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
2020-05-15 20:00:15 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
2020-05-15 20:00:15 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
One of:
|
2020-05-19 15:17:22 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-05-15 20:00:15 +03:00
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-05-15 20:00:15 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-06-02 11:24:47 +03:00
|
|
|
|
Detailed usage for command: push-image
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze push_image [FLAGS]
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
|
|
|
|
Pushes images to docker registry. You can push the images to DockerHub registry (default)
|
2020-06-16 13:36:46 +03:00
|
|
|
|
or to the GitHub registry (if --github-registry flag is used).
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
|
|
|
|
For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
|
2020-08-19 01:53:04 +03:00
|
|
|
|
the repository to push to. For GitHub repository, the --github-repository
|
|
|
|
|
flag can be used for the same purpose. You can also add
|
|
|
|
|
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to push image with specific
|
|
|
|
|
SHA tag or run id. In case you specify --github-repository or --github-image-id, you
|
|
|
|
|
do not need to specify --github-registry flag.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
|
|
|
|
You can also add --production-image flag to switch to production image (default is CI one)
|
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
|
|
|
|
'breeze push-image' or
|
|
|
|
|
'breeze push-image --dockerhub-user user' to push to your private registry or
|
|
|
|
|
'breeze push-image --production-image' - to push production image or
|
2020-06-16 13:36:46 +03:00
|
|
|
|
'breeze push-image --github-registry' - to push to GitHub image registry or
|
2020-08-19 01:53:04 +03:00
|
|
|
|
'breeze push-image \
|
|
|
|
|
--github-repository user/airflow' - to push to your user's fork
|
|
|
|
|
'breeze push-image \
|
|
|
|
|
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - to push with COMMIT_SHA
|
|
|
|
|
'breeze push-image \
|
|
|
|
|
--github-image-id 209845560' - to push with RUN_ID
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-D, --dockerhub-user DOCKERHUB_USER
|
2020-06-02 11:24:47 +03:00
|
|
|
|
DockerHub user used to pull, push and build images. Default: apache.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-H, --dockerhub-repo DOCKERHUB_REPO
|
2020-06-02 11:24:47 +03:00
|
|
|
|
DockerHub repository used to pull, push, build images. Default: airflow.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-c, --github-registry GITHUB_REGISTRY
|
2020-06-16 13:36:46 +03:00
|
|
|
|
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
|
2020-08-19 01:53:04 +03:00
|
|
|
|
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
|
2020-06-16 13:36:46 +03:00
|
|
|
|
and you need to be committer to push to Apache Airflow' GitHub registry.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-g, --github-repository GITHUB_REPOSITORY
|
2020-08-19 01:53:04 +03:00
|
|
|
|
GitHub repository used to pull, push images when cache is used.
|
|
|
|
|
Default: apache/airflow.
|
|
|
|
|
|
|
|
|
|
If you use this flag, automatically --github-registry flag is enabled.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-s, --github-image-id COMMIT_SHA|RUN_ID
|
2020-08-19 01:53:04 +03:00
|
|
|
|
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
|
|
|
|
|
to be able to easily find the image for particular CI runs. Once you know the
|
|
|
|
|
<RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
|
|
|
|
|
automatically pull and use that image so that you can easily reproduce a problem
|
|
|
|
|
that occurred in CI.
|
|
|
|
|
|
|
|
|
|
If you use this flag, automatically --github-registry is enabled.
|
|
|
|
|
|
2020-11-27 22:06:44 +03:00
|
|
|
|
|
2020-08-19 01:53:04 +03:00
|
|
|
|
Default: latest.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-06-02 11:24:47 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Detailed usage for command: initialize-local-virtualenv
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze initialize-local-virtualenv [FLAGS]
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-03-22 15:34:48 +03:00
|
|
|
|
Initializes locally created virtualenv installing all dependencies of Airflow
|
2020-07-20 15:36:03 +03:00
|
|
|
|
taking into account the constraints for the version specified.
|
|
|
|
|
This local virtualenv can be used to aid auto-completion and IDE support as
|
2020-02-25 00:31:50 +03:00
|
|
|
|
well as run unit tests directly from the IDE. You need to have virtualenv
|
|
|
|
|
activated before running this command.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
Detailed usage for command: prepare-airflow-packages
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
breeze prepare-airflow-packages [FLAGS]
|
|
|
|
|
|
|
|
|
|
Prepares airflow packages (sdist and wheel) in dist folder. Note that
|
|
|
|
|
prepare-provider-packages command cleans up the dist folder, so if you want also
|
|
|
|
|
to generate provider packages, make sure you run prepare-provider-packages first,
|
|
|
|
|
and prepare-airflow-packages second.
|
|
|
|
|
|
|
|
|
|
General form:
|
|
|
|
|
|
|
|
|
|
'breeze prepare-airflow-packages
|
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
|
|
|
|
--package-format PACKAGE_FORMAT
|
|
|
|
|
|
|
|
|
|
Chooses format of packages to prepare.
|
|
|
|
|
|
|
|
|
|
One of:
|
|
|
|
|
|
|
|
|
|
wheel,sdist,both
|
|
|
|
|
|
|
|
|
|
Default:
|
|
|
|
|
|
|
|
|
|
-v, --verbose
|
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Detailed usage for command: setup-autocomplete
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze setup-autocomplete
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
|
|
|
|
|
shell and when typing breeze command <TAB> will provide autocomplete for
|
|
|
|
|
parameters and values.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-09-27 19:31:50 +03:00
|
|
|
|
Detailed usage for command: start-airflow
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
breeze start-airflow
|
|
|
|
|
|
|
|
|
|
Like the Shell command this will enter the interactive shell, but it will also start
|
|
|
|
|
automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
|
|
|
|
|
can also observe what is happening in your Airflow.
|
|
|
|
|
|
|
|
|
|
This is a convenient way to setup a development environment. Your dags will be loaded from the
|
|
|
|
|
folder 'files/dags' on your host machine (it could take some times).
|
|
|
|
|
|
|
|
|
|
If you want to load default connections and example dags you can use the dedicated flags.
|
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
|
|
|
|
--load-example-dags
|
|
|
|
|
Include Airflow example dags.
|
|
|
|
|
|
|
|
|
|
--load-default-connections
|
|
|
|
|
Include Airflow Default Connections.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Detailed usage for command: stop
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze stop
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Brings down running docker compose environment. When you start the environment, the docker
|
|
|
|
|
containers will continue running so that startup time is shorter. But they take quite a lot of
|
|
|
|
|
memory and CPU. This command stops all running containers from the environment.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-10-18 17:39:44 +03:00
|
|
|
|
Flags:
|
|
|
|
|
|
|
|
|
|
--preserve-volumes
|
|
|
|
|
Use this flag if you would like to preserve data volumes from the databases used
|
|
|
|
|
by the integrations. By default, those volumes are deleted, so when you run 'stop'
|
|
|
|
|
or 'restart' commands you start from scratch, but by using this flag you can
|
|
|
|
|
preserve them. If you want to delete those volumes after stopping Breeze, just
|
|
|
|
|
run the 'breeze stop' again without this flag.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: restart
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze restart [FLAGS]
|
2020-02-27 12:44:43 +03:00
|
|
|
|
|
|
|
|
|
Restarts running docker compose environment. When you restart the environment, the docker
|
|
|
|
|
containers will be restarted. That includes cleaning up the databases. This is
|
2020-04-13 09:51:33 +03:00
|
|
|
|
especially useful if you switch between different versions of Airflow.
|
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-10-18 17:39:44 +03:00
|
|
|
|
--preserve-volumes
|
|
|
|
|
Use this flag if you would like to preserve data volumes from the databases used
|
|
|
|
|
by the integrations. By default, those volumes are deleted, so when you run 'stop'
|
|
|
|
|
or 'restart' commands you start from scratch, but by using this flag you can
|
|
|
|
|
preserve them. If you want to delete those volumes after stopping Breeze, just
|
|
|
|
|
run the 'breeze stop' again without this flag.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: toggle-suppress-cheatsheet
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze toggle-suppress-cheatsheet
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Toggles on/off cheatsheet displayed before starting bash shell.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: toggle-suppress-asciiart
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze toggle-suppress-asciiart
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Toggles on/off asciiart displayed before starting bash shell.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: docker-compose
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
breeze docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]
|
2019-11-04 12:37:31 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Run docker-compose command instead of entering the environment. Use 'help' as command
|
|
|
|
|
to see available commands. The <EXTRA_ARGS> passed after -- are treated
|
|
|
|
|
as additional options passed to docker-compose. For example
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
'breeze docker-compose pull -- --ignore-pull-failures'
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-b, --backend BACKEND
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Backend to use for tests - it determines which database is used.
|
|
|
|
|
One of:
|
|
|
|
|
|
|
|
|
|
sqlite mysql postgres
|
|
|
|
|
|
|
|
|
|
Default: sqlite
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--postgres-version POSTGRES_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Postgres version used. One of:
|
|
|
|
|
|
2020-10-24 15:39:01 +03:00
|
|
|
|
9.6 10 11 12 13
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--mysql-version MYSQL_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Mysql version used. One of:
|
|
|
|
|
|
2020-10-04 14:45:05 +03:00
|
|
|
|
5.7 8
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Detailed usage for command: kind-cluster
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
breeze kind-cluster [FLAGS] OPERATION
|
|
|
|
|
|
|
|
|
|
Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
|
|
|
|
|
It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
|
2020-08-19 01:53:04 +03:00
|
|
|
|
This enables you to run tests inside the breeze environment with latest airflow images.
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Note that in case of deploying airflow, the first step is to rebuild the image and loading it
|
|
|
|
|
to the cluster so you can also pass appropriate build image flags that will influence
|
|
|
|
|
rebuilding the production image. Operation is one of:
|
|
|
|
|
|
2020-11-11 19:15:02 +03:00
|
|
|
|
start stop restart status deploy test shell k9s
|
|
|
|
|
|
|
|
|
|
The last two operations - shell and k9s allow you to perform interactive testing with
|
|
|
|
|
kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
|
|
|
|
|
another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
|
|
|
|
|
way to debug the kubernetes deployments.
|
|
|
|
|
|
|
|
|
|
You can read more about k9s at https://k9scli.io/
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
-F, --force-build-images
|
|
|
|
|
Forces building of the local docker images. The images are rebuilt
|
|
|
|
|
automatically for the first time or when changes are detected in
|
|
|
|
|
package-related files, but you can force it using this flag.
|
|
|
|
|
|
|
|
|
|
-P, --force-pull-images
|
|
|
|
|
Forces pulling of images from DockerHub before building to populate cache. The
|
|
|
|
|
images are pulled by default only for the first time you run the
|
|
|
|
|
environment, later the locally build images are used as cache.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
Customization options:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-E, --extras EXTRAS
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Extras to pass to build images The default are different for CI and production images:
|
|
|
|
|
|
|
|
|
|
CI image:
|
|
|
|
|
devel_ci
|
|
|
|
|
|
|
|
|
|
Production image:
|
2020-12-07 01:36:33 +03:00
|
|
|
|
async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,
|
2020-12-28 19:07:00 +03:00
|
|
|
|
http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,
|
2020-12-07 01:36:33 +03:00
|
|
|
|
virtualenv
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--image-tag TAG
|
|
|
|
|
Additional tag in the image.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--skip-installing-airflow-providers-from-sources
|
2020-10-17 12:16:28 +03:00
|
|
|
|
By default 'pip install' in Airflow 2.0 installs only the provider packages that
|
2020-12-07 01:36:33 +03:00
|
|
|
|
are needed by the extras. When you build image during the development (which is
|
|
|
|
|
default in Breeze) all providers are installed by default from sources.
|
|
|
|
|
You can disable it by adding this flag but then you have to install providers from
|
|
|
|
|
wheel packages via --install-packages-from-dist flag.
|
2020-10-17 12:16:28 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--disable-pypi-when-building
|
|
|
|
|
Disable installing Airflow from pypi when building. If you use this flag and want
|
|
|
|
|
to install Airflow, you have to install it from packages placed in
|
|
|
|
|
'docker-context-files' and use --install-from-local-files-when-building flag.
|
2020-10-15 16:19:18 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-extras ADDITIONAL_EXTRAS
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Additional extras to pass to build images The default is no additional extras.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-python-deps ADDITIONAL_PYTHON_DEPS
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Additional python dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--dev-apt-command DEV_APT_COMMAND
|
|
|
|
|
The basic command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
|
|
|
|
|
Additional command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
|
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--dev-apt-deps DEV_APT_DEPS
|
|
|
|
|
The basic apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
|
|
|
|
|
Additional environment variables set when adding dev dependencies.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-command RUNTIME_APT_COMMAND
|
|
|
|
|
The basic command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
|
|
|
|
|
Additional command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
The basic apt runtime dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt runtime dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
Additional environment variables set when adding runtime dependencies.
|
|
|
|
|
|
|
|
|
|
Build options:
|
2020-09-27 19:00:03 +03:00
|
|
|
|
|
2020-09-27 19:56:58 +03:00
|
|
|
|
--disable-mysql-client-installation
|
|
|
|
|
Disables installation of the mysql client which might be problematic if you are building
|
|
|
|
|
image in controlled environment. Only valid for production image.
|
|
|
|
|
|
2020-10-10 13:58:09 +03:00
|
|
|
|
--constraints-location
|
|
|
|
|
Url to the constraints file. In case of the production image it can also be a path to the
|
|
|
|
|
constraint file placed in 'docker-context-files' folder, in which case it has to be
|
|
|
|
|
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
|
|
|
|
|
|
|
|
|
|
--disable-pip-cache
|
2020-10-21 15:32:41 +03:00
|
|
|
|
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--install-from-local-files-when-building
|
2020-10-15 16:19:18 +03:00
|
|
|
|
This flag is used during image building. If it is used additionally to installing
|
2020-12-07 01:36:33 +03:00
|
|
|
|
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
|
2020-10-15 16:19:18 +03:00
|
|
|
|
in the 'docker-context-files' folder. The same flag can be used during entering the image in
|
2020-12-07 01:36:33 +03:00
|
|
|
|
the CI image - in this case also the .whl and .tar.gz files will be installed automatically
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
-C, --force-clean-images
|
|
|
|
|
Force build images with cache disabled. This will remove the pulled or build images
|
|
|
|
|
and start building images from scratch. This might take a long time.
|
|
|
|
|
|
2020-08-20 10:57:07 +03:00
|
|
|
|
-r, --skip-rebuild-check
|
|
|
|
|
Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
|
|
|
|
|
|
2020-06-28 18:38:17 +03:00
|
|
|
|
-L, --build-cache-local
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Uses local cache to build images. No pulled images will be used, but results of local
|
2020-06-28 18:38:17 +03:00
|
|
|
|
builds in the Docker cache are used instead. This will take longer than when the pulled
|
|
|
|
|
cache is used for the first time, but subsequent '--build-cache-local' builds will be
|
|
|
|
|
faster as they will use mostly the locally build cache.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the Production image builds.
|
|
|
|
|
|
|
|
|
|
-U, --build-cache-pulled
|
|
|
|
|
Uses images pulled from registry (either DockerHub or GitHub depending on
|
|
|
|
|
--github-registry flag) to build images. The pulled images will be used as cache.
|
|
|
|
|
Those builds are usually faster than when ''--build-cache-local'' with the exception if
|
|
|
|
|
the registry images are not yet updated. The DockerHub images are updated nightly and the
|
|
|
|
|
GitHub images are updated after merges to master so it might be that the images are still
|
|
|
|
|
outdated vs. the latest version of the Dockerfiles you are using. In this case, the
|
|
|
|
|
''--build-cache-local'' might be faster, especially if you iterate and change the
|
|
|
|
|
Dockerfiles yourself.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the CI image builds.
|
|
|
|
|
|
|
|
|
|
-X, --build-cache-disabled
|
|
|
|
|
Disables cache during docker builds. This is useful if you want to make sure you want to
|
|
|
|
|
rebuild everything from scratch.
|
|
|
|
|
|
|
|
|
|
This strategy is used by default for both Production and CI images for the scheduled
|
|
|
|
|
(nightly) builds in CI.
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
Detailed usage for command: prepare-provider-readme
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
breeze prepare-provider-readme [FLAGS] [YYYY.MM.DD] [PACKAGE_ID ...]
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Prepares README.md files for backport packages. You can provide (after --) optional version
|
|
|
|
|
in the form of YYYY.MM.DD, optionally followed by the list of packages to generate readme for.
|
|
|
|
|
If the first parameter is not formatted as a date, then today is regenerated.
|
|
|
|
|
If no packages are specified, readme for all packages are generated.
|
|
|
|
|
If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Examples:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
'breeze prepare-provider-readme' or
|
|
|
|
|
'breeze prepare-provider-readme 2020.05.10' or
|
|
|
|
|
'breeze prepare-provider-readme 2020.05.10 https google amazon'
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
General form:
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
'breeze prepare-provider-readme YYYY.MM.DD <PACKAGE_ID> ...'
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
* YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date
|
|
|
|
|
cannot be earlier than the already released version (the script will fail if it
|
|
|
|
|
will be). It can be set in the future anticipating the future release date.
|
|
|
|
|
|
|
|
|
|
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
|
|
|
|
|
'google' but in several cases, it might be one level deeper separated with
|
|
|
|
|
'.' for example 'apache.hive'
|
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
Detailed usage for command: prepare-provider-packages
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
breeze prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Prepares backport packages. You can provide (after --) optional list of packages to prepare.
|
|
|
|
|
If no packages are specified, readme for all packages are generated. You can specify optional
|
|
|
|
|
--version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
|
2020-10-19 13:32:07 +03:00
|
|
|
|
--version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
|
2020-12-07 01:36:33 +03:00
|
|
|
|
provide both suffixes in case you prepare alpha/beta versions. The packages are prepared in
|
|
|
|
|
dist folder. Note that this command also cleans up dist folder before generating the packages
|
|
|
|
|
so that you do not have accidental files there. This will delete airflow package if it is
|
|
|
|
|
prepared there so make sure you run prepare-provider-packages first,
|
|
|
|
|
and prepare-airflow-packages second.
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
2020-10-12 17:29:48 +03:00
|
|
|
|
'breeze prepare-provider-packages' or
|
|
|
|
|
'breeze prepare-provider-packages google' or
|
2020-12-07 01:36:33 +03:00
|
|
|
|
'breeze prepare-provider-packages --package-format both google' or
|
2020-10-12 17:29:48 +03:00
|
|
|
|
'breeze prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
|
|
|
|
|
'breeze prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'
|
2020-10-19 13:32:07 +03:00
|
|
|
|
'breeze prepare-provider-packages --version-suffix-for-pypi a1
|
|
|
|
|
--version-suffix-for-svn a1 http google amazon'
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
General form:
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
'breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] \
|
2020-06-16 13:36:46 +03:00
|
|
|
|
[--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
|
|
|
|
|
|
|
|
|
|
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
|
|
|
|
|
'google'), but in several cases, it might be one level deeper separated with '.'
|
|
|
|
|
for example 'apache.hive'
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Flags:
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--package-format PACKAGE_FORMAT
|
|
|
|
|
|
|
|
|
|
Chooses format of packages to prepare.
|
|
|
|
|
|
|
|
|
|
One of:
|
|
|
|
|
|
|
|
|
|
wheel,sdist,both
|
|
|
|
|
|
|
|
|
|
Default:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-S, --version-suffix-for-pypi SUFFIX
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Adds optional suffix to the version in the generated backport package. It can be used
|
|
|
|
|
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-N, --version-suffix-for-svn SUFFIX
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Adds optional suffix to the generated names of package. It can be used to generate
|
|
|
|
|
rc1/rc2 ... versions of the packages to be uploaded to SVN.
|
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: static-check
|
|
|
|
|
|
|
|
|
|
|
2020-09-02 22:55:50 +03:00
|
|
|
|
breeze static-check [FLAGS] static_check [-- <EXTRA_ARGS>]
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Run selected static checks for currently changed files. You should specify static check that
|
2020-02-25 00:31:50 +03:00
|
|
|
|
you would like to run or 'all' to run all checks. One of:
|
|
|
|
|
|
2020-11-17 20:48:57 +03:00
|
|
|
|
all all-but-pylint airflow-config-yaml airflow-providers-available
|
|
|
|
|
airflow-provider-yaml-files-ok base-operator bats-tests bats-in-container-tests
|
|
|
|
|
black build build-providers-dependencies check-apache-license check-builtin-literals
|
|
|
|
|
check-executables-have-shebangs check-hooks-apply check-integrations
|
|
|
|
|
check-merge-conflict check-xml consistent-pylint daysago-import-check
|
|
|
|
|
debug-statements detect-private-key doctoc dont-use-safe-filter end-of-file-fixer
|
2020-12-16 11:03:48 +03:00
|
|
|
|
fix-encoding-pragma flake8 forbid-tabs helm-lint identity
|
|
|
|
|
incorrect-use-of-LoggingMixin insert-license isort json-schema language-matters
|
|
|
|
|
lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending mypy mypy-helm
|
|
|
|
|
no-providers-in-core-examples no-relative-imports pre-commit-descriptions
|
2020-12-26 13:24:27 +03:00
|
|
|
|
pre-commit-hook-names provide-create-sessions providers-init-file provider-yamls
|
|
|
|
|
pydevd pydocstyle pylint pylint-tests python-no-log-warn pyupgrade
|
|
|
|
|
restrict-start_date rst-backticks setup-order setup-extra-packages shellcheck
|
|
|
|
|
sort-in-the-wild stylelint trailing-whitespace update-breeze-file update-extras
|
|
|
|
|
update-local-yml-file update-setup-cfg-file version-sync yamllint
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
You can pass extra arguments including options to to the pre-commit framework as
|
2020-02-25 00:31:50 +03:00
|
|
|
|
<EXTRA_ARGS> passed after --. For example:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
'breeze static-check mypy' or
|
|
|
|
|
'breeze static-check mypy -- --files tests/core.py'
|
|
|
|
|
'breeze static-check mypy -- --all-files'
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-11-26 16:54:23 +03:00
|
|
|
|
To check all files that differ between you current branch and master run:
|
|
|
|
|
|
|
|
|
|
'breeze static-check all -- --from-ref $(git merge-base master HEAD) --to-ref HEAD'
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
You can see all the options by adding --help EXTRA_ARG:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
'breeze static-check mypy -- --help'
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
Detailed usage for command: tests
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
breeze tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Run the specified unit test target. There might be multiple
|
|
|
|
|
targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
|
2020-06-16 13:36:46 +03:00
|
|
|
|
as additional options passed to pytest. You can pass 'tests' as target to
|
|
|
|
|
run all tests. For example:
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-11-09 12:34:06 +03:00
|
|
|
|
'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
|
2020-06-16 13:36:46 +03:00
|
|
|
|
'breeze tests tests
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
Flags:
|
|
|
|
|
|
2020-10-11 17:40:31 +03:00
|
|
|
|
--test-type TEST_TYPE
|
|
|
|
|
Type of the test to run. One of:
|
|
|
|
|
|
2020-10-28 04:29:47 +03:00
|
|
|
|
All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL,Helm
|
2020-10-11 17:40:31 +03:00
|
|
|
|
|
|
|
|
|
Default: All
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: flags
|
|
|
|
|
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Explains in detail all the flags that can be used with breeze.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: help
|
|
|
|
|
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
breeze help
|
|
|
|
|
|
|
|
|
|
Shows general help message for all commands.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Detailed usage for command: help-all
|
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
|
|
|
|
|
breeze help-all
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
|
|
|
|
Shows detailed help for all commands and flags.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
####################################################################################################
|
|
|
|
|
|
|
|
|
|
Summary of all flags supported by Breeze:
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Choose Airflow variant
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-p, --python PYTHON_MAJOR_MINOR_VERSION
|
2019-11-04 12:37:31 +03:00
|
|
|
|
Python version used for the image. This is always major/minor version.
|
2020-06-16 13:36:46 +03:00
|
|
|
|
|
|
|
|
|
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
|
|
|
|
|
--install-airflow-version or --install-airflow-reference flags.
|
|
|
|
|
|
2020-02-21 20:25:32 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-06-16 13:36:46 +03:00
|
|
|
|
2.7 3.5 3.6 3.7 3.8
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Choose backend to run for Airflow
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-b, --backend BACKEND
|
2019-11-04 12:37:31 +03:00
|
|
|
|
Backend to use for tests - it determines which database is used.
|
2020-02-21 20:25:32 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
sqlite mysql postgres
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
|
|
|
|
Default: sqlite
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--postgres-version POSTGRES_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Postgres version used. One of:
|
|
|
|
|
|
2020-10-24 15:39:01 +03:00
|
|
|
|
9.6 10 11 12 13
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--mysql-version MYSQL_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Mysql version used. One of:
|
|
|
|
|
|
2020-10-04 14:45:05 +03:00
|
|
|
|
5.7 8
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Enable production image
|
|
|
|
|
|
|
|
|
|
-I, --production-image
|
|
|
|
|
Use production image for entering the environment and builds (not for tests).
|
|
|
|
|
|
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Additional actions executed while entering breeze
|
|
|
|
|
|
2020-03-13 18:08:16 +03:00
|
|
|
|
-d, --db-reset
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Resets the database at entry to the environment. It will drop all the tables
|
2020-03-13 18:08:16 +03:00
|
|
|
|
and data and recreate the DB from scratch even if 'restart' command was not used.
|
|
|
|
|
Combined with 'restart' command it enters the environment in the state that is
|
2020-04-13 09:51:33 +03:00
|
|
|
|
ready to start Airflow webserver/scheduler/worker. Without the switch, the database
|
2020-03-13 18:08:16 +03:00
|
|
|
|
does not have any tables and you need to run reset db manually.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-i, --integration INTEGRATION
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Integration to start during tests - it determines which integrations are started
|
|
|
|
|
for integration tests. There can be more than one integration started, or all to
|
|
|
|
|
start all integrations. Selected integrations are not saved for future execution.
|
2020-02-21 20:25:32 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-12-30 15:29:40 +03:00
|
|
|
|
cassandra kerberos mongo openldap pinot presto rabbitmq redis all
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--init-script INIT_SCRIPT_FILE
|
2020-09-21 13:58:30 +03:00
|
|
|
|
Initialization script name - Sourced from files/airflow-breeze-config. Default value
|
2020-09-27 19:31:50 +03:00
|
|
|
|
init.sh. It will be executed after the environment is configured and started.
|
|
|
|
|
|
|
|
|
|
****************************************************************************************************
|
2020-10-18 17:39:44 +03:00
|
|
|
|
Additional actions executed while starting Airflow
|
2020-09-27 19:31:50 +03:00
|
|
|
|
|
|
|
|
|
--load-example-dags
|
|
|
|
|
Include Airflow example dags.
|
|
|
|
|
|
|
|
|
|
--load-default-connections
|
|
|
|
|
Include Airflow Default Connections.
|
2020-09-21 13:58:30 +03:00
|
|
|
|
|
2020-10-18 17:39:44 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Cleanup options when stopping Airflow
|
|
|
|
|
|
|
|
|
|
--preserve-volumes
|
|
|
|
|
Use this flag if you would like to preserve data volumes from the databases used
|
|
|
|
|
by the integrations. By default, those volumes are deleted, so when you run 'stop'
|
|
|
|
|
or 'restart' commands you start from scratch, but by using this flag you can
|
|
|
|
|
preserve them. If you want to delete those volumes after stopping Breeze, just
|
|
|
|
|
run the 'breeze stop' again without this flag.
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Kind kubernetes and Kubernetes tests configuration(optional)
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Configuration for the KinD Kubernetes cluster and tests:
|
2020-02-25 00:31:50 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-K, --kubernetes-mode KUBERNETES_MODE
|
2020-08-21 22:22:32 +03:00
|
|
|
|
Kubernetes mode - only used in case one of kind-cluster commands is used.
|
2020-02-21 20:25:32 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-09-21 21:02:04 +03:00
|
|
|
|
image
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
Default: image
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-V, --kubernetes-version KUBERNETES_VERSION
|
2020-08-21 22:22:32 +03:00
|
|
|
|
Kubernetes version - only used in case one of kind-cluster commands is used.
|
2020-02-21 20:25:32 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
2020-07-22 13:24:46 +03:00
|
|
|
|
v1.18.6 v1.17.5 v1.16.9
|
2020-07-01 15:50:30 +03:00
|
|
|
|
|
2020-07-22 13:24:46 +03:00
|
|
|
|
Default: v1.18.6
|
2020-07-01 15:50:30 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--kind-version KIND_VERSION
|
2020-08-21 22:22:32 +03:00
|
|
|
|
Kind version - only used in case one of kind-cluster commands is used.
|
2020-07-01 15:50:30 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
|
|
|
|
v0.8.0
|
|
|
|
|
|
|
|
|
|
Default: v0.8.0
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--helm-version HELM_VERSION
|
2020-08-21 22:22:32 +03:00
|
|
|
|
Helm version - only used in case one of kind-cluster commands is used.
|
2020-07-01 15:50:30 +03:00
|
|
|
|
One of:
|
|
|
|
|
|
|
|
|
|
v3.2.4
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-07-01 15:50:30 +03:00
|
|
|
|
Default: v3.2.4
|
2020-01-11 18:25:19 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Manage mounting local files
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-03-07 18:09:06 +03:00
|
|
|
|
-l, --skip-mounting-local-sources
|
2019-11-04 12:37:31 +03:00
|
|
|
|
Skips mounting local volume with sources - you get exactly what is in the
|
2020-04-13 09:51:33 +03:00
|
|
|
|
docker image rather than your current local sources of Airflow.
|
2020-03-15 01:57:47 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Assume answers to questions
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-11-04 12:37:31 +03:00
|
|
|
|
-y, --assume-yes
|
|
|
|
|
Assume 'yes' answer to all questions.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2019-11-04 12:37:31 +03:00
|
|
|
|
-n, --assume-no
|
|
|
|
|
Assume 'no' answer to all questions.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
-q, --assume-quit
|
|
|
|
|
Assume 'quit' answer to all questions.
|
|
|
|
|
|
|
|
|
|
****************************************************************************************************
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Choose different Airflow version to install or run
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
|
2020-04-13 09:51:33 +03:00
|
|
|
|
If specified, installs Airflow directly from PIP released version. This happens at
|
|
|
|
|
image building time in production image and at container entering time for CI image. One of:
|
|
|
|
|
|
2020-12-30 16:24:21 +03:00
|
|
|
|
2.0.0 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
|
2020-11-09 15:27:10 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
When 'none' is used, you can install airflow from local packages. When building image,
|
|
|
|
|
airflow package should be added to 'docker-context-files' and
|
|
|
|
|
--install-from-docker-context-files flag should be used. When running an image, airflow
|
|
|
|
|
package should be added to dist folder and --install-packages-from-dist flag should be used.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
|
2020-04-13 09:51:33 +03:00
|
|
|
|
If specified, installs Airflow directly from reference in GitHub. This happens at
|
|
|
|
|
image building time in production image and at container entering time for CI image.
|
2020-10-16 11:50:17 +03:00
|
|
|
|
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
|
2020-04-13 09:51:33 +03:00
|
|
|
|
|
2020-10-07 03:00:00 +03:00
|
|
|
|
--no-rbac-ui
|
|
|
|
|
Disables RBAC UI when Airflow 1.10.* is installed.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--install-packages-from-dist
|
|
|
|
|
If specified it will look for packages placed in dist folder and it will install the
|
|
|
|
|
packages after installing Airflow. This is useful for testing provider
|
|
|
|
|
packages.
|
2020-10-10 01:41:11 +03:00
|
|
|
|
|
2020-12-28 22:08:18 +03:00
|
|
|
|
--upgrade-to-newer-dependencies
|
|
|
|
|
Upgrades PIP packages to latest versions available without looking at the constraints.
|
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Credentials
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
|
|
|
|
-f, --forward-credentials
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Forwards host credentials to docker container. Use with care as it will make
|
|
|
|
|
your credentials available to everything you install in Docker.
|
2020-02-21 20:25:32 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Flags for building Docker images (both CI and production)
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
-F, --force-build-images
|
2019-11-04 12:37:31 +03:00
|
|
|
|
Forces building of the local docker images. The images are rebuilt
|
|
|
|
|
automatically for the first time or when changes are detected in
|
|
|
|
|
package-related files, but you can force it using this flag.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-04-02 20:52:11 +03:00
|
|
|
|
-P, --force-pull-images
|
2019-11-04 12:37:31 +03:00
|
|
|
|
Forces pulling of images from DockerHub before building to populate cache. The
|
|
|
|
|
images are pulled by default only for the first time you run the
|
|
|
|
|
environment, later the locally build images are used as cache.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
Customization options:
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-E, --extras EXTRAS
|
2020-04-02 20:52:11 +03:00
|
|
|
|
Extras to pass to build images The default are different for CI and production images:
|
|
|
|
|
|
|
|
|
|
CI image:
|
|
|
|
|
devel_ci
|
|
|
|
|
|
|
|
|
|
Production image:
|
2020-12-07 01:36:33 +03:00
|
|
|
|
async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,
|
2020-12-28 19:07:00 +03:00
|
|
|
|
http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,
|
2020-12-07 01:36:33 +03:00
|
|
|
|
virtualenv
|
2020-04-02 20:52:11 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--image-tag TAG
|
|
|
|
|
Additional tag in the image.
|
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--skip-installing-airflow-providers-from-sources
|
2020-10-17 12:16:28 +03:00
|
|
|
|
By default 'pip install' in Airflow 2.0 installs only the provider packages that
|
2020-12-07 01:36:33 +03:00
|
|
|
|
are needed by the extras. When you build image during the development (which is
|
|
|
|
|
default in Breeze) all providers are installed by default from sources.
|
|
|
|
|
You can disable it by adding this flag but then you have to install providers from
|
|
|
|
|
wheel packages via --install-packages-from-dist flag.
|
2020-10-17 12:16:28 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--disable-pypi-when-building
|
|
|
|
|
Disable installing Airflow from pypi when building. If you use this flag and want
|
|
|
|
|
to install Airflow, you have to install it from packages placed in
|
|
|
|
|
'docker-context-files' and use --install-from-local-files-when-building flag.
|
2020-10-15 16:19:18 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-extras ADDITIONAL_EXTRAS
|
2020-05-27 18:09:11 +03:00
|
|
|
|
Additional extras to pass to build images The default is no additional extras.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
--additional-python-deps ADDITIONAL_PYTHON_DEPS
|
2020-05-27 18:09:11 +03:00
|
|
|
|
Additional python dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--dev-apt-command DEV_APT_COMMAND
|
|
|
|
|
The basic command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
|
|
|
|
|
Additional command executed before dev apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
|
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--dev-apt-deps DEV_APT_DEPS
|
|
|
|
|
The basic apt dev dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-dev-apt-deps ADDITIONAL_DEV_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt dev dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
|
|
|
|
|
Additional environment variables set when adding dev dependencies.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-command RUNTIME_APT_COMMAND
|
|
|
|
|
The basic command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
|
|
|
|
|
Additional command executed before runtime apt deps are installed.
|
|
|
|
|
|
|
|
|
|
--runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
The basic apt runtime dependencies to use when building the images.
|
|
|
|
|
|
|
|
|
|
--additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
|
2020-06-11 19:53:26 +03:00
|
|
|
|
Additional apt runtime dependencies to use when building the images.
|
|
|
|
|
|
2020-09-29 16:30:00 +03:00
|
|
|
|
--additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
|
|
|
|
|
Additional environment variables set when adding runtime dependencies.
|
|
|
|
|
|
|
|
|
|
Build options:
|
2020-09-27 19:00:03 +03:00
|
|
|
|
|
2020-09-27 19:56:58 +03:00
|
|
|
|
--disable-mysql-client-installation
|
|
|
|
|
Disables installation of the mysql client which might be problematic if you are building
|
|
|
|
|
image in controlled environment. Only valid for production image.
|
|
|
|
|
|
2020-10-10 13:58:09 +03:00
|
|
|
|
--constraints-location
|
|
|
|
|
Url to the constraints file. In case of the production image it can also be a path to the
|
|
|
|
|
constraint file placed in 'docker-context-files' folder, in which case it has to be
|
|
|
|
|
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
|
|
|
|
|
|
|
|
|
|
--disable-pip-cache
|
2020-10-21 15:32:41 +03:00
|
|
|
|
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-12-07 01:36:33 +03:00
|
|
|
|
--install-from-local-files-when-building
|
2020-10-15 16:19:18 +03:00
|
|
|
|
This flag is used during image building. If it is used additionally to installing
|
2020-12-07 01:36:33 +03:00
|
|
|
|
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
|
2020-10-15 16:19:18 +03:00
|
|
|
|
in the 'docker-context-files' folder. The same flag can be used during entering the image in
|
2020-12-07 01:36:33 +03:00
|
|
|
|
the CI image - in this case also the .whl and .tar.gz files will be installed automatically
|
2020-10-10 13:58:09 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
-C, --force-clean-images
|
2020-01-13 22:47:37 +03:00
|
|
|
|
Force build images with cache disabled. This will remove the pulled or build images
|
2020-01-11 18:25:19 +03:00
|
|
|
|
and start building images from scratch. This might take a long time.
|
|
|
|
|
|
2020-08-20 10:57:07 +03:00
|
|
|
|
-r, --skip-rebuild-check
|
|
|
|
|
Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
|
|
|
|
|
|
2020-06-28 18:38:17 +03:00
|
|
|
|
-L, --build-cache-local
|
2020-02-25 00:31:50 +03:00
|
|
|
|
Uses local cache to build images. No pulled images will be used, but results of local
|
2020-06-28 18:38:17 +03:00
|
|
|
|
builds in the Docker cache are used instead. This will take longer than when the pulled
|
|
|
|
|
cache is used for the first time, but subsequent '--build-cache-local' builds will be
|
|
|
|
|
faster as they will use mostly the locally build cache.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the Production image builds.
|
|
|
|
|
|
|
|
|
|
-U, --build-cache-pulled
|
|
|
|
|
Uses images pulled from registry (either DockerHub or GitHub depending on
|
|
|
|
|
--github-registry flag) to build images. The pulled images will be used as cache.
|
|
|
|
|
Those builds are usually faster than when ''--build-cache-local'' with the exception if
|
|
|
|
|
the registry images are not yet updated. The DockerHub images are updated nightly and the
|
|
|
|
|
GitHub images are updated after merges to master so it might be that the images are still
|
|
|
|
|
outdated vs. the latest version of the Dockerfiles you are using. In this case, the
|
|
|
|
|
''--build-cache-local'' might be faster, especially if you iterate and change the
|
|
|
|
|
Dockerfiles yourself.
|
|
|
|
|
|
|
|
|
|
This is default strategy used by the CI image builds.
|
|
|
|
|
|
|
|
|
|
-X, --build-cache-disabled
|
|
|
|
|
Disables cache during docker builds. This is useful if you want to make sure you want to
|
|
|
|
|
rebuild everything from scratch.
|
|
|
|
|
|
|
|
|
|
This strategy is used by default for both Production and CI images for the scheduled
|
|
|
|
|
(nightly) builds in CI.
|
2020-01-13 22:47:37 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
2020-06-02 11:24:47 +03:00
|
|
|
|
Flags for pulling/pushing Docker images (both CI and production)
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-D, --dockerhub-user DOCKERHUB_USER
|
2020-01-13 22:47:37 +03:00
|
|
|
|
DockerHub user used to pull, push and build images. Default: apache.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-H, --dockerhub-repo DOCKERHUB_REPO
|
2020-01-13 22:47:37 +03:00
|
|
|
|
DockerHub repository used to pull, push, build images. Default: airflow.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-c, --github-registry GITHUB_REGISTRY
|
2020-06-16 13:36:46 +03:00
|
|
|
|
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
|
2020-08-19 01:53:04 +03:00
|
|
|
|
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
|
2020-06-16 13:36:46 +03:00
|
|
|
|
and you need to be committer to push to Apache Airflow' GitHub registry.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-g, --github-repository GITHUB_REPOSITORY
|
2020-08-19 01:53:04 +03:00
|
|
|
|
GitHub repository used to pull, push images when cache is used.
|
|
|
|
|
Default: apache/airflow.
|
|
|
|
|
|
|
|
|
|
If you use this flag, automatically --github-registry flag is enabled.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-s, --github-image-id COMMIT_SHA|RUN_ID
|
2020-08-19 01:53:04 +03:00
|
|
|
|
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
|
|
|
|
|
to be able to easily find the image for particular CI runs. Once you know the
|
|
|
|
|
<RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
|
|
|
|
|
automatically pull and use that image so that you can easily reproduce a problem
|
|
|
|
|
that occurred in CI.
|
|
|
|
|
|
|
|
|
|
If you use this flag, automatically --github-registry is enabled.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-11-27 22:06:44 +03:00
|
|
|
|
|
2020-08-19 01:53:04 +03:00
|
|
|
|
Default: latest.
|
2020-06-02 11:24:47 +03:00
|
|
|
|
|
2020-10-11 17:40:31 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Flags for running tests
|
|
|
|
|
|
|
|
|
|
--test-type TEST_TYPE
|
|
|
|
|
Type of the test to run. One of:
|
|
|
|
|
|
2020-10-28 04:29:47 +03:00
|
|
|
|
All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL,Helm
|
2020-10-11 17:40:31 +03:00
|
|
|
|
|
|
|
|
|
Default: All
|
|
|
|
|
|
2020-05-15 20:00:15 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Flags for generation of the backport packages
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-S, --version-suffix-for-pypi SUFFIX
|
2020-05-19 15:17:22 +03:00
|
|
|
|
Adds optional suffix to the version in the generated backport package. It can be used
|
|
|
|
|
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
|
|
|
|
|
|
2020-09-29 12:45:37 +03:00
|
|
|
|
-N, --version-suffix-for-svn SUFFIX
|
2020-05-19 15:17:22 +03:00
|
|
|
|
Adds optional suffix to the generated names of package. It can be used to generate
|
|
|
|
|
rc1/rc2 ... versions of the packages to be uploaded to SVN.
|
2020-05-15 20:00:15 +03:00
|
|
|
|
|
2020-02-25 00:31:50 +03:00
|
|
|
|
****************************************************************************************************
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Increase verbosity of the scripts
|
|
|
|
|
|
|
|
|
|
-v, --verbose
|
2020-08-15 21:33:02 +03:00
|
|
|
|
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
|
|
|
|
|
debugging - when you run breeze with --verbose flags you will be able to see the commands
|
|
|
|
|
executed under the hood and copy&paste them to your terminal to debug them more easily.
|
|
|
|
|
|
2020-04-13 09:51:33 +03:00
|
|
|
|
Note that you can further increase verbosity and see all the commands executed by breeze
|
|
|
|
|
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
|
2019-08-27 21:39:36 +03:00
|
|
|
|
|
2020-06-03 21:58:38 +03:00
|
|
|
|
****************************************************************************************************
|
|
|
|
|
Print detailed help message
|
|
|
|
|
|
|
|
|
|
-h, --help
|
|
|
|
|
Shows detailed help message for the command specified.
|
|
|
|
|
|
2019-11-04 12:37:31 +03:00
|
|
|
|
.. END BREEZE HELP MARKER
|