github
/
incubator-airflow
зеркало из https://github.com/github/incubator-airflow.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Updates IMAGES documentation to reflect Airflow 2.0 changes (#13312) 

Co-authored-by: Xiaodong DENG <xd.deng.r@gmail.com>
This commit is contained in:
Jarek Potiuk 2020-12-26 11:20:47 +01:00 коммит произвёл GitHub
Родитель 6f246b0d54
Коммит 6d6fe36423
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
1 изменённых файлов: 45 добавлений и 60 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

105
IMAGES.rst
Просмотреть файл

@ -39,10 +39,12 @@ The images are named as follows:
where:
* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.14``
The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. The ``1.10.*`` and in
the future ``2.*`` labels are build from git tags and they are "fixed" once built.
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.5``, ``3.7``
* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``,
``v2-0-test``, ``v1-10-test``, ``2.0.0``. The ``master``, ``v1-10-test`` ``v2-0-test`` labels are
built from branches so they change over time. The ``1.10.*`` and ``2.*`` labels are built from git tags
and they are "fixed" once built.
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.6``, ``3.7``,
``3.8``
* The ``-ci`` suffix is added for CI images
* The ``-manifest`` is added for manifest images (see below for explanation of manifest images)
@ -51,7 +53,7 @@ the CI images. Each CI image, when built uses current python version of the base
python images are regularly updated (with bugfixes/security fixes), so for example python3.8 from
last week might be a different image than python3.8 today. Therefore whenever we push CI image
to airflow repository, we also push the python image that was used to build it this image is stored
as ``apache/airflow:python-3.8-<BRANCH_OR_TAG>``.
as ``apache/airflow:python<PYTHON_MAJOR_MINOR_VERSION>-<BRANCH_OR_TAG>``.
Since those are simply snapshots of the existing python images, DockerHub does not create a separate
copy of those images - all layers are mounted from the original python images and those are merely
@ -115,7 +117,7 @@ parameter to Breeze:
.. code-block:: bash
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=1.10.14
--production-image --install-airflow-version=2.0.0
.. note::
@ -132,8 +134,8 @@ This will build the image using command similar to:
.. code-block:: bash
pip install \
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
apache-airflow[async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.0.0 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.6.txt"
You can also build production images from specific Git version via providing ``--install-airflow-reference``
parameter to Breeze (this time constraints are taken from the ``constraints-master`` branch which is the
@ -218,9 +220,10 @@ Our images are named like that:
.. code-block:: bash
apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y - for production images
apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y-ci - for CI images
apache/airflow:<BRANCH_OR_TAG>[-<PATCH>]-pythonX.Y-build - for production build stage
apache/airflow:<BRANCH_OR_TAG>-pythonX.Y - for production images
apache/airflow:<BRANCH_OR_TAG>-pythonX.Y-ci - for CI images
apache/airflow:<BRANCH_OR_TAG>-pythonX.Y-build - for production build stage
apache/airflow:pythonX.Y-<BRANCH_OR_TAG> - for python base image used for both CI and PROD image
For example:
@ -228,10 +231,9 @@ For example:
apache/airflow:master-python3.6 - production "latest" image from current master
apache/airflow:master-python3.6-ci - CI "latest" image from current master
apache/airflow:v1-10-test-python2.7-ci - CI "latest" image from current v1-10-test branch
apache/airflow:1.10.14-python3.6 - production image for 1.10.14 release
apache/airflow:1.10.14-1-python3.6 - production image for 1.10.14 with some patches applied
apache/airflow:v2-0-test-python2.7-ci - CI "latest" image from current v2-0-test branch
apache/airflow:2.0.0-python3.6 - production image for 2.0.0 release
apache/airflow:python3.6-master - base python image for the master branch
You can see DockerHub images at `<https://hub.docker.com/repository/docker/apache/airflow>`_
@ -252,6 +254,7 @@ images for GitHub registry are:
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y - for production images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-ci - for CI images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-build - for production build state
docker.pkg.github.com/apache/airflow/pythonX.Y-<BRANCH> - for base python images
Note that we never push or pull TAG images to GitHub registry. It is only used for CI builds
@ -302,7 +305,7 @@ in the `<#ci-image-build-arguments>`_ chapter below.
Here just a few examples are presented which should give you general understanding of what you can customize.
This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and
This builds the production image in version 3.7 with additional airflow extras from 2.0.0 PyPI package and
additional apt dev and runtime dependencies.
.. code-block:: bash
@ -311,9 +314,9 @@ additional apt dev and runtime dependencies.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
--build-arg AIRFLOW_VERSION="1.10.14" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_VERSION="2.0.0" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.0" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2-0" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc"
@ -328,7 +331,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
./breeze build-image -f Dockerfile.ci \
--production-image --python 3.7 --install-airflow-version=1.10.14 \
--production-image --python 3.7 --install-airflow-version=2.0.0 \
--additional-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
You can build the default production image with standard ``docker build`` command but they will only build
@ -346,9 +349,9 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
--build-arg AIRFLOW_VERSION="1.10.14" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_VERSION="2.0.0" \
--build-arg AIRFLOW_INSTALL_VERSION="==2.0.0" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2-0" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack" \
@ -379,7 +382,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
+==========================================+==========================================+==========================================+
| ``PYTHON_BASE_IMAGE`` | ``python:3.6-slim-buster`` | Base python image |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_VERSION`` | ``2.0.0.dev0`` | version of Airflow |
| ``AIRFLOW_VERSION`` | ``2.0.0`` | version of Airflow |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PYTHON_MAJOR_MINOR_VERSION`` | ``3.6`` | major/minor version of Python (should |
| | | match base image) |
@ -433,9 +436,10 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | repository from which constraints are |
| | | used. By default it is set to |
| | | ``constraints-master`` but can be |
| | | ``constraints-2-0`` for 2.0.* versions |
| | | ``constraints-1-10`` for 1.10.* versions |
| | | or it could point to specific version |
| | | for example ``constraints-1.10.12`` |
| | | for example ``constraints-2.0.0`` |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_PROVIDERS_FROM_SOURCES`` | ``true`` | If set to false and image is built from |
| | | sources, all provider packages are not |
@ -566,15 +570,24 @@ Image manifests
---------------
Together with the main CI images we also build and push image manifests. Those manifests are very small images
that contain only results of the docker inspect for the image. This is in order to be able to
determine very quickly if the image in the docker registry has changed a lot since the last time.
Unfortunately docker registry (specifically DockerHub registry) has no anonymous way of querying image
details via API, you need to download the image to inspect it. We overcame it in the way that
always when we build the image we build a very small image manifest and push it to registry together
with the main CI image. The tag for the manifest image is the same as for the image it refers
to with added ``-manifest`` suffix. The manifest image for ``apache/airflow:master-python3.6-ci`` is named
that contain only content of randomly generated file at the 'crucial' part of the CI image building.
This is in order to be able to determine very quickly if the image in the docker registry has changed a
lot since the last time. Unfortunately docker registry (specifically DockerHub registry) has no anonymous
way of querying image details via API. You really need to download the image to inspect it.
We workaround it in the way that always when we build the image we build a very small image manifest
containing randomly generated UUID and push it to registry together with the main CI image.
The tag for the manifest image reflects the image it refers to with added ``-manifest`` suffix.
The manifest image for ``apache/airflow:master-python3.6-ci`` is named
``apache/airflow:master-python3.6-ci-manifest``.
The image is quickly pulled (it is really, really small) when important files change and the content
of the randomly generated UUID is compared with the one in our image. If the contents are different
this means that the user should rebase to latest master and rebuild the image with pulling the image from
the repo as this will likely be faster than rebuilding the image locally.
The random UUID is generated right after pre-cached pip install is run - and usually it means that
significant changes have been made to apt packages or even the base python image has changed.
Pulling the Latest Images
-------------------------
@ -646,31 +659,3 @@ Using, customising, and extending the production image
You can read more about using, customising, and extending the production image in the
`documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_.
Alpha versions of 1.10.10 production-ready images
=================================================
The production images have been released for the first time in 1.10.10 release of Airflow as "Alpha" quality
ones. Between 1.10.10 the images are being improved and the 1.10.10 images should be patched and
published several times separately in order to test them with the upcoming Helm Chart.
Those images are for development and testing only and should not be used outside of the
development community.
The images were pushed with tags following the pattern: ``apache/airflow:1.10.10.1-alphaN-pythonX.Y``.
Patch level is an increasing number (starting from 1).
Those are alpha-quality releases however they contain the officially released Airflow ``1.10.10`` code.
The main changes in the images are scripts embedded in the images.
The following versions were pushed:
+-------+--------------------------------+----------------------------------------------------------+
| Patch | Tag pattern | Description |
+=======+================================+==========================================================+
| 1 | ``1.10.10.1-alpha1-pythonX.Y`` | Support for parameters added to bash and python commands |
+-------+--------------------------------+----------------------------------------------------------+
| 2 | ``1.10.10-1-alpha2-pythonX.Y`` | Added "/clean-logs" script |
+-------+--------------------------------+----------------------------------------------------------+
The commits used to generate those images are tagged with ``prod-image-1.10.10.1-alphaN`` tags.