github
/
incubator-airflow
зеркало из https://github.com/github/incubator-airflow.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Adds capability of switching to Github Container Registry (#13726) 

* Adds capability of switching to Github Container Registry

Currently we are using GitHub Packages to cache images for the
build. GitHub Packages are "legacy" storage of binary artifacts
for GitHub and as of September 2020 they introduced Github
Container Registry as more stable, easier to manage replacement
for container storage. It includes complete self-management
of the images including permission management, public access,
retention management and many more.

More about it here:

https://github.blog/2020-09-01-introducing-github-container-registry/

Recently we started to experience unstable behaviour of the
Github Packages ('unknown blob' and manifest v1 vs. v2 when
pushing images to it. So together with ASF we proposed to
enable Github Container Registry and it happened as of
January 2020.

More about it in https://issues.apache.org/jira/browse/INFRA-20959

We are currently in the testing phase, especially when it
comes to management of permissions - the model of permission
mangement is not the same for Container Registry as it was
for GitHub Packages (it was per-repository in GitHub Packages,
but it is organization-wide in the Container Registry.

This PR introduces an option to use GitHub Container Registry
rather than GitHub Packages. It is implemented in both - CI
level and Breeze level allowing to seamlessly switch between
those two solutions:

In Breeze (which we use to test pushing/pulling the images)
--github-registry option was added with `ghcr.io` (Github Container
Registry) or `docker.pkg.github.com` (GitHub Packages).

In CI the same can be achieved by setting GITHUB_REGISTRY value
(same values possible as for --github-registry Breeze parameter)

* fixup! Adds capability of switching to Github Container Registry
This commit is contained in:
Jarek Potiuk 2021-01-21 16:16:09 +01:00 коммит произвёл GitHub
Родитель 9d9ef1addc
Коммит 2c6c7fdb23
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
25 изменённых файлов: 484 добавлений и 107 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

5
.github/workflows/build-images-workflow-run.yml поставляемый
Просмотреть файл

@ -30,9 +30,14 @@ env:
DB_RESET: "true"
VERBOSE: "true"
USE_GITHUB_REGISTRY: "true"
# Might be either 'ghcr.io' or 'docker.pkg.github.com'
GITHUB_REGISTRY: "docker.pkg.github.com"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_USERNAME: ${{ github.actor }}
# This token is WRITE one - workflow_run type of events always have the WRITE token
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# This token should not be empty in workflow_run type of event.
CONTAINER_REGISTRY_TOKEN: ${{ secrets.PAT_CR }}
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
BUILD_IMAGES: ${{ secrets.AIRFLOW_GITHUB_REGISTRY_WAIT_FOR_IMAGE != 'false' }}

6
.github/workflows/ci.yml поставляемый
Просмотреть файл

@ -36,9 +36,15 @@ env:
VERBOSE: "true"
DOCKER_CACHE: "pulled"
USE_GITHUB_REGISTRY: "true"
# Might be either 'ghcr.io' or 'docker.pkg.github.com'
GITHUB_REGISTRY: "docker.pkg.github.com"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_USERNAME: ${{ github.actor }}
# In builds from forks, this token is read-only. For scheduler/direct push it is WRITE one
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# In builds from forks, this token is empty, and this is good because such builds do not even try
# to push images to the registry.
CONTAINER_REGISTRY_TOKEN: ${{ secrets.PAT_CR }}
GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.run_id }}"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
INSTALL_PROVIDERS_FROM_SOURCES: "true"

9
.github/workflows/scheduled_quarantined.yml поставляемый
Просмотреть файл

@ -33,11 +33,16 @@ env:
UPGRADE_TO_NEWER_DEPENDENCIES: false
PYTHON_MAJOR_MINOR_VERSION: 3.6
USE_GITHUB_REGISTRY: "true"
# Since we run this build on schedule, it might be that the image has never been pushed
# Because the master merge was cancelled, so we have to rebuild the image for quarantined build
# Might be either 'ghcr.io' or 'docker.pkg.github.com'
GITHUB_REGISTRY: "docker.pkg.github.com"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_USERNAME: ${{ github.actor }}
# This token is WRITE one - schedule type of events always have the WRITE token
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# This token should not be empty in schedule type of event.
CONTAINER_REGISTRY_TOKEN: ${{ secrets.PAT_CR }}
# Since we run this build on schedule, it might be that the image has never been pushed
# Because the master merge was cancelled, so we have to rebuild the image for quarantined build
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"

60
BREEZE.rst
Просмотреть файл

@ -1416,7 +1416,7 @@ This is the current syntax for `./breeze <./breeze>`_:
-U, --build-cache-pulled
Uses images pulled from registry (either DockerHub or GitHub depending on
--github-registry flag) to build images. The pulled images will be used as cache.
--use-github-registry flag) to build images. The pulled images will be used as cache.
Those builds are usually faster than when ''--build-cache-local'' with the exception if
the registry images are not yet updated. The DockerHub images are updated nightly and the
GitHub images are updated after merges to master so it might be that the images are still
@ -1439,16 +1439,26 @@ This is the current syntax for `./breeze <./breeze>`_:
-H, --dockerhub-repo DOCKERHUB_REPO
DockerHub repository used to pull, push, build images. Default: airflow.
-c, --github-registry GITHUB_REGISTRY
-c, --use-github-registry
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
If you use this flag, automatically --use-github-registry flag is enabled.
docker.pkg.github.com ghcr.io
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
Default: apache/airflow.
If you use this flag, automatically --github-registry flag is enabled.
If you use this flag, automatically --use-github-registry flag is enabled.
-s, --github-image-id COMMIT_SHA|RUN_ID
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
@ -1457,7 +1467,7 @@ This is the current syntax for `./breeze <./breeze>`_:
automatically pull and use that image so that you can easily reproduce a problem
that occurred in CI.
If you use this flag, automatically --github-registry is enabled.
If you use this flag, automatically --use-github-registry is enabled.
Default: latest.
@ -1567,14 +1577,14 @@ This is the current syntax for `./breeze <./breeze>`_:
breeze push_image [FLAGS]
Pushes images to docker registry. You can push the images to DockerHub registry (default)
or to the GitHub registry (if --github-registry flag is used).
or to the GitHub registry (if --use-github-registry flag is used).
For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
the repository to push to. For GitHub repository, the --github-repository
flag can be used for the same purpose. You can also add
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to push image with specific
SHA tag or run id. In case you specify --github-repository or --github-image-id, you
do not need to specify --github-registry flag.
do not need to specify --use-github-registry flag.
You can also add --production-image flag to switch to production image (default is CI one)
@ -1583,7 +1593,7 @@ This is the current syntax for `./breeze <./breeze>`_:
'breeze push-image' or
'breeze push-image --dockerhub-user user' to push to your private registry or
'breeze push-image --production-image' - to push production image or
'breeze push-image --github-registry' - to push to GitHub image registry or
'breeze push-image --use-github-registry' - to push to GitHub image registry or
'breeze push-image \
--github-repository user/airflow' - to push to your user's fork
'breeze push-image \
@ -1599,16 +1609,26 @@ This is the current syntax for `./breeze <./breeze>`_:
-H, --dockerhub-repo DOCKERHUB_REPO
DockerHub repository used to pull, push, build images. Default: airflow.
-c, --github-registry GITHUB_REGISTRY
-c, --use-github-registry
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
If you use this flag, automatically --use-github-registry flag is enabled.
docker.pkg.github.com ghcr.io
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
Default: apache/airflow.
If you use this flag, automatically --github-registry flag is enabled.
If you use this flag, automatically --use-github-registry flag is enabled.
-s, --github-image-id COMMIT_SHA|RUN_ID
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
@ -1617,7 +1637,7 @@ This is the current syntax for `./breeze <./breeze>`_:
automatically pull and use that image so that you can easily reproduce a problem
that occurred in CI.
If you use this flag, automatically --github-registry is enabled.
If you use this flag, automatically --use-github-registry is enabled.
Default: latest.
@ -2004,7 +2024,7 @@ This is the current syntax for `./breeze <./breeze>`_:
-U, --build-cache-pulled
Uses images pulled from registry (either DockerHub or GitHub depending on
--github-registry flag) to build images. The pulled images will be used as cache.
--use-github-registry flag) to build images. The pulled images will be used as cache.
Those builds are usually faster than when ''--build-cache-local'' with the exception if
the registry images are not yet updated. The DockerHub images are updated nightly and the
GitHub images are updated after merges to master so it might be that the images are still
@ -2543,7 +2563,7 @@ This is the current syntax for `./breeze <./breeze>`_:
-U, --build-cache-pulled
Uses images pulled from registry (either DockerHub or GitHub depending on
--github-registry flag) to build images. The pulled images will be used as cache.
--use-github-registry flag) to build images. The pulled images will be used as cache.
Those builds are usually faster than when ''--build-cache-local'' with the exception if
the registry images are not yet updated. The DockerHub images are updated nightly and the
GitHub images are updated after merges to master so it might be that the images are still
@ -2569,16 +2589,26 @@ This is the current syntax for `./breeze <./breeze>`_:
-H, --dockerhub-repo DOCKERHUB_REPO
DockerHub repository used to pull, push, build images. Default: airflow.
-c, --github-registry GITHUB_REGISTRY
-c, --use-github-registry
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: docker.pkg.github.com.
If you use this flag, automatically --use-github-registry flag is enabled.
docker.pkg.github.com ghcr.io
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
Default: apache/airflow.
If you use this flag, automatically --github-registry flag is enabled.
If you use this flag, automatically --use-github-registry flag is enabled.
-s, --github-image-id COMMIT_SHA|RUN_ID
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
@ -2587,7 +2617,7 @@ This is the current syntax for `./breeze <./breeze>`_:
automatically pull and use that image so that you can easily reproduce a problem
that occurred in CI.
If you use this flag, automatically --github-registry is enabled.
If you use this flag, automatically --use-github-registry is enabled.
Default: latest.

48
CI.rst
Просмотреть файл

@ -366,14 +366,33 @@ Our CI uses GitHub Registry to pull and push images to/from by default. You can
DockerHub registry or change the GitHub registry to interact with and use your own repo by changing
``GITHUB_REPOSITORY`` and providing your own GitHub Username and Token.
Currently we are using GitHub Packages to cache images for the build. GitHub Packages are "legacy"
storage of binary artifacts for GitHub and as of September 2020 they introduced Github Container Registry
as more stable, easier to manage replacement for container storage. It includes complete self-management
of the images including permission management, public access, retention management and many more.
More about it here:
https://github.blog/2020-09-01-introducing-github-container-registry/
Recently we started to experience unstable behaviour of the Github Packages ('unknown blob'
and manifest v1 vs. v2 when pushing images to it). So together with ASF we proposed to
enable Github Container Registry and it happened as of January 2020.
More about it in https://issues.apache.org/jira/browse/INFRA-20959
We are currently in the testing phase, especially when it comes to management of permissions -
the model of permission management is not the same for Container Registry as it was for GitHub Packages
(it was per-repository in GitHub Packages, but it is organization-wide in the Container Registry.
+--------------------------------+---------------------------+----------------------------------------------+
| Variable | Default | Comment |
+================================+===========================+==============================================+
| USE_GITHUB_REGISTRY | true | If set to "true", we interact with GitHub |
| | | Registry registry not the DockerHub one. |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_REGISTRY | ``docker.pkg.github.com`` | DNS name of the GitHub registry to |
| | | use. |
| GITHUB_REGISTRY | ``docker.pkg.github.com`` | Name of the GitHub registry to use. Can be |
| | | ``docker.pkg.github.com`` or ``ghcr.io`` |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_REPOSITORY | ``apache/airflow`` | Prefix of the image. It indicates which. |
| | | registry from GitHub to use |
@ -381,8 +400,18 @@ DockerHub registry or change the GitHub registry to interact with and use your o
| GITHUB_USERNAME | | Username to use to login to GitHub |
| | | |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_TOKEN | | Personal token to use to login to GitHub |
| | | |
| GITHUB_TOKEN | | Token to use to login to GitHub. This token |
| | | is automatically set by GitHub CI to a |
| | | to a READ-only token for PR builds from fork |
| | | and to WRITE token for direct pushes and |
| | | scheduled or workflow_run types of builds |
+--------------------------------+---------------------------+----------------------------------------------+
| CONTAINER_REGISTRY_TOKEN | | Personal token to use to login to GitHub |
| | | Container Registry. Should be retrieved |
| | | from secret (in our case it is PAT_CR secret |
| | | following example in GitHub documentation. |
| | | Only set in push/scheduled/workflow_run |
| | | type of build. |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_REGISTRY_WAIT_FOR_IMAGE | ``false`` | Wait for the image to be available. This is |
| | | useful if commit SHA is used as pull tag |
@ -394,6 +423,17 @@ DockerHub registry or change the GitHub registry to interact with and use your o
| | | default, can be commit SHA or RUN_ID. |
+--------------------------------+---------------------------+----------------------------------------------+
Authentication in Github Registry
=================================
We are currently in the process of testing using Github Container Registry as cache for our images during
the CI process. The default registry is set to "GitHub Packages", but we are testing the GitHub
Container Registry. In case of GitHub Packages, authentication uses GITHUB_TOKEN mechanism. Authentication
is needed for both pushing the images (WRITE) and pulling them (READ) - which means that GitHub token
is used in "master" build (WRITE) and in fork builds (READ). For container registry, our images are
Publicly Visible and we do not need any authentication to pull them so the CONTAINER_REGISTRY_TOKEN is
only set in the "master" builds only ("Build Images" workflow and "Scheduled quarantine" one).
Dockerhub Variables
===================

38
Dockerfile
Просмотреть файл

@ -296,6 +296,8 @@ ENV BUILD_ID=${BUILD_ID}
ARG COMMIT_SHA
ENV COMMIT_SHA=${COMMIT_SHA}
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_DATE_CREATED
LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.distro.version="buster" \
@ -304,7 +306,20 @@ LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.image="airflow-build-image" \
org.apache.airflow.version="${AIRFLOW_VERSION}" \
org.apache.airflow.buildImage.buildId=${BUILD_ID} \
org.apache.airflow.buildImage.commitSha=${COMMIT_SHA}
org.apache.airflow.buildImage.commitSha=${COMMIT_SHA} \
org.opencontainers.image.source=${AIRFLOW_IMAGE_REPOSITORY} \
org.opencontainers.image.created=${AIRFLOW_IMAGE_DATE_CREATED} \
org.opencontainers.image.authors="dev@airflow.apache.org" \
org.opencontainers.image.url="https://airflow.apache.org" \
org.opencontainers.image.documentation="https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html" \
org.opencontainers.image.source="https://github.com/apache/airflow" \
org.opencontainers.image.version="${AIRFLOW_VERSION}" \
org.opencontainers.image.revision="${COMMIT_SHA}" \
org.opencontainers.image.vendor="Apache Software Foundation" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow-build-image" \
org.opencontainers.image.title="Build Image Segment for Production Airflow Image" \
org.opencontainers.image.description="Installed Apache Airflow with build-time dependencies"
##############################################################################################
# This is the actual Airflow image - much smaller than the build one. We copy
@ -467,6 +482,9 @@ ENV BUILD_ID=${BUILD_ID}
ARG COMMIT_SHA
ENV COMMIT_SHA=${COMMIT_SHA}
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_DATE_CREATED
LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.distro.version="buster" \
org.apache.airflow.module="airflow" \
@ -475,8 +493,22 @@ LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.version="${AIRFLOW_VERSION}" \
org.apache.airflow.uid="${AIRFLOW_UID}" \
org.apache.airflow.gid="${AIRFLOW_GID}" \
org.apache.airflow.mainImage.buildId=${BUILD_ID} \
org.apache.airflow.mainImage.commitSha=${COMMIT_SHA}
org.apache.airflow.mainImage.buildId="${BUILD_ID}" \
org.apache.airflow.mainImage.commitSha="${COMMIT_SHA}" \
org.opencontainers.image.source="${AIRFLOW_IMAGE_REPOSITORY}" \
org.opencontainers.image.created=${AIRFLOW_IMAGE_DATE_CREATED} \
org.opencontainers.image.authors="dev@airflow.apache.org" \
org.opencontainers.image.url="https://airflow.apache.org" \
org.opencontainers.image.documentation="https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html" \
org.opencontainers.image.source="https://github.com/apache/airflow" \
org.opencontainers.image.version="${AIRFLOW_VERSION}" \
org.opencontainers.image.revision="${COMMIT_SHA}" \
org.opencontainers.image.vendor="Apache Software Foundation" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow" \
org.opencontainers.image.title="Production Airflow Image" \
org.opencontainers.image.description="Installed Apache Airflow"
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
CMD ["--help"]

20
Dockerfile.ci
Просмотреть файл

@ -416,6 +416,9 @@ ENV BUILD_ID=${BUILD_ID}
ARG COMMIT_SHA
ENV COMMIT_SHA=${COMMIT_SHA}
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_DATE_CREATED
LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.distro.version="buster" \
org.apache.airflow.module="airflow" \
@ -424,8 +427,21 @@ LABEL org.apache.airflow.distro="debian" \
org.apache.airflow.version="${AIRFLOW_VERSION}" \
org.apache.airflow.uid="0" \
org.apache.airflow.gid="0" \
org.apache.airflow.buildId=${BUILD_ID} \
org.apache.airflow.commitSha=${COMMIT_SHA}
org.apache.airflow.buildId="${BUILD_ID}" \
org.apache.airflow.commitSha="${COMMIT_SHA}" \
org.opencontainers.image.source="${AIRFLOW_IMAGE_REPOSITORY}" \
org.opencontainers.image.created="${AIRFLOW_IMAGE_DATE_CREATED}" \
org.opencontainers.image.authors="dev@airflow.apache.org" \
org.opencontainers.image.url="https://airflow.apache.org" \
org.opencontainers.image.documentation="https://github.com/apache/airflow/IMAGES.rst" \
org.opencontainers.image.source="https://github.com/apache/airflow" \
org.opencontainers.image.version="${AIRFLOW_VERSION}" \
org.opencontainers.image.revision="${COMMIT_SHA}" \
org.opencontainers.image.vendor="Apache Software Foundation" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow-ci-image" \
org.opencontainers.image.title="Continuous Integration Airflow Image" \
org.opencontainers.image.description="Installed Apache Airflow with Continuous Integration dependencies"
EXPOSE 8080

83
IMAGES.rst
Просмотреть файл

@ -249,6 +249,9 @@ For example:
You can see DockerHub images at `<https://hub.docker.com/r/apache/airflow>`_
Using GitHub registries as build cache
--------------------------------------
By default DockerHub registry is used when you push or pull such images.
However for CI builds we keep the images in GitHub registry as well - this way we can easily push
the images automatically after merge requests and use such images for Pull Requests
@ -259,29 +262,91 @@ significant changes are done in the Dockerfile.CI.
The images are named differently (in Docker definition of image names - registry URL is part of the
image name if DockerHub is not used as registry). Also GitHub has its own structure for registries
each project has its own registry naming convention that should be followed. The name of
images for GitHub registry are:
images for GitHub registry are different as they must follow limitation of the registry used.
We are still using Github Packages as registry, but we are in the process of testing and switching
to GitHub Container Registry, and the naming conventions are slightly different (GitHub Packages
required all packages to have "organization/repository/" URL prefix ("apache/airflow/",
where in GitHub Container Registry, all images are in "organization" not in "repository" and they are all
in organization wide "apache/" namespace rather than in "apache/airflow/" one).
We are adding "airflow-" as prefix for image names of all Airflow images instead.
The images are linked to the repository via ``org.opencontainers.image.source`` label in the image.
Naming convention for GitHub Packages
-------------------------------------
Images built as "Run ID snapshot":
.. code-block:: bash
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y - for production images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-ci - for CI images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-build - for production build state
docker.pkg.github.com/apache/airflow/pythonX.Y-<BRANCH> - for base python images
docker.pkg.github.com.io/apache-airflow/<BRANCH>-pythonX.Y-ci-v2:<RUNID> - for CI images
docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-v2:<RUNID> - for production images
docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-build-v2:<RUNID> - for production build stage
docker.pkg.github.com/apache-airflow/pythonX.Y-<BRANCH>-v2:X.Y-slim-buster-<RUN_ID> - for base python images
Note that we never push or pull TAG images to GitHub registry. It is only used for CI builds
Latest images (pushed when master merge succeeds):
.. code-block:: bash
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-ci-v2:latest - for CI images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-v2:latest - for production images
docker.pkg.github.com/apache/airflow/<BRANCH>-pythonX.Y-build-v2:latest - for production build stage
docker.pkg.github.com/apache/airflow/python-<BRANCH>-v1:X.Y-slim-buster - for base python images
Naming convention for GitHub Container Registry
-----------------------------------------------
Images built as "Run ID snapshot":
.. code-block:: bash
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:<RUNID> - for CI images
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:<RUNID> - for production images
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:<RUNID> - for production build stage
ghcr.io/apache/airflow-pythonX.Y-<BRANCH>-v2:X.Y-slim-buster-<RUN_ID> - for base python images
Latest images (pushed when master merge succeeds):
.. code-block:: bash
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:latest - for CI images
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:latest - for production images
ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:latest - for production build stage
ghcr.io/apache/airflow-python-<BRANCH>-v2:X.Y-slim-buster - for base python images
Note that we never push or pull "release" images to GitHub registry. It is only used for CI builds
You can see all the current GitHub images at `<https://github.com/apache/airflow/packages>`_
In order to interact with the GitHub images you need to add ``--github-registry`` flag to the pull/push
In order to interact with the GitHub images you need to add ``--use-github-registry`` flag to the pull/push
commands in Breeze. This way the images will be pulled/pushed from/to GitHub rather than from/to
DockerHub. Images are build locally as ``apache/airflow`` images but then they are tagged with the right
GitHub tags for you.
GitHub tags for you. You can also specify ``--github-registry`` option and choose which of the
GitHub registries are used (``docker.pkg.github.com`` chooses GitHub Packages and ``ghcr.io`` chooses
GitHub Container Registry).
You can read more about the CI configuration and how CI builds are using DockerHub/GitHub images
in `<CI.rst>`_.
Note that you need to be committer and have the right to push to DockerHub and GitHub and you need to
be logged in. Only committers can push images directly.
be logged in. Only committers can push images directly. You need to login with your
Personal Access Token with "packages" scope to be able to push to those repositories or pull from them
in case of GitHub Packages.
GitHub Packages:
.. code-block:: bash
docker login docker.pkg.github.com
GitHub Container Registry
.. code-block:: bash
docker login ghcr.io
Technical details of Airflow images
===================================

50
breeze
Просмотреть файл

@ -466,6 +466,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
Airflow source version: $(build_images::get_airflow_version_from_production_image)
EOF
else
@ -475,6 +476,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
Airflow source version: ${AIRFLOW_VERSION}
EOF
fi
@ -499,6 +501,7 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
EOF
else
cat <<EOF
@ -507,6 +510,8 @@ EOF
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Github cache prefix: ${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)
EOF
fi
cat <<EOF
@ -1071,18 +1076,26 @@ function breeze::parse_arguments() {
export FORWARD_CREDENTIALS="true"
shift
;;
-c | --github-registry)
-c | --use-github-registry)
echo
echo "Use GitHub registry"
echo
export USE_GITHUB_REGISTRY="true"
shift
;;
-g | --github-repository)
echo
echo "GitHub repository: ${2}"
--github-registry)
echo
echo "Using GitHub registry."
echo "GitHub registry used: ${2}"
echo
export GITHUB_REGISTRY="${2}"
export USE_GITHUB_REGISTRY="true"
shift 2
;;
-g | --github-repository)
echo
echo "Using GitHub registry."
echo "GitHub repository: ${2}"
echo
export GITHUB_REPOSITORY="${2}"
export USE_GITHUB_REGISTRY="true"
@ -1494,6 +1507,10 @@ function breeze::prepare_formatted_versions() {
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_INSTALLATION_METHOD
FORMATTED_GITHUB_REGISTRY=$(echo "${_breeze_allowed_github_registrys=""}" |
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_GITHUB_REGISTRY
FORMATTED_POSTGRES_VERSIONS=$(echo "${_breeze_allowed_postgres_versions=""}" |
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_POSTGRES_VERSIONS
@ -1821,14 +1838,14 @@ $(breeze::flag_verbosity)
${CMDNAME} push_image [FLAGS]
Pushes images to docker registry. You can push the images to DockerHub registry (default)
or to the GitHub registry (if --github-registry flag is used).
or to the GitHub registry (if --use-github-registry flag is used).
For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
the repository to push to. For GitHub repository, the --github-repository
flag can be used for the same purpose. You can also add
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to push image with specific
SHA tag or run id. In case you specify --github-repository or --github-image-id, you
do not need to specify --github-registry flag.
do not need to specify --use-github-registry flag.
You can also add --production-image flag to switch to production image (default is CI one)
@ -1837,7 +1854,7 @@ ${CMDNAME} push_image [FLAGS]
'${CMDNAME} push-image' or
'${CMDNAME} push-image --dockerhub-user user' to push to your private registry or
'${CMDNAME} push-image --production-image' - to push production image or
'${CMDNAME} push-image --github-registry' - to push to GitHub image registry or
'${CMDNAME} push-image --use-github-registry' - to push to GitHub image registry or
'${CMDNAME} push-image \\
--github-repository user/airflow' - to push to your user's fork
'${CMDNAME} push-image \\
@ -2558,7 +2575,7 @@ Build options:
-U, --build-cache-pulled
Uses images pulled from registry (either DockerHub or GitHub depending on
--github-registry flag) to build images. The pulled images will be used as cache.
--use-github-registry flag) to build images. The pulled images will be used as cache.
Those builds are usually faster than when ''--build-cache-local'' with the exception if
the registry images are not yet updated. The DockerHub images are updated nightly and the
GitHub images are updated after merges to master so it might be that the images are still
@ -2599,16 +2616,26 @@ function breeze::flag_pull_push_docker_images() {
-H, --dockerhub-repo DOCKERHUB_REPO
DockerHub repository used to pull, push, build images. Default: ${_breeze_default_dockerhub_repo:=}.
-c, --github-registry GITHUB_REGISTRY
-c, --use-github-registry
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
and you need to be committer to push to Apache Airflow' GitHub registry.
--github-registry GITHUB_REGISTRY
Github registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
Default: ${_breeze_default_github_registry:=}.
If you use this flag, automatically --use-github-registry flag is enabled.
${FORMATTED_GITHUB_REGISTRY}
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
Default: ${_breeze_default_github_repository:=}.
If you use this flag, automatically --github-registry flag is enabled.
If you use this flag, automatically --use-github-registry flag is enabled.
-s, --github-image-id COMMIT_SHA|RUN_ID
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
@ -2617,7 +2644,7 @@ function breeze::flag_pull_push_docker_images() {
automatically pull and use that image so that you can easily reproduce a problem
that occurred in CI.
If you use this flag, automatically --github-registry is enabled.
If you use this flag, automatically --use-github-registry is enabled.
Default: ${_breeze_default_github_image_id:=}.
@ -2939,6 +2966,7 @@ function breeze::check_and_save_all_params() {
parameters::check_and_save_allowed_param "HELM_VERSION" "Helm version" "--helm-version"
parameters::check_and_save_allowed_param "POSTGRES_VERSION" "Postgres version" "--postgres-version"
parameters::check_and_save_allowed_param "MYSQL_VERSION" "Mysql version" "--mysql-version"
parameters::check_and_save_allowed_param "GITHUB_REGISTRY" "GitHub Registry" "--github-registry"
parameters::check_allowed_param TEST_TYPE "Type of tests" "--test-type"
parameters::check_allowed_param PACKAGE_FORMAT "Format of packages to build" "--package-format"

8
breeze-complete
Просмотреть файл

@ -26,6 +26,8 @@
_breeze_allowed_python_major_minor_versions="2.7 3.5 3.6 3.7 3.8"
_breeze_allowed_backends="sqlite mysql postgres"
_breeze_allowed_integrations="cassandra kerberos mongo openldap pinot presto rabbitmq redis all"
# registrys is good here even if it is not correct english. We are adding s automatically to all variables
_breeze_allowed_github_registrys="docker.pkg.github.com ghcr.io"
_breeze_allowed_kubernetes_modes="image"
_breeze_allowed_kubernetes_versions="v1.18.6 v1.17.5 v1.16.9"
_breeze_allowed_helm_versions="v3.2.4"
@ -41,6 +43,7 @@ _breeze_allowed_installation_methods=". apache-airflow"
{
# Default values for the commands & flags used
_breeze_default_backend=$(echo "${_breeze_allowed_backends}" | awk '{print $1}')
_breeze_default_github_registry=$(echo "${_breeze_allowed_github_registrys}" | awk '{print $1}')
_breeze_default_kubernetes_mode=$(echo "${_breeze_allowed_kubernetes_modes}" | awk '{print $1}')
_breeze_default_kubernetes_version=$(echo "${_breeze_allowed_kubernetes_versions}" | awk '{print $1}')
_breeze_default_helm_version=$(echo "${_breeze_allowed_helm_versions}" | awk '{print $1}')
@ -160,7 +163,7 @@ skip-mounting-local-sources install-airflow-version: install-airflow-reference:
verbose assume-yes assume-no assume-quit forward-credentials init-script:
force-build-images force-pull-images production-image extras: force-clean-images skip-rebuild-check
build-cache-local build-cache-pulled build-cache-disabled disable-pip-cache
dockerhub-user: dockerhub-repo: github-registry github-repository: github-image-id:
dockerhub-user: dockerhub-repo: use-github-registry github-registry: github-repository: github-image-id:
postgres-version: mysql-version:
version-suffix-for-pypi: version-suffix-for-svn: backports
additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: image-tag:
@ -295,6 +298,9 @@ function breeze_complete::get_known_values_breeze() {
--installation-method)
_breeze_known_values="${_breeze_allowed_installation_methods}"
;;
--github-registry)
_breeze_known_values="${_breeze_allowed_github_registrys}"
;;
*)
_breeze_known_values=""
;;

4
scripts/ci/images/ci_prepare_ci_image_on_ci.sh
Просмотреть файл

@ -22,7 +22,7 @@
# Depending on "USE_GITHUB_REGISTRY" and "GITHUB_REGISTRY_WAIT_FOR_IMAGE" setting
function build_ci_image_on_ci() {
build_images::prepare_ci_build
start_end::group_start "Prepare CI mage ${AIRFLOW_CI_IMAGE}"
start_end::group_start "Prepare CI image ${AIRFLOW_CI_IMAGE}"
rm -rf "${BUILD_CACHE_DIR}"
mkdir -pv "${BUILD_CACHE_DIR}"
@ -35,7 +35,7 @@ function build_ci_image_on_ci() {
# skips further image checks - since we already have the target image
local python_tag_suffix=""
if [[ ${GITHUB_REGISTRY_PUSH_IMAGE_TAG} != "latest" ]]; then
if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} != "latest" ]]; then
python_tag_suffix="-${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
fi
# first we pull base python image. We will need it to re-push it after master build

2
scripts/ci/images/ci_wait_for_ci_image.sh
Просмотреть файл

@ -20,7 +20,7 @@
push_pull_remove_images::check_if_github_registry_wait_for_image_enabled
build_image::login_to_github_registry_if_needed
build_image::configure_github_docker_registry
export AIRFLOW_CI_IMAGE_NAME="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"

2
scripts/ci/images/ci_wait_for_prod_image.sh
Просмотреть файл

@ -20,7 +20,7 @@
push_pull_remove_images::check_if_github_registry_wait_for_image_enabled
build_image::login_to_github_registry_if_needed
build_image::configure_github_docker_registry
export AIRFLOW_PROD_IMAGE_NAME="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
start_end::group_start "Waiting for ${AIRFLOW_PROD_IMAGE_NAME} image to appear"

158
scripts/ci/libraries/_build_images.sh
Просмотреть файл

@ -26,7 +26,7 @@ function build_images::add_build_args_for_remote_install() {
"--build-arg" "AIRFLOW_SOURCES_FROM=empty"
"--build-arg" "AIRFLOW_SOURCES_TO=/empty"
)
if [[ ${AIRFLOW_CONSTRAINTS_REFERENCE} != "" ]]; then
if [[ -n "${AIRFLOW_CONSTRAINTS_REFERENCE}" ]]; then
EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
"--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE}"
)
@ -59,7 +59,7 @@ function build_images::add_build_args_for_remote_install() {
)
fi
fi
if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
if [[ -n "${AIRFLOW_CONSTRAINTS_LOCATION}" ]]; then
EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
"--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
)
@ -330,7 +330,7 @@ function build_images::compare_local_and_remote_build_cache_hash() {
local local_hash
local_hash=$(cat "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}")
if [[ ${remote_hash} != "${local_hash}" || ${local_hash} == "" ]] \
if [[ ${remote_hash} != "${local_hash}" || -z ${local_hash} ]] \
; then
echo
echo
@ -373,7 +373,7 @@ function build_images::get_docker_image_names() {
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
# Base production image tag - used to build kubernetes tag as well
if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
if [[ -z "${FORCE_AIRFLOW_PROD_BASE_TAG=}" ]]; then
export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
else
export AIRFLOW_PROD_BASE_TAG="${FORCE_AIRFLOW_PROD_BASE_TAG}"
@ -394,30 +394,111 @@ function build_images::get_docker_image_names() {
# File that is touched when the CI image is built for the first time locally
export BUILT_CI_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
# GitHub Registry names must be lowercase :(
github_repository_lowercase="$(echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')"
export GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_PROD_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
export GITHUB_REGISTRY_AIRFLOW_PROD_BUILD_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_PROD_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}-build"
export GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/python${GITHUB_REGISTRY_IMAGE_SUFFIX}:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# This is 1-1 mapping of image names of Apache Airflow stored in DockerHub vs. the same images stored
# in Github Registries (either Github Container Registry or Github Packages)
#
# We have to apply naming conventions used by the registries and keep multiple RUN_ID tags. We use
# common suffix ('gcr-v1') to be able to switch to different set of cache images if needed
# - for example when some images gets broken (might happen with Github Actions Registries) or when
# the storage capacity per image is reached (though it is apparently unlimited)
#
# Some examples:
#
# In case of Github Container Registry:
#
# * Prod Image: "apache/airflow:master-python3.8" -> "apache/airflow-master-python3.8-gcr-v1:<RUN_ID>"
# * Prod build image: "apache/airflow:master-python3.8-build" -> "apache/airflow-master-python3.8-build-gcr-v1:<RUN_ID>"
# * CI build image: "apache/airflow:master-python3.8-ci" -> "apache/airflow-master-python3.8-ci-gcr-v1:<RUN_ID>"
#
# The python base image/tag mapping is slightly different (the base images are shared by all Prod/Build/CI images)
# And python version is part of the tag.
#
# "apache/airflow:python-3.6 -> "apache/airflow-python-gcr-v1:3.6-slim-buster-<RUN_ID>"
#
# In case of Github Packages image must be part of the repository:
#
# * Prod Image: "apache/airflow:master-python3.8" -> "apache/airflow/master-python3.8-gcr-v1:<RUN_ID>"
# * Prod build image: "apache/airflow:master-python3.8-build" -> "apache/airflow/master-python3.8-build-gcr-v1:<RUN_ID>"
# * CI build image: "apache/airflow:master-python3.8-ci" -> "apache/airflow/master-python3.8-ci-gcr-v1:<RUN_ID>"
#
# The python base image/tag mapping is slightly different (the base images are shared by all
# Prod/Build/CI images) and python version is part of the tag.
#
# "apache/airflow:python-3.6 -> "apache/airflow/python/gcr-v1:3.6-slim-buster-<RUN_ID>"
export GITHUB_REGISTRY_AIRFLOW_CI_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_CI_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
export GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/python${GITHUB_REGISTRY_IMAGE_SUFFIX}:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
local image_name
image_name="${GITHUB_REGISTRY}/$(get_github_container_registry_image_prefix)"
local image_separator
if [[ ${GITHUB_REGISTRY} == "ghcr.io" ]]; then
image_separator="-"
elif [[ ${GITHUB_REGISTRY} == "docker.pkg.github.com" ]]; then
image_separator="/"
else
echo
echo "${COLOR_RED}ERROR: Bad value of '${GITHUB_REGISTRY}'. Should be either 'ghcr.io' or 'docker.pkg.github.com'!${COLOR_RESET}"
echo
exit 1
fi
export GITHUB_REGISTRY_AIRFLOW_PROD_IMAGE="${image_name}${image_separator}${AIRFLOW_PROD_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
export GITHUB_REGISTRY_AIRFLOW_PROD_BUILD_IMAGE="${image_name}-${AIRFLOW_PROD_BASE_TAG}${image_separator}build${GITHUB_REGISTRY_IMAGE_SUFFIX}"
export GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${image_name}${image_separator}python${GITHUB_REGISTRY_IMAGE_SUFFIX}:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
export GITHUB_REGISTRY_AIRFLOW_CI_IMAGE="${image_name}${image_separator}${AIRFLOW_CI_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
export GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${image_name}${image_separator}python${GITHUB_REGISTRY_IMAGE_SUFFIX}:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
}
# If GitHub Registry is used, login to the registry using GITHUB_USERNAME and GITHUB_TOKEN
function build_image::login_to_github_registry_if_needed() {
# If GitHub Registry is used, login to the registry using GITHUB_USERNAME and
# either GITHUB_TOKEN or CONTAINER_REGISTRY_TOKEN depending on the registry.
# In case Personal Access token is not set, skip logging in
# Also enable experimental features of docker (we need `docker manifest` command)
function build_image::configure_github_docker_registry() {
if [[ ${USE_GITHUB_REGISTRY} == "true" ]]; then
if [[ -n ${GITHUB_TOKEN=} ]]; then
start_end::group_start "Login to GitHub registry"
echo "${GITHUB_TOKEN}" | docker login \
start_end::group_start "Determine Github Registry token used and login if needed"
local token=""
if [[ "${GITHUB_REGISTRY}" == "ghcr.io" ]]; then
# For now ghcr.io can only authenticate using Personal Access Token with package access scope.
# There are plans to implement GITHUB_TOKEN authentication but this is not implemented yet
token="${CONTAINER_REGISTRY_TOKEN=}"
echo
echo "Using CONTAINER_REGISTRY_TOKEN!"
echo
elif [[ "${GITHUB_REGISTRY}" == "docker.pkg.github.com" ]]; then
token="${GITHUB_TOKEN}"
echo
echo "Using GITHUB_TOKEN!"
echo
else
echo
echo "${COLOR_RED}ERROR: Bad value of '${GITHUB_REGISTRY}'. Should be either 'ghcr.io' or 'docker.pkg.github.com'!${COLOR_RESET}"
echo
exit 1
fi
if [[ -z "${token}" ]] ; then
echo
echo "Skip logging in to Github Registry. No Token available!"
echo
fi
if [[ -n "${token}" ]]; then
echo "${token}" | docker login \
--username "${GITHUB_USERNAME:-apache}" \
--password-stdin \
"${GITHUB_REGISTRY}"
start_end::group_end
else
echo "Skip Login to GitHub Registry ${GITHUB_REGISTRY} as token is missing"
fi
echo "Make sure experimental docker features are enabled"
local new_config
new_config=$(jq '.experimental = "enabled"' "${HOME}/.docker/config.json")
echo "${new_config}" > "${HOME}/.docker/config.json"
echo "Docker config after change:"
echo "${new_config}"
start_end::group_end
fi
}
# Prepares all variables needed by the CI build. Depending on the configuration used (python version
# DockerHub user etc. the variables are set so that other functions can use those variables.
function build_images::prepare_ci_build() {
@ -433,7 +514,7 @@ function build_images::prepare_ci_build() {
export AIRFLOW_IMAGE="${AIRFLOW_CI_IMAGE}"
readonly AIRFLOW_IMAGE
build_image::login_to_github_registry_if_needed
build_image::configure_github_docker_registry
sanity_checks::go_to_airflow_sources
permissions::fix_group_permissions
}
@ -565,6 +646,18 @@ function build_images::rebuild_ci_image_if_needed_and_confirmed() {
fi
}
# Retrieves Github Container Registry image prefix from repository name
# GitHub Container Registry stores all images at the organization level, they are just
# linked to the repository via docker label - however we assume a convention where we will
# add repository name to organisation separated by '-' and convert everything to lowercase
# this is because in order for it to work for internal PR for users or other organisation's
# repositories, the other organisations and repositories can be uppercase
# container registry image name has to be lowercase
function get_github_container_registry_image_prefix() {
echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]'
}
# Builds CI image - depending on the caching strategy (pulled, local, disabled) it
# passes the necessary docker build flags via DOCKER_CACHE_CI_DIRECTIVE array
# it also passes the right Build args depending on the configuration of the build
@ -598,7 +691,7 @@ function build_images::build_ci_image() {
"--build-arg" "AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}"
)
if [[ "${AIRFLOW_CONSTRAINTS_LOCATION}" != "" ]]; then
if [[ -n "${AIRFLOW_CONSTRAINTS_LOCATION}" ]]; then
EXTRA_DOCKER_CI_BUILD_FLAGS+=(
"--build-arg" "AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION}"
)
@ -625,21 +718,20 @@ Docker building ${AIRFLOW_CI_IMAGE}.
set +u
local additional_dev_args=()
if [[ ${DEV_APT_DEPS} != "" ]]; then
if [[ -n "${DEV_APT_DEPS}" ]]; then
additional_dev_args+=("--build-arg" "DEV_APT_DEPS=\"${DEV_APT_DEPS}\"")
fi
if [[ ${DEV_APT_COMMAND} != "" ]]; then
if [[ -n "${DEV_APT_COMMAND}" ]]; then
additional_dev_args+=("--build-arg" "DEV_APT_COMMAND=\"${DEV_APT_COMMAND}\"")
fi
local additional_runtime_args=()
if [[ ${RUNTIME_APT_DEPS} != "" ]]; then
if [[ -n "${RUNTIME_APT_DEPS}" ]]; then
additional_runtime_args+=("--build-arg" "RUNTIME_APT_DEPS=\"${RUNTIME_APT_DEPS}\"")
fi
if [[ ${RUNTIME_APT_COMMAND} != "" ]]; then
if [[ -n "${RUNTIME_APT_COMMAND}" ]]; then
additional_runtime_args+=("--build-arg" "RUNTIME_APT_COMMAND=\"${RUNTIME_APT_COMMAND}\"")
fi
docker build \
"${EXTRA_DOCKER_CI_BUILD_FLAGS[@]}" \
--build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
@ -661,6 +753,8 @@ Docker building ${AIRFLOW_CI_IMAGE}.
--build-arg INSTALL_FROM_DOCKER_CONTEXT_FILES="${INSTALL_FROM_DOCKER_CONTEXT_FILES}" \
--build-arg UPGRADE_TO_NEWER_DEPENDENCIES="${UPGRADE_TO_NEWER_DEPENDENCIES}" \
--build-arg CONTINUE_ON_PIP_CHECK_FAILURE="${CONTINUE_ON_PIP_CHECK_FAILURE}" \
--build-arg AIRFLOW_IMAGE_REPOSITORY="https://github.com/${GITHUB_REPOSITORY}" \
--build-arg AIRFLOW_IMAGE_DATE_CREATED="$(date --rfc-3339=seconds | sed 's/ /T/')" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
"${additional_dev_args[@]}" \
@ -730,8 +824,7 @@ function build_images::prepare_prod_build() {
export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
readonly AIRFLOW_IMAGE
build_image::login_to_github_registry_if_needed
build_image::configure_github_docker_registry
AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="${BRANCH_NAME}"
sanity_checks::go_to_airflow_sources
}
@ -776,13 +869,12 @@ function build_images::build_prod_images() {
fi
set +u
local additional_dev_args=()
if [[ ${DEV_APT_DEPS} != "" ]]; then
if [[ -n "${DEV_APT_DEPS}" ]]; then
additional_dev_args+=("--build-arg" "DEV_APT_DEPS=\"${DEV_APT_DEPS}\"")
fi
if [[ ${DEV_APT_COMMAND} != "" ]]; then
if [[ -n "${DEV_APT_COMMAND}" ]]; then
additional_dev_args+=("--build-arg" "DEV_APT_COMMAND=\"${DEV_APT_COMMAND}\"")
fi
docker build \
"${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
--build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
@ -805,15 +897,17 @@ function build_images::build_prod_images() {
--build-arg CONTINUE_ON_PIP_CHECK_FAILURE="${CONTINUE_ON_PIP_CHECK_FAILURE}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
--build-arg AIRFLOW_IMAGE_REPOSITORY="https://github.com/${GITHUB_REPOSITORY}" \
--build-arg AIRFLOW_IMAGE_DATE_CREATED="$(date --rfc-3339=seconds | sed 's/ /T/')" \
"${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
-t "${AIRFLOW_PROD_BUILD_IMAGE}" \
--target "airflow-build-image" \
. -f Dockerfile
local additional_runtime_args=()
if [[ ${RUNTIME_APT_DEPS} != "" ]]; then
if [[ -n "${RUNTIME_APT_DEPS}" ]]; then
additional_runtime_args+=("--build-arg" "RUNTIME_APT_DEPS=\"${RUNTIME_APT_DEPS}\"")
fi
if [[ ${RUNTIME_APT_COMMAND} != "" ]]; then
if [[ -n "${RUNTIME_APT_COMMAND}" ]]; then
additional_runtime_args+=("--build-arg" "RUNTIME_APT_COMMAND=\"${RUNTIME_APT_COMMAND}\"")
fi
docker build \
@ -840,6 +934,8 @@ function build_images::build_prod_images() {
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
--build-arg AIRFLOW_IMAGE_REPOSITORY="https://github.com/${GITHUB_REPOSITORY}" \
--build-arg AIRFLOW_IMAGE_DATE_CREATED="$(date --rfc-3339=seconds | sed 's/ /T/')" \
"${additional_dev_args[@]}" \
"${additional_runtime_args[@]}" \
"${DOCKER_CACHE_PROD_DIRECTIVE[@]}" \

6
scripts/ci/libraries/_initialization.sh
Просмотреть файл

@ -491,8 +491,8 @@ function initialization::initialize_git_variables() {
function initialization::initialize_github_variables() {
# Defaults for interacting with GitHub
export USE_GITHUB_REGISTRY=${USE_GITHUB_REGISTRY:="false"}
export GITHUB_REGISTRY_IMAGE_SUFFIX="-v2"
export GITHUB_REGISTRY=${GITHUB_REGISTRY:="docker.pkg.github.com"}
export GITHUB_REGISTRY_IMAGE_SUFFIX=${GITHUB_REGISTRY_IMAGE_SUFFIX:="-v2"}
export GITHUB_REGISTRY=${GITHUB_REGISTRY:="ghcr.io"}
export GITHUB_REGISTRY_WAIT_FOR_IMAGE=${GITHUB_REGISTRY_WAIT_FOR_IMAGE:="false"}
export GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG:="latest"}
export GITHUB_REGISTRY_PUSH_IMAGE_TAG=${GITHUB_REGISTRY_PUSH_IMAGE_TAG:="latest"}
@ -860,7 +860,7 @@ function initialization::ga_output() {
}
function initialization::ga_env() {
if [[ ${GITHUB_ENV=} != "" ]]; then
if [[ -n "${GITHUB_ENV=}" ]]; then
echo "${1}=${2}" >>"${GITHUB_ENV}"
fi
}

60
scripts/ci/libraries/_push_pull_remove_images.sh
Просмотреть файл

@ -189,8 +189,7 @@ function push_pull_remove_images::push_ci_images_to_dockerhub() {
# Pushes Ci images and their tags to registry in GitHub
function push_pull_remove_images::push_ci_images_to_github() {
# Push image to GitHub registry with chosen push tag
# the PUSH tag might be:
# Push image to GitHub registry with the push tag:
# "${GITHUB_RUN_ID}" - in case of pull-request triggered 'workflow_run' builds
# "latest" - in case of push builds
AIRFLOW_CI_TAGGED_IMAGE="${GITHUB_REGISTRY_AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
@ -202,12 +201,21 @@ function push_pull_remove_images::push_ci_images_to_github() {
docker tag "${AIRFLOW_CI_IMAGE}" "${AIRFLOW_CI_SHA_IMAGE}"
push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_SHA_IMAGE}"
fi
# Push python image to GitHub registry with the push tag:
# X.Y-slim-buster-"${GITHUB_RUN_ID}" - in case of pull-request triggered 'workflow_run' builds
# X.Y-slim-buster - in case of push builds
PYTHON_TAG_SUFFIX=""
if [[ ${GITHUB_REGISTRY_PUSH_IMAGE_TAG} != "latest" ]]; then
PYTHON_TAG_SUFFIX="-${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
fi
docker tag "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
push_pull_remove_images::push_image_with_retries "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
# Label the python image for GCR, so that it is linked to the current project it is build in
echo "FROM ${PYTHON_BASE_IMAGE}" | \
docker build --label "org.opencontainers.image.source=https://github.com/${GITHUB_REPOSITORY}" \
-t "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}" -
push_pull_remove_images::push_image_with_retries \
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
}
@ -264,8 +272,8 @@ function push_pull_remove_images::push_prod_images() {
fi
}
# waits for an image to be available in the GitHub registry
function push_pull_remove_images::wait_for_github_registry_image() {
# waits for an image to be available in GitHub Packages
function push_pull_remove_images::wait_for_image_in_github_packages() {
local github_repository_lowercase
github_repository_lowercase="$(echo "${GITHUB_REPOSITORY}" |tr '[:upper:]' '[:lower:]')"
local github_api_endpoint
@ -294,6 +302,46 @@ function push_pull_remove_images::wait_for_github_registry_image() {
verbosity::print_info "Found ${image_name_in_github_registry}:${image_tag_in_github_registry} image"
}
# waits for an image to be available in GitHub Container Registry
function push_pull_remove_images::wait_for_image_in_github_container_registry() {
local image_name_in_github_registry="${1}"
local image_tag_in_github_registry=${2}
local image_to_wait_for="${GITHUB_REGISTRY}/${GITHUB_REPOSITORY}-${image_name_in_github_registry}:${image_tag_in_github_registry}"
echo
echo "Waiting for ${GITHUB_REGISTRY}/${GITHUB_REPOSITORY}-${image_name_in_github_registry}:${image_tag_in_github_registry} image"
echo
set +e
while true; do
docker manifest inspect "${image_to_wait_for}"
local res=$?
if [[ ${res} == "0" ]]; then
echo "${COLOR_GREEN}OK.${COLOR_RESET}"
break
else
echo "${COLOR_YELLOW}Still waiting for ${image_to_wait_for}!${COLOR_RESET}"
fi
sleep 30
done
set -e
verbosity::print_info "Found ${image_name_in_github_registry}:${image_tag_in_github_registry} image"
}
# waits for an image to be available in the GitHub registry
function push_pull_remove_images::wait_for_github_registry_image() {
if [[ ${GITHUB_REGISTRY} == "ghcr.io" ]]; then
push_pull_remove_images::wait_for_image_in_github_container_registry "${@}"
elif [[ ${GITHUB_REGISTRY} == "docker.pkg.github.com" ]]; then
push_pull_remove_images::wait_for_image_in_github_packages "${@}"
else
echo
echo "${COLOR_RED}ERROR: Bad value of '${GITHUB_REGISTRY}'. Should be either 'ghcr.io' or 'docker.pkg.github.com'!${COLOR_RESET}"
echo
exit 1
fi
}
function push_pull_remove_images::check_if_github_registry_wait_for_image_enabled() {
if [[ ${USE_GITHUB_REGISTRY} != "true" || ${GITHUB_REGISTRY_WAIT_FOR_IMAGE} != "true" ]]; then
echo

2
scripts/ci/selective_ci_checks.sh
Просмотреть файл

@ -136,7 +136,7 @@ function get_changed_files() {
echo
CHANGED_FILES=$(git diff-tree --no-commit-id --name-only \
-r "${INCOMING_COMMIT_SHA}^" "${INCOMING_COMMIT_SHA}" || true)
if [[ ${CHANGED_FILES} == "" ]]; then
if [[ -z "${CHANGED_FILES}" ]]; then
echo
echo "${COLOR_YELLOW}WARNING: Could not find any changed files ${COLOR_RESET}"
echo Assuming that we should run all tests in this case

6
scripts/ci/testing/ci_run_airflow_testing.sh
Просмотреть файл

@ -116,19 +116,19 @@ function prepare_tests_to_run() {
fi
readonly DOCKER_COMPOSE_LOCAL
if [[ ${TEST_TYPE=} != "" ]]; then
if [[ -n "${TEST_TYPE=}" ]]; then
# Handle case where test type is passed from outside
export TEST_TYPES="${TEST_TYPE}"
fi
if [[ ${TEST_TYPES=} == "" ]]; then
if [[ -z "${TEST_TYPES=}" ]]; then
TEST_TYPES="Core Providers API CLI Integration Other WWW Heisentests"
echo
echo "Test types not specified. Running all: ${TEST_TYPES}"
echo
fi
if [[ ${TEST_TYPE=} != "" ]]; then
if [[ -n "${TEST_TYPE=}" ]]; then
# Add Postgres/MySQL special test types in case we are running several test types
if [[ ${BACKEND} == "postgres" ]]; then
TEST_TYPES="${TEST_TYPES} Postgres"

2
scripts/docker/compile_www_assets.sh
Просмотреть файл

@ -36,7 +36,7 @@ function compile_www_assets() {
elif [[ -f "${airflow_site_package}/www/package.json" ]]; then
www_dir="${airflow_site_package}/www"
fi
if [[ ${www_dir} != "" ]]; then
if [[ -n "${www_dir}" ]]; then
pushd ${www_dir} || exit 1
yarn install --frozen-lockfile --no-cache
yarn run prod

2
scripts/docker/install_airflow.sh
Просмотреть файл

@ -56,7 +56,7 @@ function install_airflow() {
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade --upgrade-strategy eager \
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
if [[ ${AIRFLOW_INSTALL_EDITABLE_FLAG} != "" ]]; then
if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
# Remove airflow and reinstall it using editable flag
# We can only do it when we install airflow from sources
pip uninstall apache-airflow --yes

8
scripts/docker/install_from_docker_context_files.sh
Просмотреть файл

@ -39,7 +39,7 @@ function install_airflow_and_providers_from_docker_context_files(){
reinstalling_apache_airflow_package=$(ls \
/docker-context-files/apache?airflow?[0-9]*.{whl,tar.gz} 2>/dev/null || true)
# Add extras when installing airflow
if [[ "${reinstalling_apache_airflow_package}" != "" ]]; then
if [[ -n "${reinstalling_apache_airflow_package}" ]]; then
reinstalling_apache_airflow_package="${reinstalling_apache_airflow_package}[${AIRFLOW_EXTRAS}]"
fi
@ -47,8 +47,8 @@ function install_airflow_and_providers_from_docker_context_files(){
local reinstalling_apache_airflow_providers_packages
reinstalling_apache_airflow_providers_packages=$(ls \
/docker-context-files/apache?airflow?providers*.{whl,tar.gz} 2>/dev/null || true)
if [[ ${reinstalling_apache_airflow_package} == "" && \
${reinstalling_apache_airflow_providers_packages} == "" ]]; then
if [[ -z "${reinstalling_apache_airflow_package}" && \
-z "${reinstalling_apache_airflow_providers_packages}" ]]; then
return
fi
@ -95,7 +95,7 @@ install_all_other_packages_from_docker_context_files() {
# shellcheck disable=SC2010
reinstalling_other_packages=$(ls /docker-context-files/*.{whl,tar.gz} 2>/dev/null | \
grep -v apache_airflow | grep -v apache-airflow || true)
if [[ "${reinstalling_other_packages}" != "" ]]; then \
if [[ -n "${reinstalling_other_packages}" ]]; then \
pip install ${AIRFLOW_INSTALL_USER_FLAG} --force-reinstall --no-deps --no-index ${reinstalling_other_packages}
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"

4
scripts/in_container/_in_container_utils.sh
Просмотреть файл

@ -405,12 +405,12 @@ function verify_suffix_versions_for_package_preparation() {
VERSION_SUFFIX_FOR_SVN=${VERSION_SUFFIX_FOR_SVN:=""}
if [[ ${VERSION_SUFFIX_FOR_PYPI} != "" ]]; then
if [[ -n "${VERSION_SUFFIX_FOR_PYPI}" ]]; then
echo
echo "Version suffix for PyPI = ${VERSION_SUFFIX_FOR_PYPI}"
echo
fi
if [[ ${VERSION_SUFFIX_FOR_SVN} != "" ]]; then
if [[ -n "${VERSION_SUFFIX_FOR_SVN}" ]]; then
echo
echo "Version suffix for SVN = ${VERSION_SUFFIX_FOR_SVN}"
echo

2
scripts/in_container/run_ci_tests.sh
Просмотреть файл

@ -62,7 +62,7 @@ ${COLOR_RED}ERROR:
* ./breeze --backend ${BACKEND} ${EXTRA_ARGS}--python ${PYTHON_MAJOR_MINOR_VERSION} --db-reset --skip-mounting-local-sources --test-type ${TEST_TYPE} shell
*
"""
if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG=} != "" ]]; then
if [[ -n "${GITHUB_REGISTRY_PULL_IMAGE_TAG=}" ]]; then
echo """
* When you do not have sources:
*

2
scripts/in_container/run_install_and_test_provider_packages.sh
Просмотреть файл

@ -24,7 +24,7 @@ function verify_parameters(){
echo "Testing if all classes in import packages can be imported"
echo
if [[ ${INSTALL_AIRFLOW_VERSION=""} == "" ]]; then
if [[ -z "${INSTALL_AIRFLOW_VERSION=""}" ]]; then
echo
echo "${COLOR_RED}ERROR: You have to specify airflow version to install.${COLOR_RESET}"
echo

4
scripts/in_container/run_prepare_provider_packages.sh
Просмотреть файл

@ -155,7 +155,7 @@ function build_provider_packages() {
echo "-----------------------------------------------------------------------------------"
set +e
package_suffix=""
if [[ ${VERSION_SUFFIX_FOR_SVN} == "" && ${VERSION_SUFFIX_FOR_PYPI} != "" ]]; then
if [[ -z "${VERSION_SUFFIX_FOR_SVN}" && -n ${VERSION_SUFFIX_FOR_PYPI} ]]; then
# only adds suffix to setup.py if version suffix for PyPI is set but the SVN one is not
package_suffix="${VERSION_SUFFIX_FOR_PYPI}"
fi
@ -181,7 +181,7 @@ function rename_packages_if_needed() {
pushd dist >/dev/null 2>&1 || exit 1
if [[ ${FILE_VERSION_SUFFIX} != "" ]]; then
if [[ -n "${FILE_VERSION_SUFFIX}" ]]; then
# In case we have FILE_VERSION_SUFFIX we rename prepared files
if [[ "${PACKAGE_FORMAT}" == "sdist" || "${PACKAGE_FORMAT}" == "both" ]]; then
for FILE in *.tar.gz