* Adds missing schema for kerberos sidecar configuration
The kerberos support added in #11130 did not have schema added
to the values.yml. This PR fixes it.
Co-authored-by: Jacob Ferriero <jferriero@google.com>
* Update chart/values.schema.json
Co-authored-by: Jacob Ferriero <jferriero@google.com>
Adds SSL configuration for PGBouncer in the Helm Chart. PGBouncer
is crucial to handle the big number of connections that airflow
opens for the database, but often the database is outside of the
Kubernetes Cluster or generally the environment where Airflow is
installed and PGBouncer needs to connect securely to such database.
This PR adds capability of seting CA/Certificate and Private Key
in the PGBouncer configuration that allows for mTLS authentication
(both client and server are authenticated) and secure connection
even over public network.
We can now add annotations to the service accounts in a generic
way. This allows for example to add Workflow Identitty in GKE
environment but it is not limited to it.
Co-authored-by: Kamil Breguła <kamil.bregula@polidea.com>
Co-authored-by: Jacob Ferriero <jferriero@google.com>
Co-authored-by: Kamil Breguła <kamil.bregula@polidea.com>
Users of the CeleryKubernetesExecutor will require both
Celery and Kubernetes features to launch tasks.
This PR will also serve as the basis for integration tests for this
executor
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
* Multi-Namespace mode for helm chart
Users should not REQUIRE a ClusterRole/ClusterRolebinding
to run airflow via helm. This change will allow "single" and "multi"
namespace modes so users can add airflow to managed kubernetes clusters
* add namespace to role
* add rolebinding too
* add docs
* add values.schema.json change
Some of the users of Airflow are using Kerberos to authenticate
their worker workflows. Airflow has a basic support for Kerberos
for some of the operators and it has support to refresh the
temporary Kerberos tokens via `airflow kerberos` command.
This change adds support for the Kerberos side-car that connects
to the Kerberos Key Distribution Center and retrieves the
token using Keytab that should be deployed as Kubernetes Secret.
It uses shared volume to share the temporary token. The nice
thing about setting it up as a sidecar is that the Keytab
is never shared with the workers - the secret is only mounted
by the sidecar and the workers have only access to the temporary
token.
Depends on #11129
Webserver did not have a Kubernetes Service Account defined and
while we do not strictly need to use the service account for
anything now, having the Service Account defined allows to
define various capabilities for the webserver.
For example when you are in the GCP environment, you can map
the Kubernetes service account into a GCP one, using
Workload Identity without the need to define any secrets
and performing additional authentication.
Then you can have that GCP service account get
the permissions to write logs to GCS bucket. Similar mechanisms
exist in AWS and it also opens up on-premises configuration.
See more at
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
Co-authored-by: Jacob Ferriero <jferriero@google.com>
Co-authored-by: Jacob Ferriero <jferriero@google.com>
There was a problem with Mac version of pgbouncer exporter
created and released previously. This commit releases the
latest version making sure that Linux Go is used to build
the pgbouncer binary.
There was a problem with user in Git Sync mode of the Helm Chart
in connection with the git sync image and official Airflow
image. Since we are using the official image, most of the
containers are run with the "50000" user, but the git-sync image
used by the git sync user is 65533 so we have to set it as
default. We also exposed that value as parameter, so that
another image could be used here as well.
The celery flower and worker commands have changed in Airflow 2.0.
The Helm Chart supported only 1.10 version of those commands and
this PR fixes it by adding both variants of them.
* Modify helm chart to use pod_template_file
Since we are deprecating most k8sexecutor arguments
we should use the pod_template_file when launching airflow
using the KubernetesExecutor
* fix tests
* one more nit
* fix dag command
* fix pylint
The hadolint check only checked the "main dir" Dockerfile
but we have more of them now. All of them are now checked.
The following problems are fixed:
* DL3000 Use absolute WORKDIR
* DL4000 MAINTAINER is deprecated
* DL4006 Set the SHELL option -o pipefail before RUN with a pipe in it.
* SC2046 Quote this to prevent word splitting.
The followiing problems are ignored:
* DL3018 Pin versions in apk add. Instead of `apk add <package>` use `apk add
<package>=<version>`
We can now build all the images from Airlfow sources in
a reproducible fashion and our users can use the helm chart
based on the images build from official images + code in
Airflow Codebase.
We also have consistent versioning scheme based on
calver version of releasing the images coupled with
the version of the original package.
Part of #9401
Rather than only allowing specific pre-determined config settings, this
change allows the user to place _any_ config setting they like in the
generated airflow.cfg, including overwriting the "generated defaults".
This providers a nicer interface for the users of the chart (even if the
could already set these via the env vars).
* add git sync sidecars
* add a helm test
* add more tests
* allow users to provide git username and pass via a k8s secrets
* set default values for airflow worker repository & tag
* change ci timeout
* fix link
* add credentials_secret to airflow.cfg configmap
* set GIT_SYNC_ADD_USER on kubernetes worker pods, set uid
* add fsGroup to webserver and kubernete workers
* move gitSync to dags.gitSync
* rename valueFields
* turn off git sync and dag persistence by default
* provide option to specify known_hosts
* add git-sync details into the chart documentation
* Update .gitignore
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* make git sync max failures configurable
* Apply suggestions from code review
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* add back requirements.lock
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Kamil Breguła
FloChehab
Prakshal Jain
Fernando Llaca
Daniel Imberman
Kaxil Naik
Jarek Potiuk
Ruben Laguna
flvndh
Jacob Ferriero
Kamil Olszewski
gardnerdev
Can Güney Aksakalli
David Cavaletto
Muller Hsu
Vicken Simonian
Ash Berlin-Taylor
Daniel Debny
QP Hou
Aneesh Joseph