We can now build all the images from Airlfow sources in
a reproducible fashion and our users can use the helm chart
based on the images build from official images + code in
Airflow Codebase.
We also have consistent versioning scheme based on
calver version of releasing the images coupled with
the version of the original package.
Part of #9401
Rather than only allowing specific pre-determined config settings, this
change allows the user to place _any_ config setting they like in the
generated airflow.cfg, including overwriting the "generated defaults".
This providers a nicer interface for the users of the chart (even if the
could already set these via the env vars).
* add git sync sidecars
* add a helm test
* add more tests
* allow users to provide git username and pass via a k8s secrets
* set default values for airflow worker repository & tag
* change ci timeout
* fix link
* add credentials_secret to airflow.cfg configmap
* set GIT_SYNC_ADD_USER on kubernetes worker pods, set uid
* add fsGroup to webserver and kubernete workers
* move gitSync to dags.gitSync
* rename valueFields
* turn off git sync and dag persistence by default
* provide option to specify known_hosts
* add git-sync details into the chart documentation
* Update .gitignore
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* make git sync max failures configurable
* Apply suggestions from code review
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* add back requirements.lock
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
Kaxil Naik
Kamil Olszewski
Jarek Potiuk
gardnerdev
flvndh
Can Güney Aksakalli
David Cavaletto
Muller Hsu
Vicken Simonian
Ash Berlin-Taylor
Daniel Imberman
Daniel Debny
QP Hou
Aneesh Joseph
Bryant Larsen
Greg Neiheisel