The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
Local caching is now default strategy when building
the Production image.
You can still change it to pulled - similar to CI builds
by providing the right build flag and this is what
is used in CI by default. The flags in Breeze are now updated
to be more eplanatory and friendly (build-cache-*) and a flag
for "disabled" cache option is added as well.
Also the Dockerfile and Dockerfile.ci files are not needed
any more in the docker context. They used to be needed when
we built the Kubernetes image in the container, but since
we are now using production image directly - we do not need
them any nmore.
Combining setting the default strategy to local and removing
the Dockerfile from the context has the nice effect that you
can iterate much faster on the Production image without
triggering rebuilds of half of the docker image
as soon as the Dockerfile changes.
OpenShift (and other Kubernetes platforms) often use the approach
that they start containers with random user and root group. This is
described in the https://docs.openshift.com/container-platform/3.7/creating_images/guidelines.html
All the files created by the "airflow" user are now belonging to
'root' group and the root group has the same access to those
files as the Airflow user.
Additionally, the random user gets automatically added
/etc/passwd entry which is name 'default'. The name of the user
can be set by setting the USER_NAME variable when starting the
container.
Closes#9248Closes#8706
For a long time the way how entrypoint worked in ci scripts
was wrong. The way it worked was convoluted and short of black
magic. This did not allow to pass multiple test targets and
required separate execute command scripts in Breeze.
This is all now straightened out and both production and
CI image are always using the right entrypoint by default
and we can simply pass parameters to the image as usual without
escaping strings.
This also allowed to remove some breeze commands and
change names of several flags in Breeze to make them more
meaningful.
Both CI and PROD image have now embedded scripts for log
cleaning.
History of image releases is added for 1.10.10-*
alpha quality images.
The commit 5918efc86a broke
optimisation of the CI image - using the Apache Airflow
master branch as a base package installation source from PyPI.
This commit restores it including removal of the
obsolete CI_OPTIMISED arg - as now we have a separate
production and CI image and CI image is by default
CI_OPTIMISED
Tests requiring Kubernetes Cluster are now moved out of
the regular CI tests and moved to "kubernetes_tests" folder
so that they can be run entirely on host without having
the CI image built at all. They use production image
to run the tests on KinD cluster and we add tooling
to start/stop/deploy the application to the KinD cluster
automatically - for both CI testing and local development.
This is a pre-requisite to convert the tests to convert the
tests to use the official Helm Chart and Docker images or
Apache Airflow.
It closes#8782
It also installs properly on Mac as well as it auto-detects
if yarn prod is needed - based on presence of proper
package.json in either www or www_rbac which makes it simpler
for remote installations.
Jarek Potiuk
zikun
Fabian
Felix Uellendall
Kaxil Naik