Since we are mountign tmp dir now to inside container, some
of the remnants of what's going on inside remains after exit.
This is particularly bad if you are using tmux (some of the
directories remaining there prevent tmux from re-run)
This change cleans up /tmp directory on exit from Breeze command.
It does it from inside container so that we clean up all
root-owned files without sudo.
When quick-fixing Python 3.8.4 error #9820 PYTHON_BASE_IMAGE_VERSION
variable was added but it was initialized too early in Breeze and
it took the default version of Python rather than the one chosen
by --python switch. This caused the generated requirements
(locally by Breeze only) to generate wrong set of requirements
and images built locally for different python versions were
based on default Python version, not the one chosen by --python
switch.
* Constraint files are now maintained automatically
* No need to generate requirements when setup.py changes
* requirements are kept in separate orphan branches not in main repo
* merges to master verify if latest requirements are working and
push tested requirements to orphaned branches
* we keep history of requirement changes and can label them
individually for each version (by constraint-1.10.n tag name)
* consistently changed all references to be 'constraints' not
'requirements'
The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
Local caching is now default strategy when building
the Production image.
You can still change it to pulled - similar to CI builds
by providing the right build flag and this is what
is used in CI by default. The flags in Breeze are now updated
to be more eplanatory and friendly (build-cache-*) and a flag
for "disabled" cache option is added as well.
Also the Dockerfile and Dockerfile.ci files are not needed
any more in the docker context. They used to be needed when
we built the Kubernetes image in the container, but since
we are now using production image directly - we do not need
them any nmore.
Combining setting the default strategy to local and removing
the Dockerfile from the context has the nice effect that you
can iterate much faster on the Production image without
triggering rebuilds of half of the docker image
as soon as the Dockerfile changes.
For a long time the way how entrypoint worked in ci scripts
was wrong. The way it worked was convoluted and short of black
magic. This did not allow to pass multiple test targets and
required separate execute command scripts in Breeze.
This is all now straightened out and both production and
CI image are always using the right entrypoint by default
and we can simply pass parameters to the image as usual without
escaping strings.
This also allowed to remove some breeze commands and
change names of several flags in Breeze to make them more
meaningful.
Both CI and PROD image have now embedded scripts for log
cleaning.
History of image releases is added for 1.10.10-*
alpha quality images.
* Improved cloud tool available in the trimmed down CI container
The tools now have shebangs which make them available for
python tools. Also /opt/airflow is now mounted from the
host Airflow sources which makes it possible for the tools to
copy files directly to/from the sources of Airflow.
It also contains one small change for Linux users - the files
created by docker gcloud are created with root user so in order to fix
that the directories mounted from the host are fixed when you exit
the tool - their ownership is changed to be owned by the host user
Tests requiring Kubernetes Cluster are now moved out of
the regular CI tests and moved to "kubernetes_tests" folder
so that they can be run entirely on host without having
the CI image built at all. They use production image
to run the tests on KinD cluster and we add tooling
to start/stop/deploy the application to the KinD cluster
automatically - for both CI testing and local development.
This is a pre-requisite to convert the tests to convert the
tests to use the official Helm Chart and Docker images or
Apache Airflow.
It closes#8782
Breeze had --push-images switch to also push images to repo
but it was often needed to build and push images separately.
We have now a possibility to push an already built image with
separate push-image command instead and also you can choose
to push to cache registry in GitHub rather than to DockerHub
with --registry-cache switch.
scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR
is the same as AIRFLOW_SOURCES. When jumping back-and-forth between
ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track
of what is what.
This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top
level folder instead -- that means I don't have to keep as much context
in my head
We have now mechanism to keep release notes updated for the
backport operators in an automated way.
It really nicely generates all the necessary information:
* summary of requirements for each backport package
* list of dependencies (including extras to install them) when package
depends on other providers packages
* table of new hooks/operators/sensors/protocols/secrets
* table of moved hooks/operators/sensors/protocols/secrets with
information where they were moved from
* changelog of all the changes to the provider package (this will be
automatically updated with incremental changelog whenever we decide to
release separate packages.
The system is fully automated - we will be able to produce release notes
automatically (per-package) whenever we decide to release new version of
the package in the future.
* Help for breeze commands contain relevant flags.
* fixup! Help for breeze commands contain relevant flags.
* fixup! fixup! Help for breeze commands contain relevant flags.
Jarek Potiuk
Beni Ben zikry
Alexander Sutcliffe
zikun
James Timmins
Tomek Urbaszek
Ash Berlin-Taylor
Felix Uellendall
Kamil Breguła
Jiajie Zhong
Ben Anderson