- add howto docs for S3ToRedshift example dag
- add terraform which runs terraform CLI commands in an isolated docker container
NOTE: This system test uses terraform to provide the infrastructure needed to run this example dag.
Tests requiring Kubernetes Cluster are now moved out of
the regular CI tests and moved to "kubernetes_tests" folder
so that they can be run entirely on host without having
the CI image built at all. They use production image
to run the tests on KinD cluster and we add tooling
to start/stop/deploy the application to the KinD cluster
automatically - for both CI testing and local development.
This is a pre-requisite to convert the tests to convert the
tests to use the official Helm Chart and Docker images or
Apache Airflow.
It closes#8782
* Better content of backport packages CHANGELOG and INSTALL files
The content of Backport Packages CHANGELOG.txt and INSTALL files
has been updated to reflect that those are not full Airflow
releases.
1) Source package:
- INSTALL contains only references to preparing backport packages
- CHANGELOG.txt contains combined change log of all the packages
2) Binary packages:
- No INSTALL
- CHANGELOG.txt contains changelog for this package only
3) Whl packages
No change
* Update backport_packages/INSTALL
* Fixed problem that Kubernetes tests were testing latest master
rather than what came from the local sources.
* Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env
* Moved Kubernetes scripts to 'in_container' dir where they belong now
* Kubernetes cluster is reused until it is stopped
* Kubernetes image is build from image in docker already + mounted sources
* Kubectl version name is corrected in the Dockerfile
* KUBERNETES_VERSION can now be used to select Kubernetes version
* Running kubernetes scripts is now easy in Breeze
* We can start/recreate/stop cluster using --<ACTION>-kind-cluster
* Instructions on how to run Kubernetes tests are updated
* The old "bare" environment is replaced by --no-deps switch
* [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing
* Fixed problem that Kubernetes tests were testing latest master
rather than what came from the local sources.
* Moved Kubernetes scripts to 'in_container' dir where they belong now
* Kubernetes tests are now better suited for running locally
* Kubernetes cluster is not deleted until environment is stopped
* Kubernetes image is built outside of the container and passed as .tar
* Kubectl version name is corrected in the Dockerfile
* Kubernetes Version can be used to select Kubernetes versio
* Running kubernetes scripts is now easy in Breeze
* Instructions on how to run Kubernetes tests are updated
* Better flags in Breeze are used to run Kubernetes environment/tests
* The old "bare" environment is replaced by --no-deps switch
Airflow tests depend on many external services and other custom setup,
which makes it hard for contributors to work on this codebase. CI
builds have also been unreliable, and it is hard to reproduce the
causes. Having contributors trying to emulate the build environment
every time makes it easier to get to an "it works on my machine" sort
of situation.
This implements a dockerised version of the current build pipeline.
This setup has a few advantages:
* TravisCI tests are reproducible locally
* The same build setup can be used to create a local development environment
[AIRFLOW-1899] Add full deployment
- Made home directory configurable
- Documentation fix
- Add licenses
[AIRFLOW-1899] Tests for the Kubernetes Executor
Add an integration test for the Kubernetes
executor. Done by
spinning up different versions of kubernetes and
run a DAG
by invoking the REST API
Closes#3301 from Fokko/fix-kubernetes-executor
Moving the sensors to seperate files increases
readability of the
code. Also this reduces the code in the big
core.py file.
Closes#2875 from Fokko/AIRFLOW-1889-move-sensors-
to-separate-package
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-198
Testing Done:
- Local testing of dag operation with
LatestOnlyOperator
- Unit test added
Closes#1752 from gwax/latest_only
Jarek Potiuk
Kaxil Naik
Felix Uellendall
Kousuke Saruta
dstandish
Tomek
Daniel Imberman
Kamil Breguła
Joshua Carp
Jarek Potiuk
Gerardo Curiel
Verdan Mahmood
r39132
Fokko Driesprong
grantnicholas
nyeganeh
Kevin Yang
Tao feng
George Leslie-Waksman
Shintaro Murakami
Alex Guziel
Siddharth Anand
Waldemar Hummer
Bolke de Bruin
asnir
Thoralf Gutierrez
Svend Vanderveken
Jeremiah Lowin
Arthur Wiedmer
Mike Kerzhner
Maxime Beauchemin