Recently we hard code in tutorial.rst which
is hard to maintain, such as `set_upstream`
is change to shift in tutorial.py but still
in tutorial.rst. Use sphinx is a better way
We already change Airflow DAG default owner to 'airflow'
in https://github.com/apache/airflow/pull/4151 but some
of our example DAGs and docs are still using
owner = 'Airflow', This patch to unify them
Added Amazon AWS how-to documentation scaffolding, plus example DAGs for AWS DataSync Operators with their respective how-to guides.
Reworked AWS DataSync operators into a single, logically idempotent operator.
This improves the ability to configure AirFlow
using Kubernetes best practices. You can provide
for exemple AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD
referencing a shell script that computes the
connection string using Kubernetes secrets.
And that script can be provided to the container
using a configmap.
Adding a unit test to check that an option that
should NOT be overriden by a command is correctly
only read from the configuration.
* Add support for init-containers to Kubernetes Pod Operator
Enables start-up related code to be added for an app container in K8s Pod operator.
Add new init_container resource that can be attached to the K8s Pod.
* Update init_container and fix tests
Fix the error in init_container and the associated tests.
* Refactor and fix tests
Fix tests for init_containers
* Fix init_container test errors
Remove unused mocks in init_container test
* Fix init_container test errors
Update volume mount object used in init_container test
* Fix init_container test errors
Add the missing volume setup for the init_container test.
* Fix init_container test failure.
Fix the expected result in the init_container test.
* Fix init_container test failures
Update expected results in the init_container tests
* Update the KubernetesPodOperator guide
Update the KubernetesPodOperator guide to document support for init containers
* Update init-container tests
Fix test failures casued due python versions by sorting the output before assert test.
* Update init-container to use k8s V1Container object
Remove custom object InitContainer.
Allow users to pass List[k8s.V1Container] as init-container in K8sPodOperator
* Add missing init_containers initalization in K8s pod operator
Due to rebase from master, certain sections of the kubernetes_pod_operator.py file was refactored which led to missing init_containers initalization in K8s pod operator. Add missing init_containers initalization in K8s pod operator. Update kubernetes pod operator configurations in init container test.
Support getting variables in templates by string. This is necessary when
fetching variables with characters not allowed in a class attribute
name. We can then also support returning default values when a variable does
not exist.
Adds new executor that is meant to be used mainly
for debugging and DAG development purposes. This
executor executes single task instance at time and
is able to work with SQLLite and sensors.
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI
It was not deterministic because pylint usually uses as many processors as
many are available and it splits the list of .py files between the separate
pylint processors - depending on how the split is done, pylint check might
or might not detect it. The cycle is always detected when all files are used.
In order to make it more deterministic, all pylint and mypy errors were resolved
in all executors package and in dag_processor.
At the same time plugins_manager had also been moved out of the executors
and all of the operators/hooks/sensors/macros because it was also causing
cyclic dependencies and it's far easier to untangle those dependencies
in executor when we move the intialisation of all plugins to plugins_manager.
Additionally require_serial is set in pre-commit configuration to
make sure cycle detection is deterministic.
This simplifies the lineage API which was needlessy cluttered.
You can now set "inlets='auto'" rather than "inlets={'auto': True}"
and airflow will figure out what to do.
Co-Authored-By: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-Authored-By: Kaxil Naik <kaxilnaik@gmail.com>
* [AIRFLOW-YYY] Lazy load API Client
* [AIRFLOW-YYY] Introduce order in CLI's function names
* [AIRFLOW-YYY] Create cli package
* [AIRLFOW-YYY] Move user and roles command to seperate files
* [AIRLFOW-YYY] Move sync_perm command to seperate file
* [AIRLFOW-YYY] Move task commands to separate file
* [AIRLFOW-YYY] Move pool commands to separate file
* [AIRLFOW-YYY] Move variable commands to separate file
* [AIRLFOW-YYY] Move db commands to separate file
* fixup! [AIRLFOW-YYY] Move variable commands to separate file
* [AIRLFOW-YYY] Move connection commands to separate file
* [AIRLFOW-YYY] Move version command to separate file
* [AIRLFOW-YYY] Move scheduler command to separate file
* [AIRLFOW-YYY] Move worker command to separate file
* [AIRLFOW-YYY] Move webserver command to separate file
* [AIRLFOW-YYY] Move dag commands to separate file
* [AIRLFOW-YYY] Move serve logs command to separate file
* [AIRLFOW-YYY] Move flower command to separate file
* [AIRLFOW-YYY] Move kerberos command to separate file
* [AIRFLOW-YYY] Lazy load CLI commands
* [AIRFLOW-YYY] Fix migration
* fixup! [AIRFLOW-YYY] Fix migration
* fixup! fixup! [AIRFLOW-YYY] Fix migration
dstandish
Jiajie Zhong
Michał Słowikowski
Gaurav Sehgal
Bjorn Olsen
Noël Bardelot
Darren Weber
Marina Pereira
Tomek
Daniel Huang
Rich Dean
pradeepbhadani
kaverisharma09
Avetisyan Sevak
Sergio Kef
Tim Paine
TobKed
svivier-orange
Kaxil Naik
Jarek Potiuk
Kartik Khare
bolkedebruin
Daphna Shezaf
Kamil Breguła
Rich Dean
Qingping Hou