* KubernetesPodOperator can retry log tailing in case of interruption
* fix failing test
* change read_pod_logs method formatting
* KubernetesPodOperator retry log tailing based on last read log timestamp
* fix test_parse_log_line test formatting
* add docstring to parse_log_line method
* fix kubernetes integration test
* Simplify Airflow on Kubernetes Story
Removes thousands of lines of code that essentially ammount to us
re-creating the Kubernetes API. Will offer a faster, simpler
KubernetesExecutor for 2.0
* Fix podgen tests
* fix documentation
* simplify validate function
* @mik-laj comments
* spellcheck
* spellcheck
* Update airflow/executors/kubernetes_executor.py
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Ensure that K8sPodOperator can pull namespace from pod_template_file
Fixes a bug where users who run K8sPodOperator could not run because
the operator was expecting a namespace parameter
* add test
* self.pod
* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* don't create pod until run
* spellcheck
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* Modify helm chart to use pod_template_file
Since we are deprecating most k8sexecutor arguments
we should use the pod_template_file when launching airflow
using the KubernetesExecutor
* fix tests
* one more nit
* fix dag command
* fix pylint
* Add podOverride setting for KubernetesExecutor
Users of the KubernetesExecutor will now have a "podOverride"
option in the executor_config. This option will allow users to
modify the pod launched by the KubernetesExecutor using a
`kubernetes.client.models.V1Pod` class. This is the first step
in deprecating the tradition executor_config.
* Fix k8s tests
* fix docs
* Make Kubernetes tests pass locally
Currently Kuberentes tests only all pass within breeze.
This PR makes them read the local path so they can pass in any
system.
* static tests
* add git sync sidecars
* add a helm test
* add more tests
* allow users to provide git username and pass via a k8s secrets
* set default values for airflow worker repository & tag
* change ci timeout
* fix link
* add credentials_secret to airflow.cfg configmap
* set GIT_SYNC_ADD_USER on kubernetes worker pods, set uid
* add fsGroup to webserver and kubernete workers
* move gitSync to dags.gitSync
* rename valueFields
* turn off git sync and dag persistence by default
* provide option to specify known_hosts
* add git-sync details into the chart documentation
* Update .gitignore
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* make git sync max failures configurable
* Apply suggestions from code review
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* add back requirements.lock
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
When using KubernetesExecutor without any centralized PV for log storage, one has to wait until the logs get uploaded to cloud storage before viewing them on UI. With this change, the webserver will try to fetch logs from running worker pods and display them.
* [AIRFLOW-5147] extended character set for for k8s worker pods annotations
* updated UPDATING.md with new breaking changes
* excluded pylint too-many-statement check from constructor due to its nature
* [AIRFLOW-4781] Added the ability to specify ports in kubernetesOperator
* [AIRFLOW-4781] Added the ability to specify ports in kubernetesOperator
* [AIRFLOW-4781] Added the ability to specify ports in kubernetesOperator
added docstring
* [AIRFLOW-4781] Added the ability to specify ports in kubernetesOperator
add typehints
Co-Authored-By: Fokko Driesprong <fokko@driesprong.frl>
* [AIRFLOW-4781] Added the ability to specify ports in kubernetesOperator
fixed docstrings and typehints
* Move k8s executor from contrib folder
Considering that the k8s executor is now fully supported by core
committers, we should move it from contrib to the primary executor
directory.
Kaxil Naik
Kamil Breguła
Michał Misiewicz
Daniel Imberman
Jarek Potiuk
Ping Zhang
Aneesh Joseph
Sumit Maheshwari
Stijn De Haes
Brandon T. Willard
Leonardo Alves Miguel
Pete DeJoy
Matt Buell
davlum
MatthewRBruce
Tomek
George Miller
Ke Zhu
Qingping Hou
Kevin Yang
Andrei
Marcin Szymański
CatarinaSilva
Andrii Soldatenko
Ron van Weverwijk
Bruno Campos
Olivier Van Goethem
Daniel Imberman