* Make the KubernetesPodOperator backwards compatible
This PR significantly reduces the pain of upgrading to Airflow 2.0
for users of the KubernetesPodOperator. Users will be allowed to
continue using the airflow.kubernetes custom classes
* spellcheck
* spelling
* clean up unecessary files in 1.10
* clean up unecessary files in 1.10
* clean up unecessary files in 1.10
This changes XComArg string representation from 'task_instance.pull(...)'
to '{{ task_instance.xcom_pull(...) }}' so users can use XComArgs with
f-string (and other) in simpler way. Instead of doing
f'echo {{{{ {op.output} }}}}' they can simply do f'echo {op.output}'.
[ldap] section in airflow.cfg is not applicable anymore in 2.0 and master,
because the LDAP authentication (for webserver and API) is handled by FAB,
and the configuration for this is handled by webserver_config.py file.
The init_on_load method used deserialize_value method which
in case of custom XCom backends may perform requests to external
services (for example downloading file from buckets).
This is problematic because wherever we query XCom the resuest would be
send (for example when listing XCom in webui). This PR proposes implementing
orm_deserialize_value which allows overriding this behavior. By default
we use BaseXCom.deserialize_value.
closes: #12315
This commit adds provide_file_and_upload context manager
which works similar to provide_file. Users using it can
avoid boilerplate code of creating temporary file and then
uploading its content to GCS.
Loading plugins, particularly from setuptools entry points can be slow,
and since by default this happens per-task, it can slow down task
execution unexpectedly.
By having this log message users can know the source of the delay
(The change to test_standard_task_runner was to remove logging-config
side effects from that test)
Some CLI commands simply print messages when the requests fail.
The issue is the exit code for these commands are 0 while it should be non-zero.
Pursuing very detailed status code may not make sense here.
But we can at least ensure we give non-zero status by using raise SystemExit().
More proper exist status ensures people can better make use of the CLI.
(A few minor string expression issues are fixed here as well).
Crucial feature of functions decorated by @task is to be able
to invoke them multiple times in single DAG. To do this we are
generating custom task_id for each invocation. However, this didn't
work with TaskGroup as the task_id is already altered by adding group_id
prefix. This PR fixes it.
closes: #12309
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Fix full_pod_spec for k8spodoperator
Fixes a bug where the `full_pod_spec` argument is never factored
into the kubernetespodoperator. The new order of operations is as
follows:
1. Check to see if there is a pod_template_file and if so create the initial pod, else start with empty pod
2. if there is a full_pod_spec , reconcile the pod_template_file pod and the full_pod_spec pod
3. reconcile with any of the argument overrides
* add tests
Becase `2c6edca13270` (Resource based permissions) & `849da589634d` (Prefix DAG permissions)
were run before `92c57b58940d_add_fab_tables.py` and `03afc6b6f902_increase_length_of_fab_ab_view_menu_.py`,
the FAB tables were already created because those migrations imported `from airflow.www.app import create_app`
which calls the following lines that creates tables:
0e7f62418b/flask_appbuilder/security/sqla/manager.py (L86-L97)
Previously:
```
INFO [alembic.runtime.migration] Running upgrade bef4f3d11e8b -> 98271e7606e2, Add scheduling_decision to DagRun and DAG
INFO [alembic.runtime.migration] Running upgrade 98271e7606e2 -> 52d53670a240, fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL
INFO [alembic.runtime.migration] Running upgrade 52d53670a240 -> 849da589634d, Prefix DAG permissions.
[2020-11-14 02:35:43,055] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-11-14 02:35:46,790] {migration.py:515} INFO - Running upgrade 849da589634d -> 364159666cbd, Add creating_job_id to DagRun table
[2020-11-14 02:35:46,794] {migration.py:515} INFO - Running upgrade 364159666cbd -> 2c6edca13270, Resource based permissions.
[2020-11-14 02:35:46,795] {app.py:87} INFO - User session lifetime is set to 43200 minutes.
[2020-11-14 02:35:46,806] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-11-14 02:35:48,221] {migration.py:515} INFO - Running upgrade 2c6edca13270 -> 45ba3f1493b9, add-k8s-yaml-to-rendered-templates
[2020-11-14 02:35:48,226] {migration.py:515} INFO - Running upgrade 45ba3f1493b9 -> 92c57b58940d, Create FAB Tables
[2020-11-14 02:35:48,227] {migration.py:515} INFO - Running upgrade 92c57b58940d -> 03afc6b6f902, Increase length of FAB ab_view_menu.name column
```
Now:
```
INFO [alembic.runtime.migration] Running upgrade bef4f3d11e8b -> 98271e7606e2, Add scheduling_decision to DagRun and DAG
INFO [alembic.runtime.migration] Running upgrade 98271e7606e2 -> 52d53670a240, fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL
INFO [alembic.runtime.migration] Running upgrade 52d53670a240 -> 364159666cbd, Add creating_job_id to DagRun table
INFO [alembic.runtime.migration] Running upgrade 364159666cbd -> 45ba3f1493b9, add-k8s-yaml-to-rendered-templates
INFO [alembic.runtime.migration] Running upgrade 45ba3f1493b9 -> 92c57b58940d, Create FAB Tables
INFO [alembic.runtime.migration] Running upgrade 92c57b58940d -> 03afc6b6f902, Increase length of FAB ab_view_menu.name column
INFO [alembic.runtime.migration] Running upgrade 03afc6b6f902 -> 849da589634d, Prefix DAG permissions.
[2020-11-14 02:57:18,886] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-11-14 02:57:22,380] {migration.py:515} INFO - Running upgrade 849da589634d -> 2c6edca13270, Resource based permissions.
```
- Use a context manager to encapsulate task logging setup and teardown
- Create a copy, not a reference, of the handlers list
- Remove logging.shutdown(), it simply should not be called
Closes#12090
* K8s yaml templates not rendered by k8sexecutor
There is a bug in the yaml template rendering caused by the logic that
yaml templates are only generated when the current executor is the
k8sexecutor. This is a problem as the templates are generated by the
task pod, which is itself running a LocalExecutor. Also generates a
"base" template if this taskInstance has not run yet.
* fix tests
* fix taskinstance test
* fix taskinstance
* fix pod generator tests
* fix podgen
* Update tests/kubernetes/test_pod_generator.py
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* @ashb comment
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Rather than import the backend Task model directly, use the class that the backend actually uses. This could have been customised, and there is no reason not to use this reference.
This commit adds new concept of dag_policy which is checked
once for every DAG when creating DagBag. It also improves
documentation around cluster policies.
closes: #12179
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
If you have configured S3 logs, but there is a problem then this is never
surfaced to the UI (nor the webserver logs) making this very hard to
debug.
This PR exposes some of these errors to the user.
Co-authored-by: Joao Ponte <jpe@plista.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
- adding install Databricks API to databricks hook(api/2.0/libraries/install)
- adding uninstall Databricks API to databricks hook (2.0/libraries/uninstall)
* Fixes an issue that was causing an empty list being sent to the BigQuery client `list_rows` method resulting in no schema being returned.
* Added a test to check that providing an empty list for `selected_fields` results in `list_rows` being called wth `None`.
* Add wait_for_completion option to dag run operator.
* Add wait_for_completion option to dag run operator.
* Change code format to pass sanity check.
* Simplify the logic to check dag run state.
* Move sleep in the beginning of loop and update pydoc.
* Change elif to if on checking allowed_states
Co-authored-by: Kaz Ukigai <kukigai@apple.com>
Kaxil Naik
Daniel Imberman
Ryan Hamilton
Tomek Urbaszek
Jarek Potiuk
Kamil Breguła
Tobiasz Kędzierski
Xiaodong DENG
James Timmins
John Bampton
Ash Berlin-Taylor
Ace Haidrey
Martijn Pieters
Ephraim Anierobi
Nicolas Lecoy
João Ponte
Trinity Xia
Nathan Hadfield
kukigai
Peter Kosztolanyi