f69aa14a02
* [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. Rather than running a fresh python interpreter which then has to re-load all of Airflow and its dependencies we should use os.fork when it is available/suitable which should speed up task running, espeically for short lived tasks. I've profiled this and it took the task duration (as measured by the `duration` column in the TI table) from an average of 14.063s down to just 0.932s! * Allow `reap_process_group` to kill processes even when the "group leader" has already exited. * Don't re-initialize JSON/stdout logging ElasticSearch inside forked processes Most of the time we will run the "raw" task in a forked subprocess (the only time we don't is when we use impersonation) that will have the logging already configured. So if the EsTaskHandler has already been configured we don't want to "re"configure it -- otherwise it will disable JSON output for the actual task! |
||
|---|---|---|
| .. | ||
| task_runner | ||
| __init__.py | ||