The inbuilt functions `any()` support short-circuiting (evaluation stops as soon as the overall return value of the function is known), but this behavior is lost if you use comprehension. This affects performance.
'tornado' version was pinned in https://github.com/apache/airflow/pull/4815
The underlying issue is fixed for Py 3.5.2 so that is not a problem. Also since Airflow Master is already Py 3.6+ this does not apply to us.
This can happen when a task is enqueued by one executor, and then that
scheduler dies/exits.
The default fallback behaviour is unchanged -- that queued tasks are
cleared and then and then later rescheduled.
But for Celery, we can do better -- if we record the Celery-generated
task_id, we can then re-create the AsyncResult objects for orphaned
tasks at a later date.
However, since Celery just reports all AsyncResult as "PENDING", even if
they aren't tasks currently in the broker queue, we need to apply a
timeout to "unblock" these tasks in case they never actually made it to
the Celery broker.
This all means that we can adopt tasks that have been enqueued another
CeleryExecutor if it dies, without having to clear the task and slow
down. This is especially useful as the task may have already started
running, and while clearing it would stop it, it's better if we don't
have to reset it!
Co-authored-by: Kaxil Naik <kaxilnaik@apache.org>
Snakebite's kerberos support relied on a python-krbV
which has been removed from PyPI. It did not work
completely anyway due to snakebite not being officially
supported in python3 (snakebite-py3 did not work with
SSL which made Kerberos pretty much unusable.
This commit removes the snakebite's kerberos support
from setup.py so that you still can install kerberos
as extra for other uses.
Recent releases of FAB and Celery caused our installation to
fail. Luckily we have protection so that regular PRs are not
affected, however we need to update the setup.py to exclude
those dependencies that cause the problem.
Those are:
* vine - which is used by Celery Sensor (via kombu) - 5.0.0
version breaks celery-vine feature
* Flask-OauthLib and flask-login - combination of the current
requirements caused a conflict by forcing flask login to
be 0.5.0 which is not compatible with Flask Application Builder
The current latest available version is 0.5.0 (https://pypi.org/project/Flask-Login/0.5.0/)
Version 0.5.0 drops support for Python 2.6, 3.3, 3.4 and we don't use those versions in Airflow Master
We have already fixed a lot of problems that were marked
with those, also IntelluiJ gotten a bit smarter on not
detecting false positives as well as understand more
pylint annotation. Wherever the problem remained
we replaced it with # noqa comments - as it is
also well understood by IntelliJ.
mysqlclient was pinned as the functionality of Connection object of MySQL had changed and that it dropped python2.
mysqlclient has added context manager interface to Connection which closes the connection on `__exit__` in 2.0.0 and Airflow Master no longer uses Python 2
WTForms uses `MarkUp` to escape strings now and removed their internal class HTMLString in Master. Details: https://github.com/wtforms/wtforms/pull/400
That change previously broke Airflow for new users (in 2.3.0). However on users request they added `HTMLString` that just passes all args to `markupsafe.MarkUp` back for temporary Backward compatbility with deprecation warning in 2.3.1. Details: https://github.com/wtforms/wtforms/issues/581
We pulled in them because slugify _used_ to default to the GPL'd
`unidecode` module, but since Slugify 3.0[1] it has used text-unidecode
by first (and only installs the GPL library by an optional extra, not by
default) so we can now use it.
This lets us upgreade text-unidecode from 1.2 to the latest 1.3, which
is the version one of dbt's dependencies needs.
[1]: https://github.com/un33k/python-slugify/blob/4.0.0/CHANGELOG.md#300
Version 2 of this library causes the following pylint errors:
> airflow/providers/google/cloud/hooks/text_to_speech.py:59:27: E1123: Unexpected keyword argument 'client_info' in constructor call (unexpected-keyword-arg)
> airflow/providers/google/cloud/hooks/text_to_speech.py:99:15: E1123: Unexpected keyword argument 'input_' in method call (unexpected-keyword-arg)
Michał Słowikowski
Kaxil Naik
Jarek Potiuk
Ash Berlin-Taylor
Miller Tracy
Kamil Breguła
Felix Uellendall
David Cavaletto
Tomek Urbaszek
Ephraim Anierobi
yongheng.liu
QP Hou
Hartorn
crhyatt
James Timmins