on_failure_callback almost always want to know the dag_id and taskinstance that failed. These info are in the context passed to on_failure_callback, which is passed in from handle_failure(). However, in some rare scenarios, if handle_failure is called in scheduler_job.py and backfill_job.py, the only argument passed is the error message. context is left as None.
So in these cases, on_failure_callback will not even know what's the dag_id of the dag that just failed.
This PR fixes this by setting context to get_template_context() if it's not given.
Airflow should handle various charsets for connections with MySQL dbs.
This change allows to set charset in extra field of a connection when
using SqlAlchemy engine.
Why:
* In our system we had a postgres connection error during Variable.set
resulting in the variable being deleted. The intention of this change is
that an error should leave the variable unchanged.
Support getting variables in templates by string. This is necessary when
fetching variables with characters not allowed in a class attribute
name. We can then also support returning default values when a variable does
not exist.
Add dag_ids get parameter to last_dagruns endpoint so can filter by the
set of dag_ids present on the dags view. This is intended to speed up
the response time on systems running a large number of dags.
Add dag_ids parameter to the dag_stats end point and only request the
dags on the current page. This is intended to speed up the response
times for systems running a large number of DAGS.
Adds new executor that is meant to be used mainly
for debugging and DAG development purposes. This
executor executes single task instance at time and
is able to work with SQLLite and sensors.
Try to find the task in DB before regressing to searching every task,
and explicitly warn about the performance regressions.
Co-Authored-By: Ash Berlin-Taylor <ash_github@firemirror.com>
- errors in polling for job status should not fail
the airflow task when the polling hits an API throttle
limit; polling should detect those cases and retry a
few times to get the job status, only failing the task
when the job description cannot be retrieved
- added typing for the BatchProtocol method return
types, based on the botocore.client.Batch types
- applied trivial format consistency using black, i.e.
$ black -t py36 -l 96 {files}
This reverts commit da7a353e17.
This was already fixed by by #6754 which was already merged, and we ended up with
two migration files (trying) to do the same thing.
With this change you should be able to simply run `pytest` to run all the tests in the main airflow directory.
This consist of two changes:
* moving pytest.ini to the main airflow directory
* skipping collecting kubernetes tests when ENV != kubernetes
There is a bug caused by scheduler_jobs refactor which leads to task failure
and scheduler locking.
Essentially when a there is an overflow of tasks going into the scheduler, the
tasks are set back to scheduled, but are not removed from the executor's
queued_tasks queue.
This means that the executor will attempt to run tasks that are in the scheduled
state, but those tasks will fail dependency checks. Eventually the queue is
filled with scheduled tasks, and the scheduler can no longer run.
Co-Authored-By: Kaxil Naik <kaxilnaik@gmail.com>, Kevin Yang <kevin.yang@airbnb.com>
Tomek
TobKed
Nikolay
yuqian90
dependabot[bot]
Jarek Potiuk
dstandish
zuku1985
Kaxil Naik
Yingbo Wang
Bjorn Olsen
Robert Young
Daniel Huang
Rich Dean
mislo
Robin Edwards
Daniel Gorelik
pradeepbhadani
Andrey Klochkov
Joseph McCartin
Aditya Vishwakarma
Darren Weber
kaverisharma09
Alexandre Yang
Ash Berlin-Taylor
Avetisyan Sevak
Ryan Yuan
Kamil Breguła
Sergio Kef
benjamingrenier
Fokko Driesprong
Tim Paine
Aviem Zur
Kim Persson
Daniel Imberman