* Query TaskReschedule only if task is UP_FOR_RESCHEDULE
* Query for single TaskReschedule when possible
* Apply suggestions from code review
Co-authored-by: Stefan Seelmann <mail@stefan-seelmann.de>
* Adjust mocking in tests
* fixup! Adjust mocking in tests
* fixup! fixup! Adjust mocking in tests
Co-authored-by: Stefan Seelmann <mail@stefan-seelmann.de>
The documentation for the `none_failed` trigger rule
(https://airflow.apache.org/docs/stable/concepts.html#trigger-rules)
describes its behavior as "all parents have not failed (`failed` or
`upstream_failed`) i.e. all parents have succeeded or been skipped."
With that definition in mind, there is no reason that the check for
`none_failed` should ever have to check for skipped upstream tasks or
set the current task to `skipped`. The current behavior causes the rule
to skip if all upstream tasks have skipped, which is more than a little
confusing. This fixes the behavior to be consistent with the documentation.
Co-authored-by: root <jcroteau@liveramp.com>
If a task is skipped by BranchPythonOperator, BaseBranchOperator or ShortCircuitOperator and the user then clears the skipped task later, it'll execute. This is probably not the right
behaviour.
This commit changes that so it will be skipped again. This can be ignored by running the task again with "Ignore Task Deps" override.
This decreases scheduler delay between tasks by about 20% for larger DAGs,
sometimes more for larger or more complex DAGs.
The delay between tasks can be a major issue, especially when we have dags with
many subdags, figures out that the scheduling process spends plenty of time in
dependency checking, we took the trigger rule dependency which calls the db for
each task instance, we made it call the db just once for each dag_run
* adding pool capacity required for each task for dynamic pooling
* Added pool_capacity column migration script
* removed test checkedin file
* removed extra space
* correct test_database_schema_and_sqlalchemy_model_are_in_sync test case
* Added description for pool_capacity property for task instance
* Modified test cases to include pool_capacity along with pool in task instances
* Modified test cases to include pool_capacity along with pool in task instances
* Removed Column.name property, since property value is same as actual variable
* check for pool_capacity property to be always >= 1
* removed unused variable ti
* modified test cases for pool_capacity
* modified test cases for pool_capacity
* [AIRFLOW-3589] Visualize reschedule state in all views
* Add explicit `UP_FOR_RESCHEDULE` state
* Add legend and CSS to views
* [AIRFLOW-3589] Visualize reschedule state in all views
* Use set or tuple instad of list
* Use `with` statement for session handling
* [AIRFLOW-2747] Explicit re-schedule of sensors
Add `mode` property to sensors. If set to `reschedule` an
AirflowRescheduleException is raised instead of sleeping which sets
the task back to state `NONE`. Reschedules are recorded in new
`task_schedule` table and visualized in the Gantt view. New TI
dependency checks if a sensor task is ready to be re-scheduled.
* Reformat sqlalchemy imports
* Make `_handle_reschedule` private
* Remove print
* Add comment
* Add comment
* Don't record reschule request in test mode
This adds a feature to limit the concurrency of
individual tasks. The
default will be to not change existing behavior.
Closes#2624 from saguziel/aguziel-task-
concurrency
Patrick Cando
David Cavaletto
Kamil Breguła
Kaxil Naik
Jarek Potiuk
Gerard Casas Saez
crhyatt
Tomek Urbaszek
Ben Nadler
j-y-matsubara
yuqian90
Tomek Urbaszek
tooptoop4
Lokesh Lal
amichai07
Bjorn Olsen
Tomek Urbaszek
Alex Guziel
Kevin Yang
Bas Harenslak
Chao-Han Tsai
al
Stefan Seelmann
zhongjiajie
rmn36
Tao feng
Bolke de Bruin
William Pursell