This PR Standardises the callable signatures in PythonOperator, PythonSensor, ExternalTaskSensor, SimpleHttpOperator and HttpSensor.
The callable facilities in PythonOperator have been refactored into airflow.utils.helper.make_kwargs_callable. And it's used in those other places to make them work the same way.
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
In order to further reduce intra-dag task scheduling lag we add an
optimization: when a task has just finished executing (success or
failure) we can look at the downstream tasks of just that task, and then
make scheduling decisions for those tasks there -- we've already got the
dag loaded, and we know they are likely actionable as we just finished.
We should set tasks to scheduled if we can (but no further, i.e. not to
queued, as the scheduler has to make that decision with info about the
Pool usage etc.).
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
This is an improvement to the UI response time when clearing dozens of DagRuns of large DAGs (thousands of tasks) containing many ExternalTaskSensor + ExternalTaskMarker pairs. In the current implementation, clearing tasks can get slow especially if the user chooses to clear with Future, Downstream and Recursive all selected.
This PR speeds it up. There are two major improvements:
Updating self._task_group in dag.sub_dag() is improved to not deep copy _task_group because it's a waste of time. Instead, do something like dag.task_dict, set it to None first and then copy explicitly.
Pass the TaskInstance already visited down the recursive calls of dag.clear() as visited_external_tis. This speeds up the example in test_clear_overlapping_external_task_marker by almost five folds.
For real large dags containing 500 tasks set up in a similar manner, the time it takes to clear 30 DagRun is cut from around 100s to less than 10s.
We've observed the tests for last couple of weeks and it seems
most of the tests marked with "quarantine" marker are succeeding
in a stable way (https://github.com/apache/airflow/issues/10118)
The removed tests have success ratio of > 95% (20 runs without
problems) and this has been verified a week ago as well,
so it seems they are rather stable.
There are literally few that are either failing or causing
the Quarantined builds to hang. I manually reviewed the
master tests that failed for last few weeks and added the
tests that are causing the build to hang.
Seems that stability has improved - which might be casued
by some temporary problems when we marked the quarantined builds
or too "generous" way of marking test as quarantined, or
maybe improvement comes from the #10368 as the docker engine
and machines used to run the builds in GitHub experience far
less load (image builds are executed in separate builds) so
it might be that resource usage is decreased. Another reason
might be Github Actions stability improvements.
Or simply those tests are more stable when run isolation.
We might still add failing tests back as soon we see them behave
in a flaky way.
The remaining quarantined tests that need to be fixed:
* test_local_run (often hangs the build)
* test_retry_handling_job
* test_clear_multiple_external_task_marker
* test_should_force_kill_process
* test_change_state_for_tis_without_dagrun
* test_cli_webserver_background
We also move some of those tests to "heisentests" category
Those testst run fine in isolation but fail
the builds when run with all other tests:
* TestImpersonation tests
We might find that those heisentest can be fixed but for
now we are going to run them in isolation.
Also - since those quarantined tests are failing more often
the "num runs" to track for those has been decreased to 10
to keep track of 10 last runs only.
* [AIRFLOW-6586] Improvements to gcs sensor
refactors GoogleCloudStorageUploadSessionCompleteSensor to use set instead of number of objects
add poke mode only decorator
assert that poke_mode_only applied to child of BaseSensorOperator
refactor tests
remove assert
[AIRFLOW-6586] Improvements to gcs sensor
refactors GoogleCloudStorageUploadSessionCompleteSensor to use set instead of number of objects
add poke mode only decorator
assert that poke_mode_only applied to child of BaseSensorOperator
remove assert
fix static checks
add back inadvertently remove requirements
pre-commit
fix typo
* gix gcs sensor unit test
* move poke_mode_only to base_sensor_operator module
* add sensor / poke_mode_only docs
* fix ci check add sensor how-to docs
* Update airflow/providers/google/cloud/sensors/gcs.py
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
* Update airflow/sensors/base_sensor_operator.py
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
* Update airflow/sensors/base_sensor_operator.py
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* simplify class decorator
* remove type hint
* add note to UPDATING.md
* remove unecessary declaration of class member
* Fix to kwargs in UPDATING.md
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* [AIRFLOW-6721] Move WebHdfsSensor tests to own module
Move the WebHdfsSensor tests out of the operator tests and into a
separate module.
* [AIRFLOW-6721] Move TestHiveEnvironment to test module
Movee the TestHiveEnvironment class and setUp method out to the hive
test module so that it can be used in other hive tests.
* [AIRFLOW-6721] Move Hive Stats tests to separate module
Move skipped test for HiveStatsCollectionOperator to the test_hive_stats
module to make it easier to locate.
* [AIRFLOW-6721] Move NamedHivePartitionSensor tests
Move the NamedHivePartitionSensor tests from the operators module into
the test module for NamedHivePartitionSensor.
* [AIRFLOW-6721] Move HivePartitionSensor tests
Move the HivePartitionSensor tests from the operators module into
the a new test module.
* [AIRFLOW-6721] Move HiveToMySqlTransfer test
Move the HiveToMySqlTransfer test in the operators module into
the existing test module and use the shared test environment for
these tests.
* [AIRFLOW-6721] Move HdfsSensor test to separate module
Move the HdfsSensor test in the operators module into a new test
module and use the shared test environment.
* [AIRFLOW-6721] Move Hive2SambaOperator test
Move the Hive2SambaOperator test in the operators module into the
existing test module and use the shared test environment.
* [AIRFLOW-6721] Move Hive MetastorePartitionSensor test
Move the MetastorePartitionSensor test from the operators module
into a new module and use the shared test environment.
* [AIRFLOW-6721] Move Hive PrestoToMySqlTransfer test
Move the PrestoToMySqlTransfer test from the operators module
into an existing test module and use shared Hive test environment.
* [AIRFLOW-6721] Move PrestoCheckOperator test
Move the PrestoCheckOperator test from the Hive operators module
into a new Presto operators test module while still using the
shared Hive test environment.
* [AIRFLOW-6721] Move Hive SqlSensor test
Move the SqlSensor in the Hive operators module into the existing
SqlSensor test module and label it as such. Use the shared test
environment for all tests, ensuring that original setUp properties
are preserved for the existing tests.
Using the parameterized library, consolidate the true or "happy path"
tests, reducing overall code to maintain and showing each test case in
one list of tuples.
You can now choose which integration you want to start when you run
Breeze as well as when CI tests are run. Now by default Breeze
and CI runs without integrations, but you can add them via
Breeze flags or by environment variables when CI is executed.
We have pytest markers now that mark tests that can be run for
integration, backend and runtime selected.
Also we have now more test jobs - we have separate test run
for all non-integration tests (with less memory used by the
integrations) and separate jobs that run integration tests
only (more memory used for integrations but far less number
of tests to run)
yuqian90
Fai
Kaxil Naik
James Timmins
John Bampton
Denis Evseev
Yingbo Wang
Jarek Potiuk
Leon Yuan
j-y-matsubara
zikun
Kamil Breguła
Ignacio Peluffo
Tomek Urbaszek
Ace Haidrey
Jacob Ferriero
Lokesh Lal
Matt Buell
Cooper Gillan
Mustafa Gök
Tomek
marengaz
Sumit Maheshwari
Anton Zayniev
MinJae Kwon