Remove Heisentest category and quarantine test_backfill_depends_on_past (#14756)

The whole Backfill class was in Heisentest but only one of those tests
is problematic nowi: test_backfill_depends_on_past. Therfore it makes
sense to remove the class from heisentests and move the
depends_on_past to quarantine.

It turned out that this is the last "Heisentest" and with the
isolation we have now coming in parallel tests, it turns out that
Heisentests are not really good way thinking about the tests - running
them in isolation does not often help, it only makes it more difficult
to flag the tests as flaky.

The quarantine test_backfill_depends_on_past ihas been captured in
the #14755 issue - and hopefully we will make an effort to
de-quarantine some of those tests soon.

BREEZE.rst

@@ -2342,7 +2342,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --test-type TEST_TYPE
           Type of the test to run. One of:
 
-                 All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL,Helm
+                 All,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm
 
           Default: All
 
@@ -2756,7 +2756,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --test-type TEST_TYPE
           Type of the test to run. One of:
 
-                 All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL,Helm
+                 All,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm
 
           Default: All
 

CONTRIBUTORS_QUICK_START.rst

@@ -635,13 +635,6 @@ All Tests are inside ./tests directory.
 
   - Types of tests
 
-  .. code-block:: bash
-
-   $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type
-      All          CLI          Heisentests  Integration  Other        Providers
-      API          Core         Helm         MySQL        Postgres     WWW
-
-
   - Running specific type of test
 
   .. code-block:: bash
@@ -651,16 +644,6 @@ All Tests are inside ./tests directory.
 
 - Running Integration test for specific test type
 
-
-  - Types of Integration Tests
-
-  .. code-block:: bash
-
-     $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core --integration
-
-       all        kerberos   openldap   presto     redis
-       cassandra  mongo      pinot      rabbitmq
-
   - Running an Integration Test
 
   .. code-block:: bash

PULL_REQUEST_WORKFLOW.rst

@@ -112,7 +112,6 @@ pytest markers. They can be found in any of those packages and they can be selec
 pylint custom command line options. See `TESTING.rst <TESTING.rst>`_ for details but those are:
 
 * Integration - tests that require external integration images running in docker-compose
-* Heisentests - tests that are vulnerable to some side effects and are better to be run on their own
 * Quarantined - tests that are flaky and need to be fixed
 * Postgres - tests that require Postgres database. They are only run when backend is Postgres
 * MySQL - tests that require MySQL database. They are only run when backend is MySQL
@@ -165,7 +164,7 @@ The logic implemented for the changes works as follows:
       all changed files. In case there are any files changed, then we assume that some unknown files
       changed (likely from the core of airflow) and in this case we enable all test types above and the
       Core test types - simply because we do not want to risk to miss anything.
-   h) In all cases where tests are enabled we also add Heisentests, Integration and - depending on
+   h) In all cases where tests are enabled we also add Integration and - depending on
       the backend used = Postgres or MySQL types of tests.
 
 10) Quarantined tests are always run when tests are run - we need to run them often to observe how

TESTING.rst

@@ -177,7 +177,7 @@ kinds of test types:
 
        ./breeze --test-type Providers --db-reset tests
 
-* Special kinds of tests - Integration, Heisentests, Quarantined, Postgres, MySQL, which are marked with pytest
+* Special kinds of tests - Integration, Quarantined, Postgres, MySQL, which are marked with pytest
   marks and for those you need to select the type using test-type switch. If you want to run such tests
   using breeze, you need to pass appropriate ``--test-type`` otherwise the test will be skipped.
   Similarly to the per-directory tests if you do not specify the test or tests to run,
@@ -418,17 +418,6 @@ Those tests are marked with ``@pytest.mark.quarantined`` annotation.
 Those tests are skipped by default. You can enable them with ``--include-quarantined`` flag. You
 can also decide to only run tests with ``-m quarantined`` flag to run only those tests.
 
-Heisen tests
-------------
-
-Some of our tests are Heisentests. This means that they run fine in isolation but when they run together with
-others they might fail the tests (this is likely due to resource consumptions). Therefore we run those tests
-in isolation.
-
-Those tests are marked with ``@pytest.mark.heisentests`` annotation.
-Those tests are skipped by default. You can enable them with ``--include-heisentests`` flag. You
-can also decide to only run tests with ``-m heisentests`` flag to run only those tests.
-
 Running Tests with provider packages
 ====================================
 

breeze-complete

@@ -36,7 +36,7 @@ _breeze_allowed_kind_versions="v0.8.0"
 _breeze_allowed_mysql_versions="5.7 8"
 _breeze_allowed_postgres_versions="9.6 10 11 12 13"
 _breeze_allowed_kind_operations="start stop restart status deploy test shell k9s"
-_breeze_allowed_test_types="All Core Providers API CLI Integration Other WWW Heisentests Postgres MySQL Helm"
+_breeze_allowed_test_types="All Core Providers API CLI Integration Other WWW Postgres MySQL Helm"
 _breeze_allowed_package_formats="both sdist wheel"
 _breeze_allowed_installation_methods=". apache-airflow"
 

scripts/ci/selective_ci_checks.sh

@@ -199,7 +199,7 @@ function set_upgrade_to_newer_dependencies() {
 }
 
 
-ALL_TESTS="Always Core Other API CLI Providers WWW Integration Heisentests"
+ALL_TESTS="Always Core Other API CLI Providers WWW Integration"
 readonly ALL_TESTS
 
 function set_outputs_run_everything_and_exit() {
@@ -600,7 +600,7 @@ function calculate_test_types_to_run() {
             echo
             SELECTED_TESTS="${SELECTED_TESTS} WWW"
         fi
-        initialization::ga_output test-types "Always Integration Heisentests ${SELECTED_TESTS}"
+        initialization::ga_output test-types "Always Integration ${SELECTED_TESTS}"
     fi
     start_end::group_end
 }

scripts/ci/testing/ci_run_airflow_testing.sh

@@ -115,7 +115,7 @@ function prepare_tests_to_run() {
     fi
 
     if [[ -z "${TEST_TYPES=}" ]]; then
-        TEST_TYPES="Core Providers API CLI Integration Other WWW Heisentests"
+        TEST_TYPES="Core Providers API CLI Integration Other WWW"
         echo
         echo "Test types not specified. Running all: ${TEST_TYPES}"
         echo

scripts/in_container/entrypoint_ci.sh

@@ -321,7 +321,7 @@ else
     elif [[ ${TEST_TYPE:=""} == "All" || ${TEST_TYPE} == "Quarantined" || \
             ${TEST_TYPE} == "Always" || \
             ${TEST_TYPE} == "Postgres" || ${TEST_TYPE} == "MySQL" || \
-            ${TEST_TYPE} == "Heisentests" || ${TEST_TYPE} == "Long" || \
+            ${TEST_TYPE} == "Long" || \
             ${TEST_TYPE} == "Integration" ]]; then
         SELECTED_TESTS=("${ALL_TESTS[@]}")
     else
@@ -346,11 +346,6 @@ elif [[ ${TEST_TYPE:=""} == "Long" ]]; then
         "-m" "long_running"
         "--include-long-running"
     )
-elif [[ ${TEST_TYPE:=""} == "Heisentests" ]]; then
-    EXTRA_PYTEST_ARGS+=(
-        "-m" "heisentests"
-        "--include-heisentests"
-    )
 elif [[ ${TEST_TYPE:=""} == "Postgres" ]]; then
     EXTRA_PYTEST_ARGS+=(
         "--backend"

tests/conftest.py

@@ -152,11 +152,6 @@ def pytest_addoption(parser):
         action="store_true",
         help="Includes quarantined tests (marked with quarantined marker). They are skipped by default.",
     )
-    group.addoption(
-        "--include-heisentests",
-        action="store_true",
-        help="Includes heisentests (marked with heisentests marker). They are skipped by default.",
-    )
     allowed_trace_sql_columns_list = ",".join(ALLOWED_TRACE_SQL_COLUMNS)
     group.addoption(
         "--trace-sql",
@@ -242,9 +237,6 @@ def pytest_configure(config):
     config.addinivalue_line(
         "markers", "quarantined: mark test that are in quarantine (i.e. flaky, need to be isolated and fixed)"
     )
-    config.addinivalue_line(
-        "markers", "heisentests: mark test that should be run in isolation due to resource consumption"
-    )
     config.addinivalue_line(
         "markers", "credential_file(name): mark tests that require credential file in CREDENTIALS_DIR"
     )
@@ -314,14 +306,6 @@ def skip_quarantined_test(item):
         )
 
 
-def skip_heisen_test(item):
-    for _ in item.iter_markers(name="heisentests"):
-        pytest.skip(
-            "The test is skipped because it has heisentests marker. "
-            "And --include-heisentests flag is passed to pytest. {item}".format(item=item)
-        )
-
-
 def skip_if_integration_disabled(marker, item):
     integration_name = marker.args[0]
     environment_variable_name = "INTEGRATION_" + integration_name.upper()
@@ -378,7 +362,6 @@ def pytest_runtest_setup(item):
 
     include_long_running = item.config.getoption("--include-long-running")
     include_quarantined = item.config.getoption("--include-quarantined")
-    include_heisentests = item.config.getoption("--include-heisentests")
 
     for marker in item.iter_markers(name="integration"):
         skip_if_integration_disabled(marker, item)
@@ -397,8 +380,6 @@ def pytest_runtest_setup(item):
         skip_long_running_test(item)
     if not include_quarantined:
         skip_quarantined_test(item)
-    if not include_heisentests:
-        skip_heisen_test(item)
     skip_if_credential_file_missing(item)
     skip_if_airflow_2_test(item)
 

tests/jobs/test_backfill_job.py

@@ -56,7 +56,6 @@ logger = logging.getLogger(__name__)
 DEFAULT_DATE = timezone.datetime(2016, 1, 1)
 
 
-@pytest.mark.heisentests
 class TestBackfillJob(unittest.TestCase):
     def _get_dummy_dag(self, dag_id, pool=Pool.DEFAULT_POOL_NAME, task_concurrency=None):
         dag = DAG(dag_id=dag_id, start_date=DEFAULT_DATE, schedule_interval='@daily')
@@ -809,6 +808,7 @@ class TestBackfillJob(unittest.TestCase):
         ti.refresh_from_db()
         assert ti.state == State.SUCCESS
 
+    @pytest.mark.quarantined
     def test_backfill_depends_on_past(self):
         """
         Test that backfill respects ignore_depends_on_past