If you try to run `airflow config list` with an old config you upgraded
from 1.8, it would fail for any sections that have been removed from the
default cofing -- `ldap` for instance.
This would also be a problem if the user makes a typo in a config
section, or is using the airflow config for storing their own
information.
While I was changing this code, I also removed the use of private
methods/variable access in favour of public API
* Cleanup & improvement around scheduling
- Remove unneeded code line
- Remove stale docstring
- Fix wrong docstring
- Fix stale doc image link in docstring
- avoid unnecessary loop in DagRun.schedule_tis()
- Minor improvement on DAG.deactivate_stale_dags()
which is invoked inside SchedulerJob
* Revert one change, because we plan to have a dedicated project-wise PR for this issue
* One more fix: dagbag.read_dags_from_db = True in DagFileProcess.process_file() is not needed anymore
* Dagrun object doesn't exist in the TriggerDagRunOperator
fixes https://github.com/apache/airflow/issues/12587
Fixes issue where dag_run object is not populated if the dag_run already
exists and is reset
* change to get_last_dag_run
* Update airflow/operators/dagrun_operator.py
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
This PR is a followup after #12375 and #12704 it improves handling
of some errors in cli commands to avoid show users to much traceback
and uses SystemExit consistently.
`[webserver] secret_key` is also a secret like Fernet key. Allowing
it to be set via _CMD or _SECRET allows users to use the external secret store for it.
Custom operators inheriting from DummyOperator will now instead
of going to a scheduled state will go set straight to success
if they don't have callbacks set.
closes https://github.com/apache/airflow/issues/11393
This commit unifies the mechanism of rendering output of tabular
data. This gives users a possibility to eiter display a tabular
representation of data or render it as valid json or yaml payload.
Closes: #12699
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Move config item 'worker_precheck' from section [core] to [celery]
This configuration is ONLY applicable for Celery Worker.
So it should be in section [celery], rather than [core]
* Add to deprecation/migration automatic list
The chart has two jobs (migrate-database & create-user).
These jobs are run post-install and post-upgrade and only deleted on success.
So if for some reason (quick reinstall / upgrade), the job fails or is stuck then helm
will fail because the job already exists.
This commit sets the `helm.sh/hook-delete-policy` to `before-hook-creation,hook-succeeded`
so helm will always delete the jobs before creating them again.
Without this commit, the Webserver throws an error when
enabling xcom_pickling in the airflow_config by setting `enable_xcom_pickling = True`
(the default is `False`).
Example error:
```
> return pickle.loads(result.value)
E _pickle.UnpicklingError: invalid load key, '{'.
airflow/models/xcom.py:250: UnpicklingError
--------------------------------------------------
```
Previously the output of instaling remaining packges when testing
provider imports was only shown on error. However it is useful
to know what's going on even if it clutters the log.
Note that this installation is only needed until we include
apache-beam in the installed packages on CI.
Related to #12703
This PR shows the output always .
* Refine the DB query logics in www.views.task_stats()
- given filter_dag_ids is either allowed_dag_ids, or intersection of allowed_dag_ids and selected_dag_ids,
no matter if selected_dag_ids is None or not, filter_dag_ids should ALWAYS be considered into the SQL query.
Currently, if selected_dag_ids is None, the query is actually getting the full result (then 'filter' at the end).
This means more (unnecessary) data travel between Airflow and DB.
- When we join table A and B with A.id == B.id (default is INNER join), if we always confirm ALL A.id is in a specific list,
implicitly ALL ids in the result table are already guaranteed in this specific list as well.
This is why the two redundant .filter() chunks are removed.
Minor performance improvement should be expected.
Meanwhile, this change makes the code cleaner.
* Adds support for Hook discovery from providers
This PR extends providers discovery with the mechanism
of retrieving mapping of connections from type to hook.
Fixes#12456
* fixup! Adds support for Hook discovery from providers
* fixup! fixup! Adds support for Hook discovery from providers
Ash Berlin-Taylor
Xiaodong DENG
Sam Wheating
Daniel Imberman
zclimes
Kamil Breguła
Siddartha Ravichandran
Tomek Urbaszek
James Timmins
Ryan Hamilton
宋财礼
Kaxil Naik
Ephraim Anierobi
Darwin Yip
Danny
Shivansh Saini
Florent Chehab
Jarek Potiuk
Ry Walker
Mutlu Polatcan
HasanJ
raphaelauv