0be7654fd2 commit made an optimization where the plugin are lazy-loaded. However, there are use-cases where you would still want the plugins to be loaded on Airflow start-up.
Although these lists are short, there's no need to re-create them each
time, and also no need for them to be a method.
I have made them lowercase (`finished`, `running`) instead of uppercase
(`FINISHED`, `RUNNING`) to distinguish them from the actual states.
- The dag_run argument is only there for test mocks, and only to access a static method. Removing this simplifies the function, reduces confusion.
- Give optional arguments a default value, reduce indentation of arg list to PEP / Black standard.
- Clean up tests for readability
If the `kubernetes.client` import fails, then `airflow.kubernetes.pod_generator` also can't be imported, and there won't be attributes on `k8s` to use in `isinstance()` calls.
Instead of setting `k8s` to `None`, use an explicit flag so later code can disable kubernetes-specific branches explicitly.
Also, when de-serializing a Kubernetes pod with no kubernetes library installed is an error.
The scripts were using docker compose, but they
can be docker run commands. Also they are not needed to be
run by breeze directly in CI image because I've added traps
to run the commands at the exit of all "in_container" scripts.
This adds the prefix DAG: to newly created dag permissions. It supports checking permissions on both prefixed and un-prefixed DAG permission names.
This will make it easier to identify permissions that related to granular dag access.
This PR does not modify existing dag permission names to use the new prefixed naming scheme. That will come in a separate PR.
Related to issue #10469
The production image had the capability of installing images from
wheels (for security teams/air-gaped systems). This capability
might also be useful when building CI image espeically when
we are installing separately core and providers packages and
we do not yet have provider packages available in PyPI.
This is an intermediate step to implement #11490
There was a method on the DAG class called `sub_dag()` that had nothing
to do with Sub-DAGs or the SubDagOperator - It instead created a new
"partial" dag that contained only a selected subset of tasks.
To remove this confusion when seeing `dag.sub_dag()` used in code I have
renamed this function, and included a compat shim in case anyone is
using it outside of Airflow core.
Seems that port forwarding during kubernetes tests started to behave
erratically - seems that kubectl port forward sometimes might hang
indefinitely rather than connect or fail.
We change the strategy a bit to try to allocate
increasing port numbers in case something like that happens.
In trialing an upgrade of sphinx-autoapi to 1.5.1 (it has other
problem with metaclasses right now, so we cant use it just yet) I
noticed these cases of warnings that would become errors if we upgraded.
* Add reset_dag_run option on dagrun_operator so that user can clear target dag run if exists.
* Logging coding style changes.
* Make pylint check pass.
* Make pylint check pass.
* Make pylint check pass on unit test file.
* Make static check pass.
* Use settings.STORE_SERIALIZED_DAGS
Co-authored-by: Kaz Ukigai <kukigai@apple.com>
Seems that by splitting the tests into many small jobs has a bad
effect - since we only have queue size = 180 for the whole Apache
organisation, we are competing with other projects for the jobs
and with the jobs being so short we got starved much more than if
we had long jobs. Therefore we are re-combining the test types into
single jobs per Python version/Database version and run all the
tests sequentially on those machines.
Created a new Airflow tutorial to use Decorated Flows (a.k.a. functional
DAGs). Also created a DAG to perform the same operations without using
functional DAGs to be compatible with Airflow 1.10.x and to show the
difference.
* Apply suggestions from code review
It makes sense to simplify the return variables being passed around without needlessly converting to JSON and then reconverting back.
* Update tutorial_functional_etl_dag.py
Fixed data passing between tasks to be more natural without converting to JSON and converting back to variables.
* Updated dag options and task doc formating
Based on feedback on the PR, updated the DAG options (including schedule) and the fixed the task documentation to avoid indentation.
* Added documentation file for functional dag tutorial
Added the tutorial documentation to the docs directory. Fixed linting errors in the example dags.
Tweaked some doc references in the example dags for inclusion into the tutorial documentation.
Added the example dags to example tests.
* Removed multiple_outputs from task defn
Had a multiple_outputs=True defined in the Extract task defn, which was unnecessary. - Removed based on feedback.
Co-authored-by: Gerard Casas Saez <casassg@users.noreply.github.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Separate changes/readmes for backport and regular providers
We have now separate release notes for backport provider
packages and regular provider packages.
They have different versioning - backport provider
packages with CALVER, regular provider packages with
semver.
* Added support for provider packages for Airflow 2.0
This change consists of the following changes:
* adds provider package support for 2.0
* adds generation of package readme and change notes
* versions are for now hard-coded to 0.0.1 for first release
* adds automated tests for installation of the packages
* rename backport package readmes/changes to BACKPORT_*
* adds regulaar packge readmes/changes
* updates documentation on generating the provider packaes
* adds CI tests for the packages
* maintains backport packages generation with --backports flag
Fixes#11421Fixes#11424
Gerard Casas Saez
Daniel Imberman
Ryan Hamilton
Kaxil Naik
Ash Berlin-Taylor
Sumit Maheshwari
Martijn Pieters
Michał Misiewicz
Katsunori Kanda
Kamil Breguła
Martijn Pieters
Jarek Potiuk
Satyasheel
Masato Ohba
Michal Niemiec
Felix Uellendall
James Timmins
Kaashif Khawaja
Songkran Nethan
kukigai
Vikram Koka