The current LatestOnlyOperator will skip all downstream tasks blindly.
The doc shows a wrong behavior. It also shows an incorrect example about
the interaction between skipped tasks and trigger rules. I replace it with
another example using BranchingOperator in schedule level.
This fix can resolve this ticket:
https://issues.apache.org/jira/browse/AIRFLOW-3977
Also,
https://issues.apache.org/jira/browse/AIRFLOW-1784
This adds ASF license headers to all the .rst and .md files with the
exception of the Pull Request template (as that is included verbatim
when opening a Pull Request on Github which would be messy)
When Airflow was populating a DagBag from a .zip
file, if a single
file in the root directory did not contain the
strings 'airflow' and
'DAG' it would ignore the entire .zip file.
Also added a small amount of logging to not
bombard user with info
about skipping their .py files.
Closes#3505 from Noremac201/dag_name
I'd like to have how-to guides for all connection
types, or at least the
different categories of connection types. I found
it difficult to figure
out how to manage a GCP connection, this commit
add a how-to guide for
this.
Also, since creating and editing connections
really aren't all that
different, the PR renames the "creating
connections" how-to to "managing
connections".
Closes#3419 from tswast/howto
Searching through all the documentation I couldn't
find anywhere
that explained what file format it expected for
uploading settings.
Closes#2802 from bovard/variable_files_are_json
I parsed it with the ol' eyeball compiler. Someone
could flake8 it better, perhaps.
Changes:
- correct `def` syntax on line 50
- use literal dict on line 67
Closes#2479 from 0atman/patch-1
# JIRA
[x] My PR addresses the following Airflow JIRA
issues and references them in the PR title.
*
https://issues.apache.org/jira/browse/AIRFLOW-2312
# Description
[x] Here are some details about my PR
Minor typo fix in the docs. `correpsonding` to
`corresponding`
# Tests
[x] My PR adds the following unit tests OR does
not need testing for this extremely good reason:
I assume I don't need testing for docs
# Commits
[x] My commits all reference JIRA issues in their
subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
Closes#3211 from tamsanh/patch-1
Remove SQLOperator from Concepts page. Update with
a sample list of database backend-specific
operators, lile, MySqlOperator, SqliteOperator,
PostgresOperator, MsSqlOperator, OracleOperator,
JdbcOperator
Closes#2168 from Tagar/AIRFLOW-1009
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
- [AIRFLOW-96](https://issues.apache.org/jira/brow
se/AIRFLOW-96) : allow parameter "s3_conn_id" of
S3KeySensor and S3PrefixSensor to be defined using
an environment variable.
Actually, S3KeySensor and S3PrefixSensor use the
S3hook, which extends BaseHook. BaseHook has
get_connection, which looks a connection up :
- in environment variables first
- and then in the database
Closes#1517 from dm-tran/fix-jira-airflow-96
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
https://issues.apache.org/jira/browse/AIRFLOW-530
Right now, the documentation does not clearly
state that connection names are converted to
uppercase form when searched in the environment
(https://github.com/apache/incubator-airflow/blob/
master/airflow/hooks/base_hook.py#L60-L60).
This is confusing as the best practice in Airflow
seems to be to define connections in lower case
form.
Closes#1811 from danielzohar/connection_env_var
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-198
Testing Done:
- Local testing of dag operation with
LatestOnlyOperator
- Unit test added
Closes#1752 from gwax/latest_only
Currently dags are being read directly from the filesystem. Any
hierarchy (python namespaces, modules) need to be reflected on
the filesystem. This makes it hard to manage dags and their
depedencies.
This patch adds support for dags in zip files. It will add
the zip to sys.path and then it will read the zip file and
try to import any files as modules that are in the root of
the zip.
Please note that any module contained within the zip will
overwrite existing modules in the same namespace.
- Operators can be created without DAGs, but the DAG can be added at
any time thereafter (by assigning to the ‘dag’ attribute). Once a DAG
is assigned, it can not be removed or reassigned.
- Operators can infer DAGs from other operators. Setting a relationship
will also set the DAG, if possible. Operators from different DAGs and
operators with no DAGs can not be chained.
- DAGs can be used as context managers. When “inside” a DAG context
manager, the default DAG for all new Operators is that DAG (unless they
specify a different one)
- Unit tests
- Add default owner for Operators
- Support composing operators with >> and <<
Three special cases:
op1 >> op2 is equivalent to op.set_downstream(op2)
op1 << op2 is equivalent to op1.set_upstream(op2)
dag >> op1 (in any order or direction) means op1.dag = dag
These can be chained:
dag >> op1 >> op2 << op3
- Update concepts documentation
Kamil Breguła
Chen Tong
Dima Kamalov
Joshua Carp
OmerJog
BasPH
Felix
eladkal
rmn36
ron819
Ash Berlin-Taylor
Xiaodong
Taylor D. Edmiston
Tim Swast
Cameron Moberg
Moe Nadal
Bovard Doerschuk-Tiberi
Tristram Oaten
Tamu
Edan Schwartz
Trevor Edwards
Andrew Chen
Ruslan Dautkhanov
Daniel Huang
Duy-Minh TRAN
Daniel Zohar
George Leslie-Waksman
Maxime Beauchemin
Chris Riccomini
Mark Reid
Hervé Werner
Maxime Beauchemin
Joy Gao
bolkedebruin
jlowin
Antoine Augusti
lusop
Nicholas Travers
thibault ketterer
Maxime
Rimas Silkaitis
Jeremiah Lowin
Jeremiah Lowin
Jeremiah Lowin
Jeremiah Lowin
Marty Woodlee
e3