We had `from .models import *` inside test/__init__.py as a kludge around this test
file not being named according to expected Python conventions. Renaming the file
makes more test tools happier (and makes it easier to run a single test file without
importing half of the test tree which the current approach suffers from)
Add lineage support by having inlets and oulets
that
are made available to dependent upstream or
downstream
tasks.
If configured to do so can send lineage data to a
backend. Apache Atlas is supported out of the box.
Closes#3321 from bolkedebruin/lineage_exp
on_kill methods were not triggered, due to
processes
not being properly terminated. This was due to the
fact
the runners use a shell which is then replaced by
the
child pid, which is unknown to Airflow.
Closes#3204 from bolkedebruin/AIRFLOW-1623
The return from the subprocess is in bytes when
the universal
newlines is set to False (default). This will fail
in Py3 and
works fine in Py2. And with a working unit test.
Closes#2158 from abij/AIRFLOW-840
In the local api client the execution date was
hardi coded to None.
Secondly, when no execution date was specified the
execution date
was set to datetime.now(). Datetime.now() includes
the fractional seconds
that are supported in the database, but they are
not supported in
a.o. the current logging setup. Now we cut off
fractional seconds for
the execution date.
Closes#2064 from bolkedebruin/AIRFLOW-856
Submitting on behalf of plypaul
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-219
-
https://issues.apache.org/jira/browse/AIRFLOW-398
Testing Done:
- Running on Airbnb prod (though on a different
mergebase) for many months
Credits:
Impersonation Work: georgeke did most of the work
but plypaul did quite a bit of work too.
Cgroups: plypaul did most of the work, I just did
some touch up/bug fixes (see commit history,
cgroups + impersonation commit is actually plypaul
's not mine)
Closes#1934 from aoen/ddavydov/cgroups_and_impers
onation_after_rebase
- Added Apache license header for files with extension (.service, .in, .mako, .properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, .html, .xml.
- Added/Replaced shebang on all .sh files with portable version - #!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.
Closes#1598 from ajayyadava/248
It’s conceivable that the bucket starts with g or s, in which case this
lstrip would remove characters from the bucket name. Instead, use
urlparse to properly parse the string.
Also clean up the equivalent function in the S3 and GCS Hooks, using
`strip` to clean up leading/trailing slashes.
There are many ways to set configuration options in Airflow
but no way to actually see all of them (the web UI only shows
airflow.cfg). This takes the current configuration object
and writes it to a dict.
The "source" of an option can be displayed (for example,
'airflow.cfg', 'default', 'env var', etc.).
Sensitive (confidential) configuration options can be included
Unless specified, they are censored as '< hidden >'.
Kamil Breguła
Tomek
Jiajie Zhong
Joshua Carp
Peter van 't Hof
Xiaodong
Bolke de Bruin
Alexander Bij
Jeremiah Lowin
Dan Davydov
Ajay Yadav
jlowin
Maxime Beauchemin
Sunrin SHIMURA (keen)
asnir
Thoralf Gutierrez
Jeremiah Lowin