When using impersonation via `run_as_user`, the
PYTHONPATH environment
variable is not propagated hence there may be
issues when depending on
specific custom packages used in DAGs.
This PR propagates only the PYTHONPATH in the
process creating the
sub-process with impersonation, if any.
Tested in staging environment; impersonation tests
in airflow are not very portable and fixing them
would take additional work, leaving as TODO and
tracking with jira ticket: https://issues.apache.o
rg/jira/browse/AIRFLOW-1901.
Closes#2860 from edgarRd/erod-
pythonpath_run_as_user
[AIRFLOW-829][AIRFLOW-88] Reduce verbosity of
Travis tests
Remove the -s flag for Travis unit tests to
suppress output
from successful tests.
[AIRFLOW-830] Reduce plugins manager verbosity
The plugin manager prints all status to INFO,
which is unnecessary and
overly verbose.
Closes#2049 from jlowin/reduce-logs
Submitting on behalf of plypaul
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-219
-
https://issues.apache.org/jira/browse/AIRFLOW-398
Testing Done:
- Running on Airbnb prod (though on a different
mergebase) for many months
Credits:
Impersonation Work: georgeke did most of the work
but plypaul did quite a bit of work too.
Cgroups: plypaul did most of the work, I just did
some touch up/bug fixes (see commit history,
cgroups + impersonation commit is actually plypaul
's not mine)
Closes#1934 from aoen/ddavydov/cgroups_and_impers
onation_after_rebase
AIRFLOW-328
https://issues.apache.org/jira/browse/AIRFLOW-328
Previously, Airflow had both a default template for airflow.cfg AND a
dictionary of default values. Frequently, these get out of sync (an
option in one has a different value than in the other, or isn’t present
in the other). This commit removes the default dict and uses the
airflow.cfg template to provide defaults. The ConfigParser first reads
the template, loading all the options it contains, and then reads the
user’s actual airflow.cfg to overwrite the default values with any new
ones.
AIRFLOW-371
https://issues.apache.org/jira/browse/AIRFLOW-371
Calling test_mode() didn't actually change Airflow's configuration! This actually wasn't an issue in unit tests because the unit test run script was hardcoded to point at the unittest.cfg file, but it needed to be fixed.
[AIRFLOW-328] Remove redundant default configuration
Previously, Airflow had both a default template
for airflow.cfg AND a dictionary of default
values. Frequently, these get out of sync (an
option in one has a different value than in the
other, or isn’t present in the other). This commit
removes the default dict and uses the airflow.cfg
template to provide defaults. The ConfigParser
first reads the template, loading all the options
it contains, and then reads the user’s actual
airflow.cfg to overwrite the default values with
any new ones.
[AIRFLOW-371] Make test_mode() functional
Previously, calling test_mode() didn’t actually
do anything.
This PR renames it to load_test_config() (to
avoid confusion, ht @r39132).
In addition, manually entering test_mode after
Airflow launches might be too late — some
options have already been loaded (DAGS_FOLDER,
etc.). This makes it so setting
tests/unit_test_mode OR the equivalent env var
(AIRFLOW__TESTS__UNIT_TEST_MODE) will load the
test config immediately, prior to loading the
rest of Airflow.
Closes#1677 from jlowin/Simplify-config
- Added Apache license header for files with extension (.service, .in, .mako, .properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, .html, .xml.
- Added/Replaced shebang on all .sh files with portable version - #!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.
Closes#1598 from ajayyadava/248
There are many ways to set configuration options in Airflow
but no way to actually see all of them (the web UI only shows
airflow.cfg). This takes the current configuration object
and writes it to a dict.
The "source" of an option can be displayed (for example,
'airflow.cfg', 'default', 'env var', etc.).
Sensitive (confidential) configuration options can be included
Unless specified, they are censored as '< hidden >'.
- we now can specify nose parameter on the command line of ./run_unit_test.sh.
- if unspecified, the script falls back to previous behaviour, i.e. run all UTs
Bolke de Bruin
Joy Gao
Edgar Rodriguez
Jeremiah Lowin
Dan Davydov
Li Xuanji
jlowin
Ajay Yadav
Nicholas Travers
Thoralf Gutierrez
Maxime Beauchemin
Marcin Tustin
Svend Vanderveken
Jack Danger Canty
Maxime
Arthur Wiedmer
e3
Mike Kerzhner