We added (and used) fallback as an argument on `getboolean` but didn't
add it to the method, or add tests covering those "casting" accessors,
so they broke.
This fixes those methods, and adds tests covering them
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
* [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator
This is a minor change to allow Jinja templating in parameters where it
makes sense for SparkSubmitOperator.
To enable to spark behavior of transporting and extracting an archive
on job launch, making the _contents_ of the archive available to the
driver as well as the workers (not just the jar or archive as a zip
file) - this configuration attribute is necessary.
This is required if you have no ability to modify the Python env on
the worker / driver nodes, but you wish to use versions, modules, or
features not installed.
We transport a full Python 3.5 environment to our CDH cluster using
this option and the alias "#PYTHON" paired an additional configuration
to spark to use it:
--archives "hdfs:///user/myuser/my_python_env.zip#PYTHON"
--conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PYTHON/python35/bin/python3"
* [AIRFLOW-3813] Add CLI commands to manage roles
Here is the help text of the new command `airflow roles`:
usage: airflow roles [-h] [-c] [-l] [role [role ...]]
positional arguments:
role The name of a role
optional arguments:
-h, --help show this help message and exit
-c, --create Create a new role
-l, --list List roles
Create is reentrant, i.e., it only adds a new role if it does not exist.
* Update docs on role creation
* [AIRFLOW-2694] Declare permissions in DAG definition
This PR adds support for declaratively assigning DAG-level permissions
to a role via the `DAG.__init__()` method.
When the DAG definition is evaluated and the `access_control` argument
is supplied, we update the permissions on the ViewMenu associated with
this DAG according to the following rules:
- If the role does not exist, we raise an exception.
- If the role exists, we ensure that it has the specified set of
permissions on the DAG
- If any other permissions exist for the DAG that are not specified in
`access_control`, we revoke them
* Move RBAC constants to break circular dependency
* Add license header
* Sync DAG permissions via CLI and /refresh* endpoints
Move the DAG-level permission syncing logic into
`AirflowSecurityManager.sync_perm_for_dag`, and trigger this method from
the CLI's `sync_perm` command and from the `/refresh` and `/refresh_all`
web endpoints.
* Default access_control to None
Celery supports `autoscale` by accepting values in format "max_concurrency,min_concurrency".
But the default value in default_airflow.cfg is wrong, and the comment can be clearer.
* [AIRFLOW-3787] Import/export users from JSON file
Provide a CLI command to import or export users from a JSON file. The
CLI arguments are modeled after the import/export commands for Variables
and Pools.
Example Usage:
airflow users -i users.json
airflow users -e /tmp/exported-users.json
The import command will create any users that do not yet exist and
update any users that already exist. It never deletes users.
The format of the file produced by an export is compatible with the
import command, i.e., `import(export())` should succeed but have no
side-effects.
* Add input file format to help text
Try to make the log message clearer in the presence of rescheduled tasks -
i.e that the task exited with 0/1, not the status of the task, without having each
executor having to know about reschedule or other states we might introduce.
This argument is part of the API from our parent class, but we didn't
support it because of the various steps we perform in `get()` - this
makes it behave more like the parent class, and can simplify a few
instances in our code (I've only included one that I found here)
BasPH
Felix
Tao Feng
Sarmad Ali
Amarnath
Patouche
Kamil Breguła
Fokko Driesprong
Tanay Tummalapalli
Verdan Mahmood
Mathew Wicks
Chris Nordqvist
Penumbra69
mans2singh
Andrew Stahlman
Ryo Okubo
gseva
andyh1203
Stefan Seelmann
Xiaodong
Ash Berlin-Taylor
Drew J. Sonne
Joshua Carp