* [AIRFLOW-3813] Add CLI commands to manage roles
Here is the help text of the new command `airflow roles`:
usage: airflow roles [-h] [-c] [-l] [role [role ...]]
positional arguments:
role The name of a role
optional arguments:
-h, --help show this help message and exit
-c, --create Create a new role
-l, --list List roles
Create is reentrant, i.e., it only adds a new role if it does not exist.
* Update docs on role creation
* [AIRFLOW-2694] Declare permissions in DAG definition
This PR adds support for declaratively assigning DAG-level permissions
to a role via the `DAG.__init__()` method.
When the DAG definition is evaluated and the `access_control` argument
is supplied, we update the permissions on the ViewMenu associated with
this DAG according to the following rules:
- If the role does not exist, we raise an exception.
- If the role exists, we ensure that it has the specified set of
permissions on the DAG
- If any other permissions exist for the DAG that are not specified in
`access_control`, we revoke them
* Move RBAC constants to break circular dependency
* Add license header
* Sync DAG permissions via CLI and /refresh* endpoints
Move the DAG-level permission syncing logic into
`AirflowSecurityManager.sync_perm_for_dag`, and trigger this method from
the CLI's `sync_perm` command and from the `/refresh` and `/refresh_all`
web endpoints.
* Default access_control to None
Celery supports `autoscale` by accepting values in format "max_concurrency,min_concurrency".
But the default value in default_airflow.cfg is wrong, and the comment can be clearer.
* [AIRFLOW-3787] Import/export users from JSON file
Provide a CLI command to import or export users from a JSON file. The
CLI arguments are modeled after the import/export commands for Variables
and Pools.
Example Usage:
airflow users -i users.json
airflow users -e /tmp/exported-users.json
The import command will create any users that do not yet exist and
update any users that already exist. It never deletes users.
The format of the file produced by an export is compatible with the
import command, i.e., `import(export())` should succeed but have no
side-effects.
* Add input file format to help text
Try to make the log message clearer in the presence of rescheduled tasks -
i.e that the task exited with 0/1, not the status of the task, without having each
executor having to know about reschedule or other states we might introduce.
This argument is part of the API from our parent class, but we didn't
support it because of the various steps we perform in `get()` - this
makes it behave more like the parent class, and can simplify a few
instances in our code (I've only included one that I found here)
* [AIRFLOW-3773] Fix /refresh_all endpoint
Call `sync_perm_for_dag` for each DAG in the DagBag (`dag_id` is a
required argument).
I looked for a test suite for the web UI, but it seems the existing
tests have all been disabled since the switch to FAB. I've created a new
class for FAB tests and added a test to exercise this `/refresh_all`
endpoint.
* Move tests to www/test_views.py
I didn't realize that we already had test scaffolding in place for
testing the FAB-based UI.
In master branch, we have already decommissioned the Flask-Admin UI.
In model definitions, User and Chart are only applicable for the
"old" UI based on Flask-Admin.
Hence we should decommission these two models as well.
Related doc are updated in this commit as well.
When processing HTTP response headers, gunicorn checks that the name of each
header is a string. Here's the relevant gunicorn code:
From gunicorn/http/wsgi.py, line 257
def process_headers(self, headers):
for name, value in headers:
if not isinstance(name, string_types):
raise TypeError('%r is not a string' % name)
In Python3, `string_types` is set to the built-in `str`. For Python 2,
it's set to `basestring`. Again, the relevant gunicorn code:
From gunicorn/six.py, line 38:
if PY3:
string_types = str,
...
else:
string_types = basestring,
On Python2 the `b''` syntax returns a `str`, but in Python3 it returns `bytes`.
`bytes` != `str`, so we get the following error when returning a 404 on
Python3:
File "/usr/local/lib/python3.6/site-packages/airflow/www/app.py", line 166, in root_app
resp(b'404 Not Found', [(b'Content-Type', b'text/plain')])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 261, in start_response
self.process_headers(headers)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 268, in process_headers
raise TypeError('%r is not a string' % name)
TypeError: b'Content-Type' is not a string
Dropping the `b` prefix in favor of the single-quote string syntax should work
for both Python2 and 3, as demonstrated below:
Python 3.7.2 (default, Jan 13 2019, 12:50:15)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> isinstance('foo', str)
True
Python 2.7.15 (default, Jan 12 2019, 21:43:48)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> isinstance('foo', basestring)
True
T. Tanay
mans2singh
Andrew Stahlman
Kamil Breguła
BasPH
Ryo Okubo
gseva
andyh1203
Fokko Driesprong
Stefan Seelmann
Xiaodong
Tao Feng
Ash Berlin-Taylor
Drew J. Sonne
Joshua Carp
Felix
zhongjiajie
OmerJog
Ryan Yuan
Anoop Kunjuraman
Sid Anand
Jongyoul Lee
yangaws
Rémy Léone
Kaxil Naik
Anatoli Babenia