Граф коммитов

11268 Коммитов

Автор SHA1 Сообщение Дата
Jarek Potiuk 1a0e0bc793
Array declaration cannot be done inside function for DockerHub (#12967)
For version of bash in DockerHub the arrays should be declared
outside of functions to work.
2020-12-10 17:06:53 +01:00
Kaxil Naik a8ded09038
Remove trailing back-tick from docs (#12986) 2020-12-10 17:02:37 +01:00
Kaxil Naik 056fe81ec2
Fix command to filter package provider when building docs (#12984) 2020-12-10 15:57:21 +00:00
Xiaodong DENG fbd8348d0c
Allow all default roles to view Profile page + allow editing profile/resetting password for DB-ModelView. (#12971)
This is a change discussed long time back in https://github.com/apache/airflow/pull/3889#issuecomment-507635839

Essentially, the 7 permission-resource pairs are added for all users:
- can_this_form_post on UserInfoEditView
- can_this_form_get on UserInfoEditView
- can_userinfo on UserDBModelView
- userinfoedit on UserDBModelView
- can_this_form_post on ResetMyPasswordView
- can_this_form_get on ResetMyPasswordView
- resetmypassword on UserDBModelView

In addition, can_userinfo is added for all possible User ModelViews, so users can also view profile when
the webserver is using different setting-ups.
But they are ONLY allowed to edit profile and reset password when it's UserDBModelView
2020-12-10 16:04:37 +01:00
Vikram Koka e5ce657c5f
Cleaned up formatting for Upgrading doc (#12977)
Cleaned up some of the formatting in the document and also fixed a couple
of spelling errors.
2020-12-10 10:24:07 +00:00
Kaxil Naik 28b7ebc25a
Remove duplicate docs for check-hooks-apply pre-commit (#12973) 2020-12-09 22:31:52 +01:00
Kaxil Naik db166ba75c
Update CI to run tests againt v2-0-test branch (#10891) 2020-12-09 19:35:04 +00:00
Jarek Potiuk 32971a1a2d
Updates providers versions to 1.0.0 (#12955)
Please enter the commit message for your changes. Lines starting
2020-12-09 20:25:41 +01:00
Kaxil Naik 4fe156f98a
Remove unused pre-commit and Fix CI (#12964) 2020-12-09 18:44:10 +00:00
Ash Berlin-Taylor bee2cb5317
Don't reference sphinx airflow theme via `@` URL in requirements. (#12957)
PyPI rejects uploading a dist with an `@` in the requirements.
2020-12-09 16:29:22 +00:00
John Bampton b8844535d7
Minor HTML fixes (#12477)
- indent HTML code
- add DOCTYPE
- add lang="en" to the HTML start tag
2020-12-09 11:20:14 -05:00
Ash Berlin-Taylor f96b3dfbe3
Update version to 2.0.0 (#12954) 2020-12-09 15:37:27 +00:00
John Bampton 5eeb66ce18
Fix case of GitHub (#12950)
Changed `Github` to `GitHub`
2020-12-09 16:36:25 +01:00
Kamil Breguła ec3b159884
Clean up redirects.txt (#12948) 2020-12-09 16:24:10 +01:00
dstandish 62a5b2dfa4
Fix return type in prev-date context variables (#12910) 2020-12-09 15:10:03 +00:00
Ash Berlin-Taylor d2b4dea6b8
Update theme to show hidden ToC in sidebar (#12949)
Closes #12554
2020-12-09 14:55:15 +00:00
Anderson Reyes 1d91ca70f2
Infer multiple outputs from dict annotations in TaskFlow API (#10349) 2020-12-09 14:45:15 +00:00
Ash Berlin-Taylor 0bf386fdf2
Rename airflow.operators.dagrun_operator to airflow.operators.trigger_dagrun (#12933)
Part of AIP-21

Co-authored-by: Kishore Vancheeshwaran <24776049+kishvanchee@users.noreply.github.com>
2020-12-09 14:00:51 +00:00
Vikram Koka 3e402880bf
Upgrading to Airflow 2.0 doc (#12930)
This is based heavily on the UPGRADING_to_2.0.md file. The intent here is
to make user facing and available as part of the Airflow documentation.
This is still a work in progress.

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
2020-12-09 13:02:00 +00:00
Jarek Potiuk 7bd867d745
Adds documentation about custom providers. (#12921)
Closes: #11429
2020-12-09 11:35:54 +01:00
Ash Berlin-Taylor 55573f684f
Fix typos/spelling in docs (#12942)
When building the docs locally (not docker, cos I'm on linux and a
grouch at times 😁) I appear to be using a different backend to
enchant to check spelling, and it noticed these two issues

- spelling of yandex in extra-packages unrecognized
- a smart quote instead of a backtick breaking a link
2020-12-09 10:06:41 +00:00
Vikram Koka 18a7a35eab
Enhanced TaskFlow API tutorial to use @dag decorator (#12937)
Updated both the tutorial python file in the example_dags directory
and the tutorial documentation
2020-12-09 09:40:45 +00:00
Jarek Potiuk 5c74c3a5c1
Add explanation of the Presto Kerberos + SSL + snowflake problem (#12939) 2020-12-09 09:38:53 +01:00
Kaxil Naik 8d6840b19e
Fix typo in setup.cfg (#12935)
```
airflow.api.connextion.openaip=*.yaml
```
to
```
airflow.api_connexion.openapi=*.yaml
```
2020-12-09 08:53:43 +01:00
Jarek Potiuk d7ddf16a36
Fixed failed pylint in master (#12938) 2020-12-09 08:28:06 +01:00
dstandish 53c48471a6
shorten name of hook re imports of provide_session and create_session (#12936) 2020-12-09 06:27:11 +01:00
Ash Berlin-Taylor 63ea88d1b1
Apply labels to Docker images in a single instruction (#12931)
* Apply labels to Docker images in a single instruction

While looking at the build logs for something else I noticed this
oddity at the end of the CI logs:

```
Tue, 08 Dec 2020 21:20:19 GMT Step 125/135 : LABEL org.apache.airflow.distro="debian"
...
Tue, 08 Dec 2020 21:21:14 GMT Step 133/135 : LABEL org.apache.airflow.commitSha=${COMMIT_SHA}
Tue, 08 Dec 2020 21:21:14 GMT  ---> Running in 1241a5f6cdb7
Tue, 08 Dec 2020 21:21:21 GMT Removing intermediate container 1241a5f6cdb7
```

Applying all the labels took 1m2s! Hopefully applying these in a single
layer/command should speed things up.

A less extreme example still took 43s

```
Tue, 08 Dec 2020 20:44:40 GMT Step 125/135 : LABEL org.apache.airflow.distro="debian"
...
Tue, 08 Dec 2020 20:45:18 GMT Step 133/135 : LABEL org.apache.airflow.commitSha=${COMMIT_SHA}
Tue, 08 Dec 2020 20:45:18 GMT  ---> Running in dc601207dbcb
Tue, 08 Dec 2020 20:45:23 GMT Removing intermediate container dc601207dbcb
Tue, 08 Dec 2020 20:45:23 GMT  ---> 5aae5dd0f702
```

* Update Dockerfile
2020-12-09 06:19:38 +01:00
Daniel Imberman 38350620bc
Add Clarification to pod_template_file exmaples (#12932)
Addresses https://github.com/apache/airflow/issues/11686

Clarifies why we have three files of pod_template_file examples
2020-12-08 20:37:32 -08:00
Ian Stanton bfbe486d9d
Fix typo in metrics doc (#12934) 2020-12-09 00:12:53 +00:00
Kaxil Naik a075b6df99
Rename remaining Sensors to match AIP-21 (#12927)
As discussed in AIP-21

* Rename airflow.sensors.external_task_sensor to airflow.sensors.external_task
* Rename airflow.sensors.sql_sensor to airflow.sensors.sql
* Rename airflow.contrib.sensors.weekday_sensor to airflow.sensors.weekday
2020-12-09 00:09:08 +00:00
Kamil Breguła e595d35bf4
Simplify publishing of documentation (#12892)
Close: #11423
Close: #11152
2020-12-09 00:03:22 +00:00
James Timmins 73843d05f7
Disable Pause/Unpause switch if user doesn't have edit access for DAG. (#12911) 2020-12-08 21:26:24 +00:00
Xiaodong DENG 3da939bbe2
Update doc to reflect the changes for DAG-Level access control (#12928)
In 2.0 and master branch, there are permission & view name changes for DAG-level access control.

- Permission: "can_dag_read"/"can_dag_edit" -> "can_read"/"can_edit"
- View: "all_dags" -> "DAGs"

These were missed to be reflected in the doc, and this PR addresses it.
2020-12-08 20:30:38 +01:00
Kamil Breguła abce78c53b
Disable experimental REST API by default (#12337) 2020-12-08 19:27:27 +00:00
Kishore Vancheeshwaran d5589673a9
Move dummy_operator.py to dummy.py (#11178) (#11293)
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
2020-12-08 19:19:46 +00:00
Ruslan c704293f75
Warning about unsafe migrations (#10254)
Co-authored-by: Kaxil Naik <kaxilnaik@apache.org>
2020-12-08 18:32:29 +00:00
Kaxil Naik 493981600f
Increase timeout for SQLite tests to 80 minutes (#12926)
We already have 80 minutes timeout for Postgres and MySQL tests.

Currently some of Sqlite tests on PRs are timing out since they
are taking more than 60 mintues.

Example: https://github.com/apache/airflow/runs/1518719040
2020-12-08 18:31:56 +00:00
Ash Berlin-Taylor b40dffa085
Rename remaing modules to match AIP-21 (#12917)
As discussed in AIP-21

* Rename airflow.hooks.base_hook to airflow.hooks.base
* Rename airflow.hooks.dbapi_hook to airflow.hooks.dbapi
* Rename airflow.sensors.base_sensor_operator to airflow.sensors.base
* Rename airflow.sensors.date_time_sensor to airflow.sensors.date_time
* Rename airflow.sensors.time_delta_sensor to airflow.sensors.time_delta

Co-authored-by: Kaxil Naik <kaxilnaik@apache.org>
2020-12-08 18:01:58 +00:00
Ryan Hamilton ed6eb4278d
Use "auto" instead of "scroll" to prevent visible (unused) scrollbars in some browsers (#12923) 2020-12-08 12:28:28 -05:00
Jarek Potiuk 738c953677
Add missing licences. (#12922)
This PR adds missing licences before the RC release or adds
appropriate exclusions.

Closes: #11632
2020-12-08 17:32:29 +01:00
Kishore Vancheeshwaran bfbd4bbb70
Moved subdag_operator.py to subdag.py (#11307)
Part of #11178
2020-12-08 16:26:39 +00:00
Vikram Koka 0da5232632
Added documentation for Airflow Upgrade Check (#12872) 2020-12-08 16:19:47 +00:00
Sam Wheating 7c028f5278
ImportErrors are collapsed by default (#12811) 2020-12-08 11:09:23 -05:00
Kaxil Naik bc4d9d04cd
Remove duplicate entry in Updating.md (#12919) 2020-12-08 16:41:47 +01:00
Ash Berlin-Taylor e7c1771cba
Adds predefined providers to install_requires. (#12916)
* The 4 providers (http, ftp, sqlite, imap) are popular
and they do not require any additionl dependencies so we decided
to include them by default in Airflow 2.0

Co-authored-by: Jarek Potiuk <jarek.potiuk@polidea.com>

* Update setup.py

Co-authored-by: Jarek Potiuk <jarek.potiuk@polidea.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
2020-12-08 16:22:47 +01:00
Jarek Potiuk 9b39f24780
Add support for dynamic connection form fields per provider (#12558)
Connection form behaviour depends on the connection type. Since we've
separated providers into separate packages, the connection form should
be extendable by each provider. This PR implements both:

  * extra fields added by provider
  * configurable behaviour per provider

This PR will be followed by separate documentation on how to write your
provider.

Also this change triggers (in tests only) the snowflake annoyance
described in #12881 so we had to xfail presto test where monkeypatching
of snowflake causes the test to fail.

Part of #11429
2020-12-08 16:00:37 +01:00
Jarek Potiuk 4d24c5ee55
Update comments in setup.py (#12903)
Updates comments in setup.py according to the latest 2.0 agreements
about semver and clarifies use for arrays defined in setup.py.

We will refactor those after 2.0 further, but for now we keep
the current structure.
2020-12-08 15:59:37 +01:00
Kaxil Naik ff25bd6ffe
Make xcom_pull results order deterministic (#12905)
closes https://github.com/apache/airflow/issues/11858
2020-12-08 13:00:06 +00:00
James Timmins 85dd092a8e
Make sure perms list isn't None before looking within list (#12915) 2020-12-08 12:44:38 +00:00
Jarek Potiuk f9e9ad2b09
Builds prod images on DockerHub from packages (#12908)
This build combines building both CI and PROD image in one
script execution on DockerHub per python version.

First the CI image is build and secondly, the image is used
to build all the packages from sources and then those
packages are used to build the PROD image.

Resulting image will be package image built from latest sources.

Closes: #12261
2020-12-08 12:45:03 +01:00