Граф коммитов

10400 Коммитов

Автор SHA1 Сообщение Дата
Jarek Potiuk c78329b750
Few more small fixes found during preparation of the backports (#11848) 2020-10-26 11:40:20 +01:00
yangrong688 a5dd18a5ed
Fix commands in docs/usage-cli.rst(#11847)
* Add uSmart Securities to the INTHEWILD.md

* change airflow dag to airflow dags in 2.0
2020-10-26 10:34:23 +01:00
Jarek Potiuk 00bec1b09e
Fix the script that builds source for backports (#11846)
First time preparing backports after converting scripts to
also support regular providers. Some small bugs were found
and fixed.
2020-10-26 10:18:16 +01:00
John Bampton 8afdb6ac6a
Fix spellings (#11825) 2020-10-26 01:27:11 +00:00
Ephraim Anierobi c62a49af07
Dag Run endpoints returns count total results after filtering (#11832) 2020-10-25 22:35:30 +01:00
Jarek Potiuk 872b1566a1
Generated backport providers readmes/setup for 2020.10.29 (#11826) 2020-10-25 19:45:50 +01:00
Kamil Breguła 998bb0d5fc
Fix example DAGs tests + add sanity checks (#11840) 2020-10-25 17:25:52 +01:00
Jarek Potiuk 5a396835e6
Fixes typos in production-deployment docs (#11833) 2020-10-25 15:09:39 +01:00
Jarek Potiuk f5410f2486
Removes duplicates from DISABLED_INTEGRATIONS variable (#11831)
Presto DB is checked several times but it also means that
it is added several times to DISABLED_INTEGRATIONS in case
it is not enabled. This commit fixes it.
2020-10-25 11:30:23 +01:00
Kaxil Naik 330cd427e1
Fix bug when marking tasks when DAG Serialization is enabled (#11803)
This is because `current_task` is of SerializedBaseOperator type.
2020-10-24 16:01:54 +01:00
Kaxil Naik 97da08cc8a
Update download url for Airflow Providers Version (#11823)
All the versions are available at https://archive.apache.org/dist/airflow/ and this link appears in https://pypi.org/project/apache-airflow/
2020-10-24 17:00:48 +02:00
Kaxil Naik f93175d5a7
Fix typo in scripts/in_container/entrypoint_ci.sh (#11824)
* Fix typo in scripts/in_container/entrypoint_ci.sh

* Update bats_tests.sh
2020-10-24 16:58:01 +02:00
Jarek Potiuk 8d94214575
Switch postgres from 10 to 13 (#11785)
Seems that postgres is really stable when it comes to upgrades,
so we take the assumption that if we test 9.6 and 13, and they
work, all the versions between will also work.

This PR changes Postgres 10 to 13 in tests  and updates documentation
with all the versions in between.
2020-10-24 14:39:01 +02:00
Jarek Potiuk dff9c9a96f
Fixes flaky test test_should_response_200_with_reset_dag_run (#11817)
Sometimes the rows were returned in reverse order.

Fixes #11816
2020-10-24 14:37:08 +02:00
John Bampton 6ce855af11
Fix spelling (#11821) 2020-10-24 14:36:52 +02:00
John Bampton aacf6025f6
Fix spelling (#11457) 2020-10-24 12:23:52 +02:00
Jarek Potiuk e4adfd7a24
Add links to PyPI packages. (#11818)
Same as in #11801 but in provider's packages.
2020-10-24 12:23:04 +02:00
Kaxil Naik 3934ef2249
Remove redundant builtins imports (#11809)
These imports are not needed in Python3
2020-10-24 10:36:55 +02:00
John Bampton 4c8e033c0e
Fix spelling and grammar (#11814) 2020-10-24 10:06:04 +02:00
Kaxil Naik 0218bcf838
Use LocalExecutor by default with tmux + Breeze (#11791)
* Use LocalExecutor by default with tmux + Breeze

* Update run_tmux.sh

* Update run_tmux.sh
2020-10-24 08:07:19 +02:00
Kaxil Naik 7b89a04522
Replace io.open with builtin open (#11807)
From Python 3 `io.open` is an alias for the builtin `open()` function.

Docs: https://docs.python.org/3/library/io.html#io.open
2020-10-24 05:06:33 +01:00
Kaxil Naik 56d72e3ff8
Replace non-empty sets with set literals (#11810)
Example:

```python
{paused_dag_id for paused_dag_id, in paused_dag_ids}
```

Instead of

```python
set(paused_dag_id for paused_dag_id, in paused_dag_ids)
```
2020-10-24 04:49:31 +01:00
Kaxil Naik ee9db04084
Remove redundant blank-line and parenthesis (#11811) 2020-10-24 04:26:09 +01:00
Kaxil Naik 9c6f9de796
Update to latest isort & pre-commit-hooks (#11813) 2020-10-24 04:25:15 +01:00
Kaxil Naik b4c4849471
Remove redundant "UTF-8" in Python 3 (#11808)
```
In [1]: "result".encode("UTF-8")
Out[1]: b'result'

In [2]: "result".encode()
Out[2]: b'result'
```
2020-10-24 03:49:46 +01:00
Kaxil Naik 9bc861b59f
Use PEP 380: Syntax for Delegating to a Subgenerator (#11805)
Since Master is Py 3.6>= we can use PEP 380 which was introduced in Python 3.3.

https://docs.python.org/3/whatsnew/3.3.html#pep-380
2020-10-24 03:31:56 +01:00
Kaxil Naik 4830687453
Use Python 3 style super classes (#11806)
example:

```
super().__init__(label, validators, **kwargs)
```

instead of
```
super(DateTimeWithTimezoneField, self).__init__(label, validators, **kwargs)
```
2020-10-24 03:23:02 +01:00
Kaxil Naik 6ca933b334
Remove loading DagBag multiple times in test_views.py:TestDecorators (#11804)
Previously we were loading DagBag twice, once in `setUpClass` and once in `setUp` (which is called everytime a test in that class is run)
2020-10-24 02:48:57 +01:00
Kaxil Naik 966a8cce10
Bump attrs to > 20.0 (#11799)
fixes https://github.com/apache/airflow/issues/11756
2020-10-24 01:58:26 +01:00
Kaxil Naik 081ba616ad
Update download url for Airflow Version (#11800)
All the versions are available at https://archive.apache.org/dist/airflow/ and this link appears in https://pypi.org/project/apache-airflow/
2020-10-24 00:33:00 +01:00
Kaxil Naik 0f647115a9
Add Project URLs for PyPI page (#11801)
These links would show up at https://pypi.org/project/apache-airflow/
2020-10-24 00:27:28 +01:00
Jed Cunningham c8145a697f
Fix backwards compatibility with k8s executor_config resources (#11796) 2020-10-23 14:23:07 -07:00
Ash Berlin-Taylor affee2938e
Fix locking issue stopping tasks running with SequentialExecutor (#11797)
Missing a commit means that the the row level lock was not released
before `executor.heartbeat()` was called.  This was only a problem for
the SequentialExecutor, as all the other executors would continue
running the scheduler code so the lock would be released shortly after
tasks are sent to the executor anyway. (Where as SequentialExecutor
doesn't return control until tasks have run!)
2020-10-23 22:21:03 +01:00
Marian Cepok 24e2152864
[AIRFLOW-6585] Fixed Timestamp bug in RefreshKubeConfigLoader (#11219)
Co-authored-by: Jan Brusch <jan.brusch@neuland-bfi.de>
Co-authored-by: Marian Cepok <marian.cepok@neuland-bfi.de>
2020-10-23 22:19:43 +01:00
John Bampton 5e25c167cd
Standardize the Airflow CLI help descriptions (#11790) 2020-10-23 22:51:37 +02:00
Martijn Pieters 7fec7ede96
Log task_instance execution duration as milliseconds (#10632)
This is best achieved by passing a `timedelta()` to `Stats.timing()`, and leave
worrying about time units to that method.
2020-10-23 19:20:41 +01:00
Ryan Hamilton 36a4d6c1dc
Improve interaction with Recent Tasks/DAG Runs circles by ignoring pointer-events (#11786)
Resolves #11563. With the hover event triggering the tooltips, the tooltip was triggering a mouse-out event when the cursor was at the top of the circle (where the tooltip is presented). This was causing the fluttering behavior due to the looping off these events.
2020-10-23 19:19:14 +01:00
Ryan Hamilton 73743f3421
Improve the visual presentation of the variable import form (#11783)
No functional changes, just visual enhancement of the variable import
form. This simply refactors the markup slightly to better utilize some Bootstrap
styling and add an icon to the button.
2020-10-23 19:18:29 +01:00
Jarek Potiuk 53e5d8f1f2
The .pypirc file is read from docker-context-files (#11779)
If you used context from git repo, the .piprc file was missing and
COPY in Dockerfile is not conditional.

This change copies the .pypirc conditionally from the
docker-context-files folder instead.

Also it was needlessly copied in the main image where it is not
needed and it was even dangerous to do so.
2020-10-23 17:55:15 +02:00
Kaxil Naik cc9c4c682e
Add reference link for KubernetesPodOperator in kubernetes.rst (#11782)
This makes it easy to go to the class definition and find the arguments/params that can be passed to the Operator
2020-10-23 16:36:11 +01:00
Jarek Potiuk 4d04bb663c
The .tar.gz provider packages are installable now. (#11630)
The packages lacked setup.py and they could not be installed.

This change automatically generates setup.py for the packages and
adds them to the packages.

Fixes: #11546
2020-10-23 16:47:47 +02:00
yangrong688 5fc06a6ef9
Add uSmart Securities to the INTHEWILD.md (#11757) 2020-10-23 14:08:06 +01:00
Ryan Hamilton 2df9d1c4d7
fix: Override FAB styling (#11752) 2020-10-23 14:06:52 +01:00
Jarek Potiuk cca5d63186
Add test types displayed in CI (#11770)
This is a follow up after #11659. Thanks to this one, it will
be immediately visible in CI which types of tests are run.

This change has been lost during one of the rebases, so it is
brought back now.
2020-10-23 14:19:40 +02:00
Ash Berlin-Taylor 28229e9908
Fix FAB actions with for models with composite PKs (#11753)
This fixes #11513 -- and has been submitted upstream to FAB as
https://github.com/dpgaspar/Flask-AppBuilder/pull/1493, once that is
merged we can remove this override.
2020-10-23 11:05:25 +01:00
Ash Berlin-Taylor f603b36aa4
Ensure that manually creating a DAG run doesn't "block" the scheduler (#11732)
It was possible to "block" the scheduler such that it would not
schedule or queue tasks for a dag if you triggered a DAG run when the
DAG was already at the max active runs.

This approach works around the problem for now, but a better longer term
fix for this would be to introduce a "queued" state for DagRuns, and
then when manually creating dag runs (or clearing) set it to queued, and
only have the scheduler set DagRuns to running, nothing else -- this
would mean we wouldn't need to examine active runs in the TI part of the
scheduler loop, only in DagRun creation part.

Fixes #11582
2020-10-23 09:51:03 +01:00
Darwin Yip 0df60b7736
Add reattach flag to ECSOperator (#10643)
..so that whenever the Airflow server restarts, it does not leave rogue ECS Tasks. Instead the operator will seek for any running instance and attach to it.
2020-10-23 09:10:07 +02:00
Kamil Breguła e4c239fc98
Improve web server stopping (#11734) 2020-10-22 23:36:01 +02:00
Jarek Potiuk 0647888c15
Enables splitting tests into smaller chunks (#11659)
We've implemented the capability of running the tests in smaller
chunks and selective running only some of those, but this
capability have been disabled by mistake by default setting of
TEST_TYPE to "All" and not removing it when TEST_TYPES are set
to the sets of tests that should be run.

This should speed up many of our tests and also hopefully
lower the chance of EXIT 137 errors.
2020-10-22 23:25:00 +02:00
Kaxil Naik 95be3eec42
Fix Static Checks (#11749) 2020-10-22 19:00:15 +01:00