There is two parts to this PR:
1. Only copying www/webpack.config.js and www/static/ before running the
asset pipeline
2. Making sure that _all_ files (not just the critical ones) have the
same permissions.
The goal of both of these is to make sure that the docker build cache for the "expensive"
operations (installing NPM modules, running asset pipeline, installing python modules)
isn't run when it isn't necessary.
This PR does two things:
1. It enables the mypy cache (default folder name .mypy_cache) so that
repeated runs locally are quicker
2. It _disables_ passing only the changed files in.
Point 2 seems counter-intuitave, but in my testing running with all
files (airflow docs tests) was about twice as fast as without. My
hypothesis for why this happens is that when mypy is checking file x, it
has to check dependencies/imports for it too, and when we have
pass_filenames set runs multiple processes in parallel, and each of them
have to do this work!
Timings before and after:
- Before:
For all files
```
❯ time pre-commit run mypy -a
Run mypy.................................................................Passed
pre-commit run mypy -a 0.31s user 0.07s system 2% cpu 17.140 total
```
With only a single file
```
❯ time pre-commit run mypy --files airflow/configuration.py
Run mypy.................................................................Passed
pre-commit run mypy --files airflow/configuration.py 0.30s user 0.07s system 5% cpu 6.724 total
```
- After:
With a clean cache (`rm -rf .mypy_cache`):
```
$ time pre-commit run mypy
Run mypy.................................................................Passed
pre-commit run mypy -a 0.26s user 0.10s system 2% cpu 17.226 total
```
Clean cache with single file:
```
$ time pre-commit run mypy --file airflow/version.py
Run mypy.................................................................Passed
pre-commit run mypy --file airflow/version.py 0.23s user 0.07s system 4% cpu 7.091 total
```
Repeated run (cache folder exists):
```
$ time pre-commit run mypy --file airflow/version.py
Run mypy.................................................................Passed
pre-commit run mypy --file airflow/version.py 0.23s user 0.05s system 6% cpu 4.178 total
```
and for all files
```
airflow ❯ time pre-commit run mypy -a
Run mypy.................................................................Passed
pre-commit run mypy -a 0.25s user 0.09s system 6% cpu 4.833 total
```
commit 50efda5c69 introduced a bug that
prevents scheduler from scheduling tasks with the following properties:
* has depends on past set to True
* has custom concurrency limit
* [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support
* [AIRFLOW-6531] Move from contrib to providers. Drop py2 support
* [AIRFLOW-6531] Add service account support to YC connection
* [AIRFLOW-6531] Use Dataproc wrapper
* [AIRFLOW-6531] Move base classes to init to test "test_providers_modules_should_have_tests"
* [AIRFLOW-6531] Add type annotations. Remove op base class
* [AIRFLOW-6531] Add cluster id to templated fields
commit 50efda5c69 introduced a bug that
prevents the scheduler from enforcing max active run config for all
DAGs.
this commit fixes the regression as well as the test.
* [AIRFLOW-6590] Use batch db operations in jobs
The PR changes numerous single selects / updates in base,
scheduler, and backfill jobs to bulk operations.
* fixup! [AIRFLOW-6590] Use batch db operations in jobs
* fixup! fixup! [AIRFLOW-6590] Use batch db operations in jobs
Don't try to find changed files unless we are building a pull request.
This only caused a problem on build of tags, but we were also doing this
for master/branch builds, but it was always saying finding no files
changed.
By checking this early we can make the other conditions in this function
simpler.
* [AIRFLOW-6770] Run particular test using breeze CLI bug fix
* [AIRFLOW-6770] Fix typo in travis config
* [AIRFLOW-6770] Fix variable name and remove unnecessary travis command
* [AIRFLOW-6766] Fix "cannot import ensure_text" error for pre-commit
As of today Travis bundles six version 1.11.0 with their python
3.6 image and it misses ensure_text method. Bumping to 1.14+
solves the problem.
* [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs
* fixup! [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs
Co-Authored-By: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
If the number of dags was large and/or the length of the DAG ids were too large this would exceed the maximum possible query string limit.
To work around that we have made these endpoints always make POST requests
* [AIRFLOW-6733] Extend, rather than replace, the base_template
This commit just moves the existing template to airflow/master.html
(without further changes)
* [AIRFLOW-6733] Only change blocks we have customized
This makes is easier to see which parts of the template we have changed.
This parameter is deprecated by werkzeug, see:
https://github.com/pallets/werkzeug/blob/0.16.1/src/werkzeug/middleware/proxy_fix.py#L113-L120
However, it is also broken. The value is acquired as string from the
config, while it should be int like the other `x_*` attributes. Those
were fixed in #6901, but `num_proxies` was forgotten.
I think we can safely remove it because:
* There is virtually no possibility that someone is using that parameter
in their config without raising an exception.
* The configuration variable is not present in Airflow's docs or
anywhere else anymore. The removed line is the only trace of it.
More details:
https://issues.apache.org/jira/browse/AIRFLOW-6740
Kubernetes Git test uses always apache/airflow:master as source of files
This made it impossible to test how the Kubernetes GitSync test will behave
when merged. The git sync should always be done with the original repo/branch
or from the TRAVIS_BRANCH if this is a push build
Jarek Potiuk
Ash Berlin-Taylor
QP Hou
peter-volkov
yuqian90
Tomek Urbaszek
davidshopback
chandu kavar
damon09273@gmail.com
Kamil Breguła
Kumpan Anton
Cyril Shcherbin
davlum
Jithin Sukumar
Michael Spector
Bhavika Tekwani
Jarek Potiuk
Louis Guitton
Van-Duyet Le
Kaxil Naik
Madison Bowden
Anita Fronczak
Daniel Cohen
Sebastian Brandt
Clinton Dow
Kamil Breguła