[AIRFLOW-XXXX] Speed up mypy runs. (#7421)

This PR does two things:

1. It enables the mypy cache (default folder name .mypy_cache) so that
   repeated runs locally are quicker
2. It _disables_ passing only the changed files in.

Point 2 seems counter-intuitave, but in my testing running with all
files (airflow docs tests) was about twice as fast as without. My
hypothesis for why this happens is that when mypy is checking file x, it
has to check dependencies/imports for it too, and when we have
pass_filenames set runs multiple processes in parallel, and each of them
have to do this work!

Timings before and after:

- Before:

  For all files
  ```
  ❯ time pre-commit run mypy -a
  Run mypy.................................................................Passed
  pre-commit run mypy -a  0.31s user 0.07s system 2% cpu 17.140 total
  ```

  With only a single file

  ```
  ❯ time pre-commit run mypy --files airflow/configuration.py
  Run mypy.................................................................Passed
  pre-commit run mypy --files airflow/configuration.py  0.30s user 0.07s system 5% cpu 6.724 total
  ```

- After:

  With a clean cache (`rm -rf .mypy_cache`):

  ```
  $ time pre-commit run mypy
  Run mypy.................................................................Passed
  pre-commit run mypy -a  0.26s user 0.10s system 2% cpu 17.226 total
  ```

  Clean cache with single file:

  ```
  $ time pre-commit run mypy  --file airflow/version.py
  Run mypy.................................................................Passed
  pre-commit run mypy --file airflow/version.py  0.23s user 0.07s system 4% cpu 7.091 total
  ```

  Repeated run (cache folder exists):

  ```
  $ time pre-commit run mypy  --file airflow/version.py
  Run mypy.................................................................Passed
  pre-commit run mypy --file airflow/version.py  0.23s user 0.05s system 6% cpu 4.178 total
  ```

  and for all files

  ```
  airflow ❯ time pre-commit run mypy  -a
  Run mypy.................................................................Passed
  pre-commit run mypy -a  0.25s user 0.09s system 6% cpu 4.833 total
  ```
This commit is contained in:
Ash Berlin-Taylor 2020-02-15 14:30:31 +00:00 коммит произвёл GitHub
Родитель edcad79b8d
Коммит cc6e46f98d
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
3 изменённых файлов: 16 добавлений и 25 удалений

Просмотреть файл

@ -286,7 +286,7 @@ repos:
entry: "./scripts/ci/pre_commit_mypy.sh"
files: \.py$
exclude: ^airflow/_vendor/.*$
pass_filenames: true
pass_filenames: false
- id: pylint
name: Run pylint for main sources
language: system

Просмотреть файл

@ -708,30 +708,21 @@ function run_check_license() {
function run_mypy() {
FILES=("$@")
if [[ "${#FILES[@]}" == "0" ]]; then
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
--entrypoint "/usr/local/bin/dumb-init" \
--env PYTHONDONTWRITEBYTECODE \
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
--env AIRFLOW_CI_SILENT \
--env HOST_USER_ID="$(id -ur)" \
--env HOST_GROUP_ID="$(id -gr)" \
--rm \
"${AIRFLOW_CI_IMAGE}" \
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "airflow" "tests" "docs" \
| tee -a "${OUTPUT_LOG}"
else
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
--entrypoint "/usr/local/bin/dumb-init" \
--env PYTHONDONTWRITEBYTECODE \
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
--env AIRFLOW_CI_SILENT \
--env HOST_USER_ID="$(id -ur)" \
--env HOST_GROUP_ID="$(id -gr)" \
--rm \
"${AIRFLOW_CI_IMAGE}" \
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "${FILES[@]}" \
| tee -a "${OUTPUT_LOG}"
FILES=(airflow tests docs)
fi
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
--entrypoint "/usr/local/bin/dumb-init" \
--env PYTHONDONTWRITEBYTECODE \
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
--env AIRFLOW_CI_SILENT \
--env HOST_USER_ID="$(id -ur)" \
--env HOST_GROUP_ID="$(id -gr)" \
"-v" "${AIRFLOW_SOURCES}/.mypy_cache:/opt/airflow/.mypy_cache" \
--rm \
"${AIRFLOW_CI_IMAGE}" \
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "${FILES[@]}" \
| tee -a "${OUTPUT_LOG}"
}
function run_pylint_main() {

Просмотреть файл

@ -33,7 +33,7 @@ print_in_container_info "Running mypy with parameters: $*"
print_in_container_info
print_in_container_info
mypy --cache-dir=/dev/null "$@"
mypy "$@"
RES="$?"