[AIRFLOW-XXXX] Speed up mypy runs. (#7421)
This PR does two things: 1. It enables the mypy cache (default folder name .mypy_cache) so that repeated runs locally are quicker 2. It _disables_ passing only the changed files in. Point 2 seems counter-intuitave, but in my testing running with all files (airflow docs tests) was about twice as fast as without. My hypothesis for why this happens is that when mypy is checking file x, it has to check dependencies/imports for it too, and when we have pass_filenames set runs multiple processes in parallel, and each of them have to do this work! Timings before and after: - Before: For all files ``` ❯ time pre-commit run mypy -a Run mypy.................................................................Passed pre-commit run mypy -a 0.31s user 0.07s system 2% cpu 17.140 total ``` With only a single file ``` ❯ time pre-commit run mypy --files airflow/configuration.py Run mypy.................................................................Passed pre-commit run mypy --files airflow/configuration.py 0.30s user 0.07s system 5% cpu 6.724 total ``` - After: With a clean cache (`rm -rf .mypy_cache`): ``` $ time pre-commit run mypy Run mypy.................................................................Passed pre-commit run mypy -a 0.26s user 0.10s system 2% cpu 17.226 total ``` Clean cache with single file: ``` $ time pre-commit run mypy --file airflow/version.py Run mypy.................................................................Passed pre-commit run mypy --file airflow/version.py 0.23s user 0.07s system 4% cpu 7.091 total ``` Repeated run (cache folder exists): ``` $ time pre-commit run mypy --file airflow/version.py Run mypy.................................................................Passed pre-commit run mypy --file airflow/version.py 0.23s user 0.05s system 6% cpu 4.178 total ``` and for all files ``` airflow ❯ time pre-commit run mypy -a Run mypy.................................................................Passed pre-commit run mypy -a 0.25s user 0.09s system 6% cpu 4.833 total ```
This commit is contained in:
Родитель
edcad79b8d
Коммит
cc6e46f98d
|
@ -286,7 +286,7 @@ repos:
|
|||
entry: "./scripts/ci/pre_commit_mypy.sh"
|
||||
files: \.py$
|
||||
exclude: ^airflow/_vendor/.*$
|
||||
pass_filenames: true
|
||||
pass_filenames: false
|
||||
- id: pylint
|
||||
name: Run pylint for main sources
|
||||
language: system
|
||||
|
|
|
@ -708,30 +708,21 @@ function run_check_license() {
|
|||
function run_mypy() {
|
||||
FILES=("$@")
|
||||
if [[ "${#FILES[@]}" == "0" ]]; then
|
||||
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
|
||||
--entrypoint "/usr/local/bin/dumb-init" \
|
||||
--env PYTHONDONTWRITEBYTECODE \
|
||||
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
|
||||
--env AIRFLOW_CI_SILENT \
|
||||
--env HOST_USER_ID="$(id -ur)" \
|
||||
--env HOST_GROUP_ID="$(id -gr)" \
|
||||
--rm \
|
||||
"${AIRFLOW_CI_IMAGE}" \
|
||||
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "airflow" "tests" "docs" \
|
||||
| tee -a "${OUTPUT_LOG}"
|
||||
else
|
||||
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
|
||||
--entrypoint "/usr/local/bin/dumb-init" \
|
||||
--env PYTHONDONTWRITEBYTECODE \
|
||||
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
|
||||
--env AIRFLOW_CI_SILENT \
|
||||
--env HOST_USER_ID="$(id -ur)" \
|
||||
--env HOST_GROUP_ID="$(id -gr)" \
|
||||
--rm \
|
||||
"${AIRFLOW_CI_IMAGE}" \
|
||||
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "${FILES[@]}" \
|
||||
| tee -a "${OUTPUT_LOG}"
|
||||
FILES=(airflow tests docs)
|
||||
fi
|
||||
|
||||
docker run "${AIRFLOW_CONTAINER_EXTRA_DOCKER_FLAGS[@]}" \
|
||||
--entrypoint "/usr/local/bin/dumb-init" \
|
||||
--env PYTHONDONTWRITEBYTECODE \
|
||||
--env AIRFLOW_CI_VERBOSE="${VERBOSE}" \
|
||||
--env AIRFLOW_CI_SILENT \
|
||||
--env HOST_USER_ID="$(id -ur)" \
|
||||
--env HOST_GROUP_ID="$(id -gr)" \
|
||||
"-v" "${AIRFLOW_SOURCES}/.mypy_cache:/opt/airflow/.mypy_cache" \
|
||||
--rm \
|
||||
"${AIRFLOW_CI_IMAGE}" \
|
||||
"--" "/opt/airflow/scripts/ci/in_container/run_mypy.sh" "${FILES[@]}" \
|
||||
| tee -a "${OUTPUT_LOG}"
|
||||
}
|
||||
|
||||
function run_pylint_main() {
|
||||
|
|
|
@ -33,7 +33,7 @@ print_in_container_info "Running mypy with parameters: $*"
|
|||
print_in_container_info
|
||||
print_in_container_info
|
||||
|
||||
mypy --cache-dir=/dev/null "$@"
|
||||
mypy "$@"
|
||||
|
||||
RES="$?"
|
||||
|
||||
|
|
Загрузка…
Ссылка в новой задаче