We have fairly complex python version detection in our CI scripts.
They have to handle several cases:
1) Running builds on DockerHub (we cannot pass different environment
variables there, so we detect python version based on the image
name being build (airflow:master-python3.7 -> PYTHON_VERSION=3.7)
2) Running builds on Travis CI. We use python version determined
from default python3 version available on the path. This way we
do not have to specify PYTHON_VERSION separately in each job,
we just specify which host python version is used for that job.
This makes a nice UI experience where you see python version in
Travis UI.
3) Running builds locally via scripts where we can pass PYTHON_VERSION
as environment variable.
4) Running builds locally for the first time with Breeze. By default
we determine the version based on default python3 version we have
in the host system (3.5, 3.6 or 3.7) and we use this one.
5) Selecting python version with Breeze's --python switch. This will
override python version but it will also store the last used version
of python in .build directory so that it is automatically used next
time.
This change adds necessary explanations to the code that works for
all the cases and fixes some of the edge-cases we had. It also
extracts the code to common directory.
The fuzzy licence matching implemented by Jarek Potiuk
was accepted and merged by Lucas-C in his pre-commit
hooks implementation (released today ver. 1.1.7)
so we can switch back to it.
Licence check for RAT runs too often (every time pre-commit is modified) and it
should only be run (locally) when any of *LICEN[S|C]E* files change. We anyhow
run full check on CI so this is local optimisation (it runs too long while you
play with .pre-commit-config.yaml - and we will probably be able to detect some
problems locally as well when new modules are added.
Jarek Potiuk
Jarek Potiuk