SOURCE_BRANCH was set to TAG_NAME in Dockerhub. Now that we changed
DockerHub to build from tags, it caused AIRFLOW_BRANCH arg
to be set to master-nightly instead of master and we had a cache
miss.
SOURCE_BRANCH variable is not realy needed any more. Removed it
There is two parts to this PR:
1. Only copying www/webpack.config.js and www/static/ before running the
asset pipeline
2. Making sure that _all_ files (not just the critical ones) have the
same permissions.
There is two parts to this PR:
1. Only copying www/webpack.config.js and www/static/ before running the
asset pipeline
2. Making sure that _all_ files (not just the critical ones) have the
same permissions.
The goal of both of these is to make sure that the docker build cache for the "expensive"
operations (installing NPM modules, running asset pipeline, installing python modules)
isn't run when it isn't necessary.
It is:
- quicker to install
- easier to get repeatable results
- Takes up less space (130MB/15k files vs 190MB/23k files)
- nicer to user (has better help)
This change is a further step of simplifying the set of scripts
used by CI. The separate checklicence image was implemented as an
optimisation of the licence check time. The image to download was
small and could be downloaded slightly faster in CI. However that
made all the management script more complex and lead to having
separate jobs for check licence and static checks. That lead to
actually longer time of Travis jobs - because new machine had to
be spun-off for checklicence check only.
With this change, the CI image is the only one left and it is slightly
bigger (with RAT tool added) but the same image is used for all the
tests - unit tests, static checks and checklicence checks.
This also makes it easier to manage the images and decreases update
overhead on the developers using Breeze.
This is needed to keep breeze --help in sync with the documentation.
It makes it easier for the follow-up changes needed for production
image to keep the docs in sync with the code.
The detection of python version is complex because we need to handle
several cases - including determining the version from image name
on DockerHub, detecting python version from python in the environment,
finally forced from python version. This caused multiple problems
with Travis where we run tests with different version (auto-detected
from current python - especially when python3 became present in
Travis' python 2.7 images. Now all the jobs in Travis have
PYTHON_VERSION forced and the code responsible for detecting current
python version has been removed as it is not needed in this case.
We have fairly complex python version detection in our CI scripts.
They have to handle several cases:
1) Running builds on DockerHub (we cannot pass different environment
variables there, so we detect python version based on the image
name being build (airflow:master-python3.7 -> PYTHON_VERSION=3.7)
2) Running builds on Travis CI. We use python version determined
from default python3 version available on the path. This way we
do not have to specify PYTHON_VERSION separately in each job,
we just specify which host python version is used for that job.
This makes a nice UI experience where you see python version in
Travis UI.
3) Running builds locally via scripts where we can pass PYTHON_VERSION
as environment variable.
4) Running builds locally for the first time with Breeze. By default
we determine the version based on default python3 version we have
in the host system (3.5, 3.6 or 3.7) and we use this one.
5) Selecting python version with Breeze's --python switch. This will
override python version but it will also store the last used version
of python in .build directory so that it is automatically used next
time.
This change adds necessary explanations to the code that works for
all the cases and fixes some of the edge-cases we had. It also
extracts the code to common directory.
Jarek Potiuk
Ash Berlin-Taylor
Jarek Potiuk
bolkedebruin