All files are mounted in CI now and checked using the RAT tool.
As opposed to only the runtime-needed files. This is enabled for CI
build only as mounting all local files to Docker (especially on Mac)
has big performance penalty when running the checks (slow osxfs
volume and thousands of small node_modules files generated make the
check runs for a number of minutes). The RAT checks will by default
use the selective volumes but on CI they will mount the whole
source directory.
Also latest version of RAT tool is used now and the output - list
of checked files - is additionally printed as output of the RAT
check so that we are sure the files we expect to be there, are
actually verified.
If you execute the scripts/ci/6-check-license.sh script in a running airflow env it checks a few more files that don't need to be checked.
This includes logs, unittests.cfg, _api (from docs) and sources (from www).
nvd3 has a dependency on python-slugify which pulls in a
GPL dependency by default, which we don't want.
This commit brings in nvd3 0.15.0 and slugify 2.0.1 WITH NO CHANGES -
those will come in the next commit
* Refactor Kubernetes operator with git-sync
Currently the implementation of git-sync is broken because:
- git-sync clones the repository in /tmp and not in airflow-dags volume
- git-sync add a link to point to the revision required but it is not
taken into account in AIRFLOW__CORE__DAGS_FOLDER
Dags/logs hostPath volume has been added (needed if airflow run in
kubernetes in local environment)
To avoid false positive in CI `load_examples` is set to `False`
otherwise DAGs from `airflow/example_dags` are always loaded. In this
way is possible to test `import` in DAGs
Remove `worker_dags_folder` config:
`worker_dags_folder` is redundant and can lead to confusion.
In WorkerConfiguration `self.kube_config.dags_folder` defines the path of
the dags and can be set in the worker using airflow_configmap
Refactor worker_configuration.py
Use a docker container to run setup.py
Compile web assets
Fix codecov application path
* Fix kube_config.dags_in_image
This adds ASF license headers to all the .rst and .md files with the
exception of the Pull Request template (as that is included verbatim
when opening a Pull Request on Github which would be messy)
JIRA:
https://issues.apache.org/jira/browse/AIRFLOW-1291
* Update NOTICE with proper year range for ASF
copyright
* Break down LICENSE into
licenses/LICENSE-[project].txt
* add license header to jqClock.min.js
[AIRFLOW-1291] Update NOTICE and LICENSE files to
match ASF requirements
* Update NOTICE with proper year range for ASF
copyright
* Break down LICENSE into
licenses/LICENSE-[project].txt
* add license header to jqClock.min.js
fix license check
Closes#2354 from
mistercrunch/copyright_license_touchups
Per Apache requirements Airflow should be branded
Apache Airflow.
It is impossible to provide a forward compatible
automatic update
path and users will be required to manually
upgrade.
Closes#2172 from bolkedebruin/AIRFLOW-1000
This implements a framework for API calls to Airflow. Currently
all access is done by cli or web ui. Especially in the context
of the cli this raises security concerns which can be alleviated
with a secured API call over the wire.
Secondly integration with other systems is a bit harder if you have
to call a cli. For public facing endpoints JSON is used.
As an example the trigger_dag functionality is now made into a
API call.
Backwards compat is retained by switching to a LocalClient.
Currently dags are being read directly from the filesystem. Any
hierarchy (python namespaces, modules) need to be reflected on
the filesystem. This makes it hard to manage dags and their
depedencies.
This patch adds support for dags in zip files. It will add
the zip to sys.path and then it will read the zip file and
try to import any files as modules that are in the root of
the zip.
Please note that any module contained within the zip will
overwrite existing modules in the same namespace.
Jarek Potiuk
Bas Harenslak
Felix Uellendall
Bolke de Bruin
Tao Feng
Riccardo Bini
Ash Berlin-Taylor
Verdan Mahmood
Moe Nadal
Maxime Beauchemin
bolkedebruin