By default one of Apache Airflow's dependencies pulls in a GPL
library. Airflow should not install (and upgrade) without an explicit choice.
This is part of the Apache requirements as we cannot depend on Category X
software.
This PR adds a sensor for MongoDB,
which waits for some document that
matches the given query to be
inserted to the specified collection.
Closes#3611 from sekikn/AIRFLOW-2758
[AIRFLOW-2424] Add dagrun status endpoint and
increase k8s test coverage
[AIRFLOW-2424] Added minikube fixes by @kimoonkim
[AIRFLOW-2424] modify endpoint to remove 'status'
Closes#3320 from dimberman/add-kubernetes-test
[AIRFLOW-1899] Add full deployment
- Made home directory configurable
- Documentation fix
- Add licenses
[AIRFLOW-1899] Tests for the Kubernetes Executor
Add an integration test for the Kubernetes
executor. Done by
spinning up different versions of kubernetes and
run a DAG
by invoking the REST API
Closes#3301 from Fokko/fix-kubernetes-executor
Make sure you have checked _all_ steps below.
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2335
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
- [x] Here are some details about my PR, including
screenshots of any UI changes:
There is an issue with travis pulling jdk8 that is
preventing CI jobs from running. This blocks
further development of the project.
Reference: https://github.com/travis-ci/travis-
ci/issues/9512#issuecomment-382235301
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
This PR can't be unit tested since it is just
configuration. However, the fact that unit tests
run successfully should show that it works.
- [ ] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
- [ ] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
- [ ] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes#3236 from dimberman/AIRFLOW-
2335_travis_issue
sla_miss and task_instances cannot have NULL
execution_dates. The timezone
migration scripts forgot to set this properly. In
addition to make sure
MySQL does not set "ON UPDATE CURRENT_TIMESTAMP"
or MariaDB "DEFAULT
0000-00-00 00:00:00" we now check if
explicit_defaults_for_timestamp is turned
on and otherwise fail an database upgrade.
Closes#2969, #2857Closes#2979 from bolkedebruin/AIRFLOW-1895
The celery config is currently part of the celery executor definition.
This is really inflexible for users wanting to change it. In addition
Celery 4 is moving to lowercase.
Closes#2542 from bolkedebruin/upgrade_celery
Submitting on behalf of plypaul
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-219
-
https://issues.apache.org/jira/browse/AIRFLOW-398
Testing Done:
- Running on Airbnb prod (though on a different
mergebase) for many months
Credits:
Impersonation Work: georgeke did most of the work
but plypaul did quite a bit of work too.
Cgroups: plypaul did most of the work, I just did
some touch up/bug fixes (see commit history,
cgroups + impersonation commit is actually plypaul
's not mine)
Closes#1934 from aoen/ddavydov/cgroups_and_impers
onation_after_rebase
This implements a framework for API calls to Airflow. Currently
all access is done by cli or web ui. Especially in the context
of the cli this raises security concerns which can be alleviated
with a secured API call over the wire.
Secondly integration with other systems is a bit harder if you have
to call a cli. For public facing endpoints JSON is used.
As an example the trigger_dag functionality is now made into a
API call.
Backwards compat is retained by switching to a LocalClient.
Travis' ubuntu trusty provides a more up to date
environment for CI. It allows for better testing
by integration more services like kerberos and
celery. Also it comes closer to actual production
environments (e.g. MySQL 5.6).
- Added Apache license header for files with extension (.service, .in, .mako, .properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, .html, .xml.
- Added/Replaced shebang on all .sh files with portable version - #!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.
Closes#1598 from ajayyadava/248
- added Travis config to also run integration tests on sqlite
- added Travis config to also run integration test on postgre, and commented them out as they seem to fail
- added a bit more logs in bash scripts executed by Travis
bolkedebruin
Fokko Driesprong
Kengo Seki
Daniel Imberman
Fokko Driesprong
Bolke de Bruin
Benjamin Goldberg
grantnicholas
GRANT NICHOLAS
Dan Davydov
Ajay Yadav
Svend Vanderveken
Maxime Beauchemin