Replace sensor by operator in the comment:
`:param http_conn_id: The connection to run the
sensor against` by `:param http_conn_id: The
connection to run the operator against`
Closes#3544 from thibaultclem/patch-1
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2661
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Changes the use of `log_volume_subpath` and
`dags_volume_subpath` which are
now passed into the construction of the worker
pod's volumeMounts instead of
the volume section (where subPath is not valid).
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
Unit tests have been added but I'm not sure how to
add integration tests
for this without breaking the other minikube tests
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
No new functionality added
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes#3537 from r4vi/AIRFLOW-2661
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2652
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
When running ``airflow backfill`` on
pythonOperator, it will do / trigger a deepcopy of
the task_instance. If some objects can't be
deepcopy in certain python version(e.g Protobuf in
python 2.7) , an exception will be thrown. We
should just do a shallow copy instead of deep copy
for the object.
The pr here is to copy the ``_deepcopy__`` method
in BaseOperator, but skip doing deepcopy for
`op_kwargs` and `python_callable`.
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
I can't think of a good way to test. We encounter
this in our production.
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes#3528 from feng-tao/airflow-2652
* Get failed attempts from `task_fail` table and show them in Gantt view.
* Fix TaskFail class column definition to match actual database schema.
Change was required to retrieve all records for one task instance run
otherwise only one was returned by SQLAlchemy.
Set worker_container_image_pull_policy in default_airflow.cfg
As AIRFLOW-2617 added worker_container_image_pull_policy config
to the section of kubernetes, but the airflow_default.cfg
was not updated, this PR add worker_container_image_pull_policy
to default_airflow.cfg.
Closes#3521 from imroc/AIRFLOW-2645
Just like a partition sensor for Hive,
this PR adds a sensor that waits for
a table to be created in Cassandra cluster.
Closes#3518 from sekikn/AIRFLOW-2640
setup.py has a logic to avoid installing
snakebite on Python3, but it doesn't work.
This is because the variable devel_all is
now a tuple of lists. This PR fixes
that variable to be a flat list and
makes the logic work as expected.
Closes#3522 from sekikn/AIRFLOW-2646
There is several HiveServer2-related tests
which are not run by default in
tests/operators/hive_operator.py,
but AIRFLOW-2534 implemented almost all of them
in airflow/hooks/hive_hooks.py.
This PR removes duplicated tests from
tests/operators/hive_operator.py and
moves an unimplemented test to
airflow/hooks/hive_hooks.py with a bit of fix.
Closes#3498 from sekikn/AIRFLOW-2612
Sometimes, it's desirable to be able to use
REPLACE INTO instead of
INSERT INTO for insert_rows method (if importing
the same data multiple
times).
This adds an optional parameter to the insert_rows
column that flips the
generated sql statement from "INSERT INTO" to
"REPLACE INTO".
Closes#3517 from flokli/dbapi_hook-replace
- Improved the retries times to jobs below 60s
- Renamed property queue to job_queue to prevent
AWS Batch and CeleryExecutor queue conflict
- Added Breaking Chain note for the UPDATING.md
master
- Fixed operator infinit loop
- Added documentation warning about the Breaking
chain
- Fixed the commit parameter to keep it on Airflow
guidelines
- Fixed logging typo
- rebased with master
Changes to be committed:
modified: ../../../UPDATING.md
modified: awsbatch_operator.py
modified: ../../../tests/contrib/operators/test_
awsbatch_operator.py
Closes#3436 from hprudent/master
* Add test that verifies that database schema and SQLAlchemy model are in sync
* Add exception for users.password that doesn't exist in model and tables created by other tests
* Add migration script to merge the two heads
* Add migration script to fix not-null constrains for MySQL that were lost by 0e2a74e0fc9f_add_time_zone_awareness
* Add migration script to fix FK constraint for existing SQLite DBs
* Enable ForeignKey support for SQLite, otherwise 2e82aab8ef20_rename_user_table won't change FK in chart and known_event tables
Soft-fail sensor failure causes skip of all
downstream tasks. It also enables ability to set
up non-blocking and soft-fail sensors in the same
way as for regular sensors.
Closes#3509 from artem-kirillov/AIRFLOW-1786
Parameters passed through airflow trigger_dag -c
'{"text": "blah"}' can be accessed through
{{ dag_run.conf.text }} in subdag.
Closes#3460 from milton0825/trigger-dag-subdag
When Airflow was populating a DagBag from a .zip
file, if a single
file in the root directory did not contain the
strings 'airflow' and
'DAG' it would ignore the entire .zip file.
Also added a small amount of logging to not
bombard user with info
about skipping their .py files.
Closes#3505 from Noremac201/dag_name
There are two way of syncing dags, pvc and git-
sync, if set both
(dags_volume_claim and git_subpath), I won't get
the mountPath what I
want. I think the priority of pvc should higher
than git-sync, so I
think when dags_volume_claim is been set, the
mountPath shuold not join the git_subpath.
Closes#3497 from imroc/AIRFLOW-2611
Add Google Kubernetes Engine create_cluster,
delete_cluster operators
This allows users to use airflow to create or
delete clusters in the
google cloud platform
Closes#3477 from Noremac201/gke_create
This commit also adds numerous tests for
HiveServer2 and switches
Impyla for PyHive (0.6.0), making HiveServer2
Python 2 compatible.
Closes#3432 from gglanzani/AIRFLOW-2534
Kaxil Naik
niels
Joshua Iverson
Kevin Yang
pengchen
Thibault Clement
Cameron Moberg
Joy Gao
Stefan Seelmann
Ravi Kotecha
Kaxil Naik
Tao feng
roc
Kengo Seki
Christian Trebing
Verdan Mahmood
Florian Klink
Hugo Prudente
Shintaro Murakami
r39132
Artem Kirillov
milton0825
Giovanni Lanzani
djordje