Update PostgresHook's get_conn method to directly call the specified
conn_name_attr rather that always using self.postgres_conn_id.
Currently subclassing PostgresHook requires overriding the
postgres_conn_id attribute in order to establish a separate connection.
Add an additional unit test for this case checking that the subclassed
PostgresHook's get_conn calls the correct arguments and that the hook
calls the correction connection_id in get_connection.
This PR implements bulk_dump and bulk_load,
which are inherited from DbApiHook and
already implemented for MySqlHook.
Closes#3456 from sekikn/AIRFLOW-59
The previous commit on this issue (#3421)
introduced a new bug on COPY FROM operation.
This PR fixes it by opening a file with 'r+'
mode instead of 'w+' to avoid truncating it.
Closes#3423 from sekikn/AIRFLOW-2525-2
For now PostgresHook.copy_expert supports
"COPY TO" but not "COPY FROM", because it
opens a file with write mode and doesn't
commit operations. This PR fixes it by
opening a file with read and write mode
and committing operations at last.
- Changed single triple quotes to double quote
characters to be consistent with the docstring
convention in PEP 257
Closes#3396 from kaxil/AIRFLOW-2502
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow 2017]
(https://issues.apache.org/jira/browse/AIRFLOW-201
7/) issues and references them in the PR title.
For example, "[AIRFLOW-2017] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2017
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Currently we're not getting the output logs of the
postgres operator that you would get otherwise if
you ran a psql command. It's because the postgres
conn has an attribute called [notices](http://init
d.org/psycopg/docs/connection.html#connection.noti
ces) which contains this information.
We need to just print the results of this to get
that output in the airflow logs, which makes it
easy to debug amongst other things.
I've included some images for before and after
pictures.
**BEFORE**
<img width="1146" alt="screen shot 2018-01-19 at 4
46 59 pm" src="https://user-images.githubuserconte
nt.com/10408007/35178405-6f6a1da8-fd3d-11e7-8f50-0
dbd567d8ab4.png">
**AFTER**
<img width="1147" alt="screen shot 2018-01-19 at 4
46 25 pm" src="https://user-images.githubuserconte
nt.com/10408007/35178406-74ea4ae6-fd3d-11e7-9551-6
31eac6bfe7b.png">
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
There isn't anything to test, there is nothing
changing to the current implementation besides an
addition of logging.
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes#2959 from Acehaidrey/AIRFLOW-2017
Executes SQL using psycopg2 copy_expert method
Necessary to execute COPY command without access to a superuser
Closes#2698 from andyxhadji/AIRFLOW-1726
Controls the number of seconds of inactivity after
which TCP
should send a keepalive message to the server.
A value of zero uses the system default.
Important for Redshift which requires a setting
lower than 300.
Closes#2650 from bolkedebruin/AIRFLOW-1688
Instead of creating a sql statement with all
values, we send the values
separately to prevent sql injection
Closes#2270 from NielsZeilemaker/AIRFLOW-1170
The server-side autocommit setting was removed and reimplemented
in client applications and languages. Server-side autocommit was
causing too many problems with languages and applications that
wanted to control their own autocommit behavior,
so autocommit was removed from the server and added to individual client APIs as appropriate
Closes#1821 from danielzohar/AIRFLOW-
139_vacuum_operator
Kamil Breguła
dstandish
MinJae Kwon
Kevin Yang
Cooper Gillan
Andrew Harmon
Chao-Han Tsai
Kengo Seki
Kaxil Naik
Bolke de Bruin
Ace Haidrey
Andy Hadjigeorgiou
Niels Zeilemaker
Richard Lee
Ján Koščo
Daniel Zohar
John Bodley
gtoonstra
jlowin
Bence Nagy
Zack Shapiro
Maxime Beauchemin
Steve Mardenfeld
Maxime
Mike Kerzhner