Граф коммитов

43 Коммитов

Автор SHA1 Сообщение Дата
Jarek Potiuk 070e3c3364
Tests are working for newly added backport providers (#9739)
* Tests are working for newly added backport providers
2020-07-09 20:13:58 +02:00
Jarek Potiuk 44d4ae809c
Upgrade to latest pre-commit checks (#9686) 2020-07-06 11:37:22 +02:00
Kaxil Naik 87fdbd0708
Use literal syntax instead of function calls to create data structure (#9516)
It is slower to call e.g. dict() than using the empty literal, because the name dict must be looked up in the global scope in case it has been rebound. Same for the other two types like list() and tuple().
2020-06-25 16:35:37 +01:00
Jarek Potiuk ec0025f35b
Add stats to backport packages (#9501) 2020-06-25 10:02:23 +02:00
Kaxil Naik 57b495caa4
Add PyDocstyle Precommit Hook (#9456) 2020-06-21 09:34:41 +01:00
Kaxil Naik d5262c01a9
Remove unused variables in backport_packages (#9425)
There were few instance where we didn't need all the variables from the returned functions. We can remove them.
2020-06-20 10:38:52 +02:00
Jarek Potiuk d0e7db4024
Fixed release number for fresh release (#9408) 2020-06-19 15:56:22 +02:00
Tomek Urbaszek 416334e2ec
Properly propagated warnings in operators (#9348)
* Test warnings are properly propagated
* Adjust deprecation warnings
* Separate tests and deprecated classes lists
2020-06-19 14:29:33 +02:00
Jarek Potiuk c7e5bce57f
Prepare backport release candidate for 2020.6.23rc1 (#9370)
* Prepare backport release candidate for 2020.6.23rc1

* fixup! Prepare backport release candidate for 2020.6.23rc1

* fixup! fixup! Prepare backport release candidate for 2020.6.23rc1
2020-06-19 10:07:06 +02:00
Jarek Potiuk f6bd817a3a
Introduce 'transfers' packages (#9320)
* Consistent naming of transfer operators

Transfer operators have consistent names and are grouped in
the 'transfer' packages.

* fixup! Consistent naming of transfer operators

* Introduces 'transfers' packages.

Closes #9161 and #8620

* fixup! Introduces 'transfers' packages.

* fixup! fixup! Introduces 'transfers' packages.

* fixup! fixup! fixup! Introduces 'transfers' packages.
2020-06-16 22:55:42 +02:00
Kaxil Naik d066d33062
Remove trailing comma in setup_backport_packages.py (#9284) 2020-06-14 12:15:31 +01:00
Jarek Potiuk 92d68807f3
Split utils sh (#9132)
* Split _utils.sh into separate files
2020-06-04 22:12:56 +02:00
Jarek Potiuk 05e4c39eb6
Provide_context coma is added only when there is not one already (#9064) 2020-05-30 17:37:51 +02:00
Jarek Potiuk 0b0e4f7a4c
Preparing for RC3 relase of backports (#9026) 2020-05-26 21:35:06 +02:00
Jarek Potiuk 00642a46d0
Fixed name of 20 remaining wrongly named operators. (#8994) 2020-05-26 19:12:21 +02:00
Jarek Potiuk 9764c90b92
Better content of backport packages CHANGELOG and INSTALL files (#9013)
* Better content of backport packages CHANGELOG and INSTALL files

The content of Backport Packages CHANGELOG.txt and INSTALL files
has been updated to reflect that those are not full Airflow
releases.

1) Source package:
- INSTALL contains only references to preparing backport packages
- CHANGELOG.txt contains combined change log of all the packages

2) Binary packages:
- No INSTALL
- CHANGELOG.txt contains changelog for this package only

3) Whl packages

No change

* Update backport_packages/INSTALL
2020-05-26 13:54:58 +02:00
Jarek Potiuk cdb3f25456
All classes in backport providers are now importable in Airflow 1.10 (#8991)
* All classes in backport providers are now importable in Airflow 1.10

* fixup! All classes in backport providers are now importable in Airflow 1.10

* fixup! fixup! All classes in backport providers are now importable in Airflow 1.10
2020-05-26 02:43:16 +02:00
Jarek Potiuk 375d1ca229
Release candidate 2 for backport packages 2020.05.20 (#8898)
Release candidate 2 for backport packages 2020.05.20
2020-05-19 14:17:22 +02:00
Jarek Potiuk 12c5e5d8ae
Prepare release candidate for backport packages (#8891)
After preparing the 2020.5.19 release candidate and
reviewing the packages, some changes turned out to be necessary.

Therefore the date was changed to 2020.5.20 with the folowing
fixes:

* cncf.kubernetes.example_dags were hard-coded and added for all
  packagesa and they were removed
* Version suffix is only used to rename the binary packages not for
  the version itself
* Release process description is updated with the release process
* Package version is consistent - leading 0s are skipped in month
  and day
2020-05-17 20:38:46 +02:00
Jarek Potiuk a3a3411838
Fix master failing on generating requirements (#8885)
By default github actions checks out only latest commit but in order to
see if there are any changes since the last readme generated
we need to see the whole history so we need to fetch it all.

We also skip generating the new README in case there is only one
commit in the history since the last release. The nature of readme
generation is that the commit with the README itself will never
be in the list of commits for the previous release so there is
always at least one commit more than the one listed in the readme.
2020-05-16 12:53:30 +02:00
Jarek Potiuk 92585ca4cb
Added automated release notes generation for backport operators (#8807)
We have now mechanism to keep release notes updated for the
backport operators in an automated way.

It really nicely generates all the necessary information:

* summary of requirements for each backport package
* list of dependencies (including extras to install them) when package
  depends on other providers packages
* table of new hooks/operators/sensors/protocols/secrets
* table of moved hooks/operators/sensors/protocols/secrets with
  information where they were moved from
* changelog of all the changes to the provider package (this will be
  automatically updated with incremental changelog whenever we decide to
  release separate packages.

The system is fully automated - we will be able to produce release notes
automatically (per-package) whenever we decide to release new version of
the package in the future.
2020-05-15 19:00:15 +02:00
Kaxil Naik 4b06fde0f1
Fix Flake8 errors (#8841) 2020-05-12 11:07:29 +01:00
Jarek Potiuk 1fb9f0722a
Synchronize extras between airflow and providers (#8819) 2020-05-11 19:25:15 +02:00
Jarek Potiuk 791d1a786f
Backport packages are renamed to include backport in their name (#8767) 2020-05-09 14:09:12 +02:00
Felix Uellendall ff5b70149b
Add google_api_to_s3_transfer example dags and system tests (#8581)
- add amazon system helper for easier testing amazon aws systems / services
- fix TESTING docs
2020-05-07 09:32:29 +02:00
Jarek Potiuk 3c4ca62af1
Building backport packages is now done inside Breeze container (#8558)
This way you do not have to worry about setting up your environment.

Fixes #8537
2020-04-26 12:35:27 +02:00
S S Rohit eee4ebaeeb
Added Facebook Ads Operator #7887 (#8008) 2020-04-14 15:08:35 +02:00
Pieter Mulder 69dc91b4ef
[AIRFLOW-6982] add native python exasol support (#7621)
* [AIRFLOW-6982] add native python exasol support

This adds exasol DB support, including a hook, connection type &
operator. The [pyexasol](https://github.com/badoo/pyexasol) library is
used to interact with the database.

* Add exasol to EXTRAS documentation

* Add exasol requirements to requirements files

* Add exasol to backport packages setup

Co-authored-by: Jan Omar <jan.omar@wooga.net>
2020-04-02 11:53:20 -07:00
Roi Teveth 1ccafc617c
Add spark_kubernetes system test (#7875)
fix reformat

fix default connection sort
2020-04-02 11:43:35 -07:00
Tomek Urbaszek 8e89780071
Add more refactor steps for providers.google (#8010)
* Add more refactor steps for providers.google

* fixup! Add more refactor steps for providers.google
2020-03-31 14:06:12 +02:00
Jarek Potiuk aae3b8fb27
Individual package READMEs (#8012) 2020-03-31 08:36:10 +02:00
Tomek Urbaszek 0f19a930d1
Remove GKEStartPodOperator when backporting (#7908) 2020-03-29 14:10:06 +02:00
Kamil Breguła 02b71f9e8e
Remove unused import - BowlerTool (#7865) 2020-03-25 13:56:55 +01:00
Kamil Breguła 6df6b8c9cc
Fix example DAG for MLEngine in backport package (#7813) 2020-03-25 10:15:51 +01:00
Tomek Urbaszek 3cc37b1fed
Remove super().__init__() call in backported hooks (#7833) 2020-03-23 22:35:06 +01:00
Jarek Potiuk 87149a791f
Test installation of all provider packages (#7797) 2020-03-22 20:52:15 +01:00
Jarek Potiuk 305e1e535e
Verify that all providers packages are included (#7793) 2020-03-22 16:58:38 +01:00
Bas Harenslak 5f784ae5c0
[AIRFLOW-7061] Rename openfass to openfaas (#7721) 2020-03-14 10:47:50 +01:00
Jarek Potiuk 462225a054
[AIRFLOW-7041] make bowler dependency local (#7691) 2020-03-11 13:25:31 +01:00
Tomek Urbaszek 1f77f943d5
[AIRFLOW-6980] Improve system tests and building providers package (#7615)
This PR removes initdb from system tests setup as it seems unneccessary operation.
Also some automatic code changes has been added before building providers package.

fixup! [AIRFLOW-6980] Improve system tests and building providers package

fixup! [AIRFLOW-6980] Improve system tests and building providers package

fixup! fixup! [AIRFLOW-6980] Improve system tests and building providers package
2020-03-10 14:05:08 +01:00
Jarek Potiuk 1766e0b03f
[AIRFLOW-XXXX] Fix gitignore (#7660) 2020-03-09 09:33:20 +01:00
Jarek Potiuk 42eef38217
[AIRFLOW-6877] Add cross-provider dependencies as extras (#7506)
Cross-provider dependencies are now extras in the backport
packages.
2020-03-07 20:55:27 +01:00
Jarek Potiuk ccb2899e5b
[AIRFLOW-6663] Prepare backporting packages (#7391) 2020-02-24 10:29:38 +01:00