This is a complete refactor of the setup.py providers/dependencies.
It much better reflects the current setup where we have most of
the extras 1-1 reflecting providers but also some extras that do
not have their own providers.
The pre-commits that were verifying setup versus documentation
can now be vastly simplified (no more need to parse the
comments so we can import setup.py variables directly rather
than parse it via regexps. Also we can better categorize the
extras - separate out (and verify) whether we correctly
described deprecated extras and to mark extras that install
additional providers as such.
Fixes: #13309
This change introduces improvements in the way logs are displayed
in CI jobs and in amount of logs produced in general for CI jobs
due to much smarter cache usage.
Logs in all CI jobs are now grouped in groups which are folded
by default when there is no error generated in such group. Similar
solution has been already used in docs job and it improved
both readability and speed of loading of the logs in CI after
recent improvements in Github UI (previously the speed of loading
the logs was not improved by groups).
Also cache usage has been reviewed and fixed in a number of places
which will result in much shorted setup times for static checks
and kubernetes virtualenv but also far shorter logs generated by
cache setup (we are using restore-keys feature that implements
incremental approach for cache building even if cache keys in
GitHub Actions are immutable.
* Changes release image preparation to use PyPI packages
Since we released all teh provider packages to PyPI now in
RC version, we can now change the mechanism to prepare the
production to use released packages in case of tagged builds.
The "branch" production images are still prepared using the
CI images and .whl packages built from sources, but the
release packages are built from officially released PyPI
packages.
Also some corrections and updates were made to the release process:
* the constraint tags when RC candidate is sent should contain
rcn suffix.
* there was missing step about pushing the release tag once the
release is out
* pushing tag to GitHub should be done after the PyPI packages
are uploaded, so that automated image building in DockerHub
can use those packages.
* added a note that in case we will release some provider
packages that depend on the just released airflow version
they shoudl be released after airflow is in PyPI but before
the tag is pushed to GitHub (also to allow the image to be
build automatically from the released packages)
Fixes: #12970
* Update dev/README_RELEASE_AIRFLOW.md
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Update dev/README_RELEASE_AIRFLOW.md
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Setuptools has a built in mechanism for adding a `suffix` to a version,
so we don't have to edit a file and then remember to delete it!
It's a little bit messy to get the suffix like this -- using sed to
strip out any leading digits or periods, but it's copy-pasteable this
way.
As discussed in AIP-21
* Rename airflow.sensors.external_task_sensor to airflow.sensors.external_task
* Rename airflow.sensors.sql_sensor to airflow.sensors.sql
* Rename airflow.contrib.sensors.weekday_sensor to airflow.sensors.weekday
As discussed in AIP-21
* Rename airflow.hooks.base_hook to airflow.hooks.base
* Rename airflow.hooks.dbapi_hook to airflow.hooks.dbapi
* Rename airflow.sensors.base_sensor_operator to airflow.sensors.base
* Rename airflow.sensors.date_time_sensor to airflow.sensors.date_time
* Rename airflow.sensors.time_delta_sensor to airflow.sensors.time_delta
Co-authored-by: Kaxil Naik <kaxilnaik@apache.org>
So far, the production images of Airflow were using sources
when they were built on CI. This PR changes that, to build
airflow + providers packages first and install them
rather than use sources as installation mechanism.
Part of #12261
This PR implements discovering and readin provider information from
packages (using entry_points) and - if found - from local
provider yaml files for the built-in airflow providers,
when they are found in the airflow.provider packages.
The provider.yaml files - if found - take precedence over the
package-provided ones.
Add displaying provider information in CLI
Closes: #12470
* Make the KubernetesPodOperator backwards compatible
This PR significantly reduces the pain of upgrading to Airflow 2.0
for users of the KubernetesPodOperator. Users will be allowed to
continue using the airflow.kubernetes custom classes
* spellcheck
* spelling
* clean up unecessary files in 1.10
* clean up unecessary files in 1.10
* clean up unecessary files in 1.10
The inbuilt functions all() and any() in python also support
short-circuiting (evaluation stops as soon as the overall return value
of the function is known), but this behavior is lost if you use
comprehension. This affects performance.
If you have more than a single private key in your GPG trust store, gpg
will use the first one, which for me is not right.
This changes the script to by default use any key with `apache.org` in
the name. This is a patch I've been carrying locally for about 8
releases now :D
Temporary fix to unblock master PRs.
There is a problem with comparing/fetching commits during tests
and it should be investigated properly - this is temporary fix
to workaround it.
Kaxil Naik
Jarek Potiuk
John Bampton
Ash Berlin-Taylor
Kamil Breguła
Kishore Vancheeshwaran
Tomek Urbaszek
Daniel Imberman