The provider.yaml contains more information that required at
runtime (specifically about documentation building). Those
fields are not needed at runtime and their presence is optional.
Also the runtime check for provider information should be more
relexed and allow for future compatibility (with
additional properties set to false). This way we can add new,
optional fields to provider.yaml without worrying about breaking
future-compatibility of providers with future airflow versions.
This changei restores 'additionalProperties': false in the
main, development-focused provider.yaml schema and introduced
new runtime schema that is used to verify the provider info when
providers are discovered by airflow.
This 'runtime' version should change very rarely as change to
add a new required property in it breaks compatibility of
providers with already released versions of Airflow.
We also trim-down the provider.yaml file when preparing provider
packages to only contain those fields that are required in the
runtime schema.
Rather than having to mirror all the repos we can instead use
git submodules to pull in the third party actions we want to use - with
recent(ish) changes in review for submodules on GitHub we still get the
same "review/audit" visibility for changes, but this way we don't have
to either "pollute" our repo with the actions code, nor do we have to
maintain a fork of the third party action.
This is a complete refactor of the setup.py providers/dependencies.
It much better reflects the current setup where we have most of
the extras 1-1 reflecting providers but also some extras that do
not have their own providers.
The pre-commits that were verifying setup versus documentation
can now be vastly simplified (no more need to parse the
comments so we can import setup.py variables directly rather
than parse it via regexps. Also we can better categorize the
extras - separate out (and verify) whether we correctly
described deprecated extras and to mark extras that install
additional providers as such.
Fixes: #13309
Updates comments in setup.py according to the latest 2.0 agreements
about semver and clarifies use for arrays defined in setup.py.
We will refactor those after 2.0 further, but for now we keep
the current structure.
* Adds support for Hook discovery from providers
This PR extends providers discovery with the mechanism
of retrieving mapping of connections from type to hook.
Fixes#12456
* fixup! Adds support for Hook discovery from providers
* fixup! fixup! Adds support for Hook discovery from providers
* Providers in extras are properly configured and verified
This fixes#12255 - where we published beta2 release with some
extras pulling non-existing providers.
The exact list of providers that had problems:
Wrongly named extras/providers:
* apache.presto: it was badly named -> renamed to 'presto'
* spark (badly pointing to spark instead of apache.spark)
* yandexcloud (the name remains there but we've also added 'yandex' extra to correspond 1-1 with 'yandex' provider
Extras that were wrongly marked as having providers, where they had
none:
* dask
* rabbitmq
* sentry
* statsd
* tableau
* virtualenv
* Update scripts/ci/pre_commit/pre_commit_check_extras_have_providers.py
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Update scripts/ci/pre_commit/pre_commit_check_extras_have_providers.py
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Core example DAGs should not depend on any non-core dependency
like providers packages.
closes: #12247
Co-authored-by: Xiaodong DENG <xd.deng.r@gmail.com>
The change #10806 made airflow works with implicit packages
when "airflow" got imported. This is a good change, however
it has some unforeseen consequences. The 'provider_packages'
script copy all the providers code for backports in order
to refactor them to the empty "airflow" directory in
provider_packages folder. The #10806 change turned that
empty folder in 'airflow' package because it was in the
same directory as the provider_packages scripts.
Moving the scripts to dev solves this problem.
* Migrate from helm-unittest to python unittest
* fixup! Migrate from helm-unittest to python unittest
* fixup! fixup! Migrate from helm-unittest to python unittest
`shellcheck` is slow if a Bash Script is changed, hence it is moved down in the order.
The following pre-commits are quick so moved them up the order:
- pre-commit-descriptions
- sort-in-the-wild
- helm-lint
Jarek Potiuk
Ash Berlin-Taylor
Kaxil Naik
dstandish
John Bampton
Ruben Laguna
Sam Wheating
Kamil Breguła
Tomek Urbaszek
SZN
Daniel Imberman
Katsunori Kanda
Ryan Hamilton