Disables provider's manager warning for source-installed prod image. (#13729)

When production image is built for development purpose, by default
it installs all providers from sources, but not all dependencies
are installed for all providers. Many providers require more
dependencies and when you try to import those packages via
provider's manager, they fail to import and print warnings.

Those warnings are now turned into debug messages, in case
AIRFLOW_INSTALLATION_METHOD=".", which is set when
production image is built locally from sources. This is helpful
especially when you use locally build production image to
run K8S tests - otherwise the logs are flooded with
warnings.

This problem does not happe in CI, because there by default
production image is built from locally prepared packages
and it does not contain sources from providers that are not
installed via packages.
This commit is contained in:
Jarek Potiuk 2021-01-18 09:00:32 +01:00 коммит произвёл GitHub
Родитель 1ec63123c4
Коммит f74da5025d
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
2 изменённых файлов: 26 добавлений и 6 удалений

Просмотреть файл

@ -464,6 +464,11 @@ EXPOSE 8080
USER ${AIRFLOW_UID}
# Having the variable in final image allows to disable providers manager warnings when
# production image is prepared from sources rather than from package
ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
ARG BUILD_ID
ENV BUILD_ID=${BUILD_ID}
ARG COMMIT_SHA

Просмотреть файл

@ -289,12 +289,27 @@ class ProvidersManager:
self._add_customized_fields(provider_package, hook_class, field_behaviours)
except Exception as e: # noqa pylint: disable=broad-except
log.warning(
"Exception when importing '%s' from '%s' package: %s",
hook_class_name,
provider_package,
e,
)
if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
# print providers manager warning when airflow is not installed from sources in the
# production image
log.warning(
"Exception when importing '%s' from '%s' package: %s",
hook_class_name,
provider_package,
e,
)
else:
# This is special case - when airflow is installed from sources in production
# image, AIRFLOW_INSTALLATION_METHOD is set to ".". In this case we know that there
# Will be some warnings generated by ProviderManager, when it tries to import providers
# With missing dependencies, therefore we are turning such warnings into debug message
# so that we do not pollute logs.
log.debug(
"Exception when importing '%s' from '%s' package: %s",
hook_class_name,
provider_package,
e,
)
return
conn_type: str = self._get_attr(hook_class, 'conn_type')