In case docker-context files are not used during build, they
shoudl be cleaned just before the build to make sure that
docker context does not contain extra files here. Otherwise
files left from previous runs might be in the context and cause
cache invalidation if you are building the images locally.
This change integrates an AWS glue crawler operator, hook and sensor that can be used to trigger glue crawlers from Airflow.
Co-authored-by: Kamil Breguła <kamil.bregula@polidea.com>
google_cloud_storage_conn_id parameter has been deprecated by GCSHook,
and should be replaced by gcp_conn_id parameter. google_cloud_storage_conn_id
was still in use in many Operators.
GCSHook renders a DeprecationWarning message everytime one of those operators uses
google_cloud_storage_conn_id. This PR avoid triggering DeprecationWarning when using
GCSHook in the codebase.
Co-authored-by: Kamil Breguła <kamil.bregula@polidea.com>
* Fix Google Spanner example dag
Some tasks requires upstream
Without upstream they want perform operations on resources which does not exist yet
* fixup! Fix Google Spanner example dag
Some users were not aware that we are not relasing images from
`stable` branch. This change clarifies branching strategy used
and what they can expect from the reference image published in
DockerHub.
Sometimes in our tests we get more than one deprecation
warnings. It is likely caused by transitive warnings
from importing other external libraries.
In order to get rid of those side effects, we are now
accepting more than one warning and we expect that at least
one of the warnings will come from the file being tested
The #13726 introduced possibility of using Github Container
Registry, but for bulding from Package Registry there was a
mistake - only visible after merging - that introduced
failed master (naming of build image contained / rather than -)
This PR fixes it.
* Fix error with quick-failing tasks in KubernetesPodOperator
Addresses an issue with the KubernetesPodOperator where tasks that die
quickly are not patched with "already_checked" because they never make
it to the monitoring logic.
* static fix
* Adds capability of switching to Github Container Registry
Currently we are using GitHub Packages to cache images for the
build. GitHub Packages are "legacy" storage of binary artifacts
for GitHub and as of September 2020 they introduced Github
Container Registry as more stable, easier to manage replacement
for container storage. It includes complete self-management
of the images including permission management, public access,
retention management and many more.
More about it here:
https://github.blog/2020-09-01-introducing-github-container-registry/
Recently we started to experience unstable behaviour of the
Github Packages ('unknown blob' and manifest v1 vs. v2 when
pushing images to it. So together with ASF we proposed to
enable Github Container Registry and it happened as of
January 2020.
More about it in https://issues.apache.org/jira/browse/INFRA-20959
We are currently in the testing phase, especially when it
comes to management of permissions - the model of permission
mangement is not the same for Container Registry as it was
for GitHub Packages (it was per-repository in GitHub Packages,
but it is organization-wide in the Container Registry.
This PR introduces an option to use GitHub Container Registry
rather than GitHub Packages. It is implemented in both - CI
level and Breeze level allowing to seamlessly switch between
those two solutions:
In Breeze (which we use to test pushing/pulling the images)
--github-registry option was added with `ghcr.io` (Github Container
Registry) or `docker.pkg.github.com` (GitHub Packages).
In CI the same can be achieved by setting GITHUB_REGISTRY value
(same values possible as for --github-registry Breeze parameter)
* fixup! Adds capability of switching to Github Container Registry
This change will BaseBranchOperator to do xcom push of the branch it choose to follow.
It will also add support to use the do_xcom_push parameter.
The added change returns the result received by running choose_branch().
Closes: #13704
* Correct the logic for webserver choosing number of workers to spawn (#13469)
A key consequence of this fix is that webserver will properly
exit when gunicorn master dies and stops responding to signals.
We've introduced chmod a+x for installation scripts in Dockerfiles.
but this turned out to be a bad idea. This was to accomodate
building on Azure Deveops which has filesystem that does not
keep executable bit. But the side-effect of it that the
layer of the script is invalidated when the permission is changed
to +x on linux. The problem is that the script has locally (on
checkout) different permissions depending on umask setting.
Therefore changing permissions for the image to +a is not best.
Instead we are running the scripts with bash directly, which does
not require changing of executable bit.
* Adds automated user creation in the production image
This PR implements automated user creation for the production image
controlled by environment variables.
This is a solution for anyone who would like to make a quick test
of the production image and would like to:
* init/upgrade the DB automatically
* create a user
This is particularly useful for internal SQLite db initialization
but can also be used to initialize the user in docker-compose
or similar cases where there is no equivalent of init containers
that are usually used to perform the initialization.
Closes#860
We are getting close to the previous timeouts for tests and some
tests are crossing the 80 minutes.
While we should speed it up in general, for now increasing
timeouts should do the job.
We have recently seen a number of issues created by users who
tried to install airflow with poetry or pip-tools or who had
successes with using the latest pip 20.3.3. This change aims
to update the 'note' content and make sure installation
instructions are consistent everywhere, so that new users
are warned against using anything else than PIP and that they
are aware about potential problems with 'pip 20.3' and ways
to mitigate the problems.
This responds to the needs of confused users such as
one in https://github.com/apache/airflow/issues/13711#issuecomment-761694781
This race condition resulted in task success and failure callbacks being
called more than once. Here is the order of events that could lead to
this issue:
* task started running within process 2
* (process 1) local_task_job checked for task return code, returns None
* (process 2) task exited with failure state, task state updated as failed in DB
* (process 2) task failure callback invoked through taskinstance.handle_failure method
* (process 1) local_task_job heartbeat noticed task state set to
failure, mistoken it as state bing updated externally, also invoked task
failure callback
To avoid this race condition, we need to make sure task callbacks are
only invoked within a single process.
Vivek Bhojawala
Jarek Potiuk
Marshall Mamiya
Mahesh Panati
Aakcht
Vladimir Mikhaylov
Loïc Messal
Kaxil Naik
Tomek Urbaszek
Tobiasz Kędzierski
Ephraim Anierobi
Joshua Carp
Daniel Imberman
Ashmeet Lamba
Griffin Cosgrove
Andrii Soldatenko
Jennifer Melot
André Amaral
drago-f5a
Brent Bovenzi
JavierLopezT
Kamil Breguła
QP Hou