BATS has additional libraries of asserts that are much more
straightforward and nicer to write tests for bash scripts
There is no dockerfile from BATS that contains those, so we
had to build our own (but it follows the same structure
as #9652 - where we keep our dev docker image
sources inside our repository and the generated docker images
in "apache/airflow:<tool>-CALVER-TOOLVER format.
We have more BATS unit test to add - following #10576
and this change will be of great help.
* Make grace_period_seconds option on K8sPodOperator
This PR allows users to choose whether they want to gracefully kill
pods when they delete tasks in the UI or if they would like to
immediately kill them.
* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
The current latest available version is 0.5.0 (https://pypi.org/project/Flask-Login/0.5.0/)
Version 0.5.0 drops support for Python 2.6, 3.3, 3.4 and we don't use those versions in Airflow Master
It could have wedged, (but the process still be alive) and we would
never notice.
In this I use `time.monotonic` rather than a `datetime` object for two
reasons:
1. We don't need the expense of a "full" date time object since all we
care about is the second diff between two points in time.
2. It is "more correct" as `datetime.now()` would be inaccurate if the
system clock changes (NTP etc.)
If we run this test
(TestTriggerRuleDep::test_get_states_count_upstream_ti specifically)
more than once without clearing the DB in between it would fail due to a
unique constraint violation.
The `@provide_session` wrapper will already commit the transaction when
returned, unless an explicit session is passed in -- removing this
parameter changes the behaviour to be:
- If session explicitly passed in: don't commit (caller's
responsibility)
- If no session passed in, `@provide_session` will commit for us already.
Add jupytercmd in Qubole Operator which fires a JupyterNotebookCommand to the jupyter notebooks running on user's QDS account. Along with this, we have fixed a minor bug that caused the tasks to fail with --notify is set in Qubole Operator.
Co-authored-by: Aaditya Sharma <asharma@qubole.com>
Inspired by the Google Shell Guide where they mentioned
separating package names with :: I realized that this was
one of the missing pieces in the bash scripts of ours.
While we already had packages (in libraries folders)
it's been difficult to realise which function is where.
With introducing packages - equal to the library file name
we are *almost* at a level of a structured language - and
it's easier to find the functions if you are looking for them.
Way easier in fact.
Part of #10576
(cherry picked from commit cc551ba793)
(cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
* Implement Google Shell Conventions for breeze script … (#10651)
Part of #10576
First (and the biggest of the series of commits to introduce
Google Shell Conventions in our bash scripts.
This is about the biggest and the most complex breeze script
so it is rather huge but it is difficult to split it into
smaller pieces.
The rules implemented (from the conventions):
* constants and exported variables are CAPITALIZED, where
local/temporary variables are lowercase
* following the shell guide, once all the variables are set to their
final values (either from exported variables, calculation or --switches
) I have a single function that makes all the variables read-only. That
helped to clean-up a lot of places where same functions was called
several times, or where variables were defined in a few places. Now the
behavior should be rather consistent and we should easily catch some
duplications
* function headers (following the guide) explaining arguments,
variables expected, variables modified in the functions used.
* setting the variables as read-only also helped to clean-up the "ifs"
where we often had ":=}" in variables and != "" or == "". Those are
replaced with `=}` and tests are replaced with `-n` and `-z` - also
following the shell guide (readonly helped to detect and clean all
such cases). This also should be much more robust in the future.
* reorganized initialization of those constants and variables - simplified
a few places where initialization was overlapping. It should be much more
straightforward and clean now
* a number of internal function breeze variables are "local" - this is
helpful in accidental variables overwriting and keeping stuff localized
* trap_add function is separated out to help in cases where we had
several traps handling the same signals.
(cherry picked from commit 46c8d6714c)
(cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246)
* fixup! Implement Google Shell Conventions for breeze script … (#10651)
* Revert "Add packages to function names in bash (#10670)"
This reverts commit cc551ba793.
* Revert "Implement Google Shell Conventions for breeze script … (#10651)"
This reverts commit 46c8d6714c.
This PR ensures that when a user kills a KubernetesPodOperator task
in the airflow UI, that the associated pod is also killed using the
on_kill method.
* Add `log_id` field to log lines on ES handler
* Add `offset` field to log lines on ES handler
it will be set to the epoch timestamp in nanoseconds (this will just be
used for ordering log lines when displayed in the webserver UI).
* Update UPDATING.md
With information regarding log_id and offset fields in JSON log lines written to stdout
Inspired by the Google Shell Guide where they mentioned
separating package names with :: I realized that this was
one of the missing pieces in the bash scripts of ours.
While we already had packages (in libraries folders)
it's been difficult to realise which function is where.
With introducing packages - equal to the library file name
we are *almost* at a level of a structured language - and
it's easier to find the functions if you are looking for them.
Way easier in fact.
Part of #10576
The requirements are not needed any more. We replaced them
with a new, better "constraints" mechanism where constraints
are stored in a separate, orphaned branches in the repository
and they are automatically maintained by the CI process.
See more about our dependency management process here:
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files
Varun Dhussa
Shekhar Singh
Kaxil Naik
Ryan Yuan
QP Hou
Antonio Davide Calì
Jarek Potiuk
Daniel Imberman
Kamil Breguła
Chris McLennon
Sumit Maheshwari
Ash Berlin-Taylor
João Marques
Tomek Urbaszek
Aaditya Sharma
Diego Lopes
Kamil Olszewski
Yannick Weiss
Nick
Ry Walker
Eyal Zekaria
Michał Słowikowski
Shoichi Kagawa