Since we are mountign tmp dir now to inside container, some
of the remnants of what's going on inside remains after exit.
This is particularly bad if you are using tmux (some of the
directories remaining there prevent tmux from re-run)
This change cleans up /tmp directory on exit from Breeze command.
It does it from inside container so that we clean up all
root-owned files without sudo.
When quick-fixing Python 3.8.4 error #9820 PYTHON_BASE_IMAGE_VERSION
variable was added but it was initialized too early in Breeze and
it took the default version of Python rather than the one chosen
by --python switch. This caused the generated requirements
(locally by Breeze only) to generate wrong set of requirements
and images built locally for different python versions were
based on default Python version, not the one chosen by --python
switch.
In spark 3 they log the exit code with a lowercase
e, in spark 2 they used an uppercase E.
Also made the exception a bit clearer when running
on kubernetes.
Support for getting current context at any code location that runs
under the scope of BaseOperator.execute function. This functionality
is part of AIP-31.
Co-authored-by: Jonathan Shir <jonathan.shir@databand.ai>
* Import Iterable from collections.abc in firestore operators
DeprecationWarning: Using or importing the ABCs from 'collections'
instead of from 'collections.abc' is deprecated since Python 3.3,
and in 3.9 it will stop working
* Remove the type hint
* Constraint files are now maintained automatically
* No need to generate requirements when setup.py changes
* requirements are kept in separate orphan branches not in main repo
* merges to master verify if latest requirements are working and
push tested requirements to orphaned branches
* we keep history of requirement changes and can label them
individually for each version (by constraint-1.10.n tag name)
* consistently changed all references to be 'constraints' not
'requirements'
Before this change, if DAG Serialization was enabled the Webserver would not update the DAGs once they are fetched from DB. The default worker_refresh_interval was `30` so whenever the gunicorn workers were restarted, they used to pull the updated DAGs when needed.
This change will allow us to have a larged worker_refresh_interval (e.g 30 mins or even 1 day)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning
Johan Eklund
Kamil Breguła
Jarek Potiuk
Dani Hodovic
retornam
Beni Ben zikry
Stijn De Haes
QP Hou
chipmyersjr
Nathan Hadfield
Karthikeyan Singaravelan
BaoshanGu
Kamil Olszewski
Tomek Urbaszek
zikun
Shekhar Singh
Ephraim Anierobi
Ryan Hamilton
Kaxil Naik
morrme
Tanjin Panna
Jannik F
vanka56
Rafferty Chen