DataProcSparkSqlOperator and DataProcHiveOperator are working either with query
or query_uri. Passing both doesn't make sense so check at construction time.
It defaults to the same location but this allows the location to be changed.
We may remove this test config file entirely in the future, but for now this lets the
location of this file be controlled.
- all related DagRun, TaskInstance and TaskFail objects are properly removed on tear down
- replaced "models.<ModelName>" importing with "<ModelName>" direct usage
- extracted test specific DAG ids into test suit constants
- small linter improvements
Moved query to fetch zombies from DagFileProcessorManager to DagBag class. Changed query to only look for DAGs of the current DAG bag. The query now uses index ti_dag_state instead of ti_state. Removed no longer required zombies parameters from many function signatures.
The query is now executed on every call to DagBag.kill_zombies which is called when the DAG file is processed which frequency depends on scheduler_heartbeat_sec and processor_poll_interval (AFAIU). The query is faster than the previous one (see also stats below). It's also negligible IMHO because during DAG file processing many other queries (DAG runs and task instances are created, task instance dependencies are checked) are executed.
Instantiating `Resources` and its child classes takes non-negligible
time when users create many operators. To save time, don't create the
resources object until it is needed.
While initializing a DAG, default_args is being mutated. If another
DAG is created in the same file with the same default_args, it gets
initialized with the incorrect Timezone information.
All this needed to enable the form was to add `can_add` to the
permission list
The run_id field is a required form (though the DB doesn't have it as
not nullable) - the scheduler requires it. so I have enabled the
required validation for it.
The `validators_columns` attribute on the view was ignored by FAB
because we set `add_form` and `edit_form` directly, so I have removed
the property
To answer why 'raise e' was commented:
1. This try-except is inside method execute_work() & there are other operations
after the try-except and after invoking this method.
Raising exception here will prevent all following steps from taking place.
2. The exception itself is already marked properly by labelling state to be FAILED,
and the exception is printed out using self.log.error().
Jarek Potiuk
eladkal
Facu
Felix Uellendall
Bluesn0w
Andrii Soldatenko
Fokko Driesprong
Matthew Rastovac
smogili1
Bhavika Tekwani
Christopher Alcock
Canh Duong
Luqman Abdul Mushawwir
Jarek
Stefan Seelmann
Xtreak
Alessandro Puccetti
Joshua Carp
Tomek
Alex
Philippe Gagnon
MisLink
Abhishek Ray
Evan Savage
Saurabh Wahile
Nicolas Crocfer
Xiaodong
Asgeir Berland
Ash Berlin-Taylor
Tanuj Gupta
César Reyes
Kamil Breguła
Chaithra Yenikapati
Kaxil Naik
Alejandro Rojas
Calvin Canh Tran
marsewe