There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI
It was not deterministic because pylint usually uses as many processors as
many are available and it splits the list of .py files between the separate
pylint processors - depending on how the split is done, pylint check might
or might not detect it. The cycle is always detected when all files are used.
In order to make it more deterministic, all pylint and mypy errors were resolved
in all executors package and in dag_processor.
At the same time plugins_manager had also been moved out of the executors
and all of the operators/hooks/sensors/macros because it was also causing
cyclic dependencies and it's far easier to untangle those dependencies
in executor when we move the intialisation of all plugins to plugins_manager.
Additionally require_serial is set in pre-commit configuration to
make sure cycle detection is deterministic.
Now that the webserver is more stateless, if the scheduler is not
running the list of dags won't populate, making it harder for new
starters to work out what is going on.
New dep is BSD-2 which is Cat-A under ASF
- Changed single triple quotes to double quote
characters to be consistent with the docstring
convention in PEP 257
Closes#3396 from kaxil/AIRFLOW-2502
Adding back support for filter in max_partition()
which could be used by some valid use cases. It
will work for tables with multiple partitions,
which is the behavior before (tho the doc stated
it only works for single partitioned table). This
change also kept the behavior when trying to get
max partition on a sub-partitioned table without
supplying filter--it will return the max partition
value of the partition key even it is not unique.
Some extra checks are added to provide more
meaningful exception messages.
Closes#3117 from yrqls21/kevin_yang_add_filter
Call self.metastore.get_partition_names() instead of
self.metastore.get_partitions(), which is extremely expensive for
large tables, in HiveMetastoreHook().max_partition().
Closes#3082 from
yrqls21/kevin_yang_fix_hive_max_partition
If we're looking for the closest date before, we should take the latest date in the list of date before.
Closes#2257 from julien-gm/fix_closest-date-
partition
* Distinguish between module and non-module plugin
components
* Fix handling of non-module plugin components
* admin views, flask blueprints, and menu links
need to not be
wrapped in modules
* Fix improper use of zope.deprecation.deprecated
* zope.deprecation.deprecated does NOT support
classes as
first parameter
* deprecating classes must be handled by calling
the deprecate
function on the class name
* Added tests for plugin loading
* Updated plugin documentation to match test
plugin
* Updated executors to always load plugins
* More logging
Closes#1738 from gwax/plugin_module_fixes
- Added Apache license header for files with extension (.service, .in, .mako, .properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, .html, .xml.
- Added/Replaced shebang on all .sh files with portable version - #!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.
Closes#1598 from ajayyadava/248
David Cavaletto
Kamil Breguła
Jarek Potiuk
Tomek
Kevin Yang
Pedro Ortiz Monasterio
Ash Berlin-Taylor
Jiajie Zhong
Kaxil Naik
Holden Karau
Taylor D. Edmiston
Tao feng
Bolke de Bruin
Kengo Seki
julien-gm
George Leslie-Waksman
Stanislav Kudriashev
Ajay Yadav
jlowin
Maxime
Jeremiah Lowin
Maxime Beauchemin
Arthur Wiedmer
e3
LDAP/krishna_bhupatiraju
Arthur Wiedmer