This PR is a first step to more advanced S3 operators.
The hook can currently check for buckets, prefixes and keys,
and will parse boto, AWS sdks or s3cmd config files if they
are defined in the connection in the Airflow connection db.
The sensor uses the Hook to check for a particular file
(or key to follow amazon's nomenclature). I also added wrappers
for boto methods to list keys and prefixes.
@mistercrunch
PrestoValueCheck allows to define the value the SQL query should return
in order to pass the check. This can enable more readable SQL checks.
PrestoValueCheck can accomodate string values or numeric values with or
without a tolerance, which allows for close results.
@mistercrunch
This is an implementation of the PrestoIntervalCheck operator similar to
the one in the internal data-eng library at Airbnb. This operator will
allow to test if variations are within bounds over a period of time
on a week over week or YoY basis.
@mistercrunch
Maxime
Maxime Beauchemin
Arthur Wiedmer
Mike Kerzhner
LDAP/maxime_beauchemin