5f784ae5c0
[AIRFLOW-7061] Rename openfass to openfaas ( #7721 )
2020-03-14 10:47:50 +01:00
c997cab42d
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator ( #7630 )
...
Co-authored-by: michalslowikowski00 <michal.slowikowski@polidea.com>
2020-03-13 13:04:42 +01:00
6c39a3bf97
[AIRFLOW-6542] Add spark-on-k8s operator/hook/sensor ( #7163 )
2020-03-10 12:56:24 +00:00
bf9b6b6d70
[AIRFLOW-5013] Add GCP Data Catalog Hook and operators ( #7664 )
2020-03-09 23:16:32 +01:00
d1a34246ac
[AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators ( #7322 )
2020-02-25 00:42:27 +01:00
0bb687990b
[AIRFLOW-4030] second attempt to add singularity to airflow ( #7191 )
...
* adding singularity operator and tests
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>
* removing encoding pragmas and fixing up dockerfile to pass linting
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>
* make workdir in /tmp because AIRFLOW_SOURCES not defined yet
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>
* curl needs to follow redirects with -L
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>
* moving files to where they are supposed to be, more changes to mock, no clue
Signed-off-by: vsoch <vsochat@stanford.edu>
* removing trailing whitespace, moving example_dag for singularity, adding licenses to empty init files
Signed-off-by: vsoch <vsochat@stanford.edu>
* ran isort on example dags file
Signed-off-by: vsoch <vsochat@stanford.edu>
* adding missing init in example_dags folder for singularity
Signed-off-by: vsoch <vsochat@stanford.edu>
* removing code from __init__.py files for singularity operator to fix documentation generation
Signed-off-by: vsoch <vsochat@stanford.edu>
* forgot to update link to singularity in operators and hooks ref
Signed-off-by: vsoch <vsochat@stanford.edu>
* command must have been provided on init of singularity operator instance
Signed-off-by: vsoch <vsochat@stanford.edu>
* I guess I'm required to have a task_id?
Signed-off-by: vsoch <vsochat@stanford.edu>
* try adding working_dir to singularity operator type definitions
Signed-off-by: vsoch <vsochat@stanford.edu>
* disable too many arguments for pylint of singularity operator init
Signed-off-by: vsoch <vsochat@stanford.edu>
* move pylint disable up to line 64 - doesnt catch at end of statement like other examples
Signed-off-by: vsoch <vsochat@stanford.edu>
* two spaces before inline comment
Signed-off-by: vsoch <vsochat@stanford.edu>
* I dont see task_id as a param for other providers, removing for singularity operator
Signed-off-by: vsoch <vsochat@stanford.edu>
* adding debug print
Signed-off-by: vsoch <vsochat@stanford.edu>
* allow for return of just image and/or lines
Signed-off-by: vsoch <vsochat@stanford.edu>
* dont understand how mock works, but the image should exist after its pulled....
Signed-off-by: vsoch <vsochat@stanford.edu>
* try removing shutil, the client should handle pull folder instead
Signed-off-by: vsoch <vsochat@stanford.edu>
* try changing pull-file to same uri that is expected to be pulled
Signed-off-by: vsoch <vsochat@stanford.edu>
* import of AirflowException moved to exceptions
Signed-off-by: vsoch <vsochat@stanford.edu>
* DAG module was moved to airflow.models
Signed-off-by: vsoch <vsochat@stanford.edu>
* ensure pull is called with pull_folder
Signed-off-by: vsoch <vsochat@stanford.edu>
2020-02-23 10:49:47 +01:00
a9ad0a9298
[AIRFLOW-6790] Add basic Tableau Integration ( #7410 )
2020-02-22 01:01:24 +00:00
35b9616378
[AIRFLOW-4973] Add Cloud Data Fusion Pipeline integration ( #7486 )
2020-02-21 18:53:47 +00:00
d3cf23dc07
[AIRFLOW-5470] Add Apache Livy REST operator ( #6090 )
2020-02-19 16:19:02 +05:30
65dd28eb77
[AIRFLOW-1202] Create Elasticsearch Hook ( #7358 )
2020-02-18 22:03:00 +01:00
9cbd7de6d1
[AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests ( #7412 )
2020-02-18 00:57:52 +01:00
ee1ab7697c
[AIRFLOW-6531] Initial Yandex.Cloud Dataproc support ( #7252 )
...
* [AIRFLOW-6531] Initial Yandex.Cloud Dataproc support
* [AIRFLOW-6531] Move from contrib to providers. Drop py2 support
* [AIRFLOW-6531] Add service account support to YC connection
* [AIRFLOW-6531] Use Dataproc wrapper
* [AIRFLOW-6531] Move base classes to init to test "test_providers_modules_should_have_tests"
* [AIRFLOW-6531] Add type annotations. Remove op base class
* [AIRFLOW-6531] Add cluster id to templated fields
2020-02-14 20:04:29 +01:00
33af616b83
[AIRFLOW-6781] Enforce example_dags and guide for Google integrations ( #7404 )
2020-02-13 14:22:11 +01:00
1e00243014
[AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator ( #5785 )
2020-02-10 20:20:03 +00:00
1e576f1234
[AIRFLOW-6680] Last changes for AIP-21 ( #7301 )
...
* Move postgres tests from test_operators to test_postgres
* Move myqsl tests from test_operators to test_mysql
* Move myqsl tests from test_operators to test_mysql
* Move myqsl_to_hive tests from test_operators
* Move GCP system tests to providers.google.cloud
* Move PythonVirtualenvOperator to test_python
* Move GenericTransfer tests to test_generic_transfer
* Move MSSQL tests to providers.microsoft.mssql.operators
* Remove duplicate test_to_gcs_to_s3
* [AIP-21] Move operators.presto_to_mysql operators.mysql.operators.presto_to_mysql
* Move Cassandra hook tests to test_cassandra
* Improve strcutre of cassandra senssor testss
* Move TestHiveCli to hooks.test_hive
2020-02-02 20:14:49 +01:00
63aa3db88f
[AIRFLOW-6258] Add CloudFormation operators to AWS providers ( #6824 )
2020-02-02 14:57:39 +05:30
373c6aa4a2
[AIRFLOW-6682] Move GCP classes to providers package ( #7295 )
...
* [AIP-21] Move gcp.hooks.automl providers.google.cloud.hooks.automl
* [AIP-21] Move gcp.hooks.bigquery providers.google.cloud.hooks.bigquery
* [AIP-21] Move gcp.hooks.bigquery_dts providers.google.cloud.hooks.bigquery_dts
* [AIP-21] Move gcp.hooks.bigtable providers.google.cloud.hooks.bigtable
* [AIP-21] Move gcp.hooks.cloud_build providers.google.cloud.hooks.cloud_build
* [AIP-21] Move gcp.hooks.cloud_memorystore providers.google.cloud.hooks.cloud_memorystore
* [AIP-21] Move gcp.hooks.cloud_sql providers.google.cloud.hooks.cloud_sql
* [AIP-21] Move gcp.hooks.cloud_storage_transfer_service providers.google.cloud.hooks.cloud_storage_transfer_service
* [AIP-21] Move gcp.hooks.compute providers.google.cloud.hooks.compute
* [AIP-21] Move gcp.hooks.dataflow providers.google.cloud.hooks.dataflow
* [AIP-21] Move gcp.hooks.datastore providers.google.cloud.hooks.datastore
* [AIP-21] Move gcp.hooks.dlp providers.google.cloud.hooks.dlp
* [AIP-21] Move gcp.hooks.functions providers.google.cloud.hooks.functions
* [AIP-21] Move gcp.hooks.gcs providers.google.cloud.hooks.gcs
* [AIP-21] Move gcp.hooks.kms providers.google.cloud.hooks.kms
* [AIP-21] Move gcp.hooks.kubernetes_engine providers.google.cloud.hooks.kubernetes_engine
* [AIP-21] Move gcp.hooks.mlengine providers.google.cloud.hooks.mlengine
* [AIP-21] Move gcp.hooks.spanner providers.google.cloud.hooks.spanner
* [AIP-21] Move gcp.hooks.speech_to_text providers.google.cloud.hooks.speech_to_text
* [AIP-21] Move gcp.hooks.text_to_speech providers.google.cloud.hooks.text_to_speech
* [AIP-21] Move gcp.hooks.translate providers.google.cloud.hooks.translate
* [AIP-21] Move gcp.hooks.discovery_api providers.google.cloud.hooks.discovery_api
* [AIP-21] Move gcp.hooks.video_intelligence providers.google.cloud.hooks.video_intelligence
* [AIP-21] Move gcp.sensors.bigquery providers.google.cloud.sensors.bigquery
* [AIP-21] Move gcp.sensors.bigquery_dts providers.google.cloud.sensors.bigquery_dts
* [AIP-21] Move gcp.sensors.bigtable providers.google.cloud.sensors.bigtable
* [AIP-21] Move gcp.sensors.cloud_storage_transfer_service providers.google.cloud.sensors.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.automl providers.google.cloud.operators.automl
* [AIP-21] Move gcp.operators.bigquery providers.google.cloud.operators.bigquery
* [AIP-21] Move gcp.operators.bigquery_dts providers.google.cloud.operators.bigquery_dts
* [AIP-21] Move gcp.operators.bigtable providers.google.cloud.operators.bigtable
* [AIP-21] Move gcp.operators.cloud_build providers.google.cloud.operators.cloud_build
* [AIP-21] Move gcp.operators.cloud_memorystore providers.google.cloud.operators.cloud_memorystore
* [AIP-21] Move gcp.operators.cloud_sql providers.google.cloud.operators.cloud_sql
* [AIP-21] Move gcp.operators.cloud_storage_transfer_service providers.google.cloud.operators.cloud_storage_transfer_service
* [AIP-21] Move gcp.operators.compute providers.google.cloud.operators.compute
* [AIP-21] Move gcp.operators.datastore providers.google.cloud.operators.datastore
* [AIP-21] Move gcp.operators.dlp providers.google.cloud.operators.dlp
* [AIP-21] Move gcp.operators.functions providers.google.cloud.operators.functions
* [AIP-21] Move gcp.operators.gcs providers.google.cloud.operators.gcs
* [AIP-21] Move gcp.operators.kubernetes_engine providers.google.cloud.operators.kubernetes_engine
* [AIP-21] Move gcp.operators.mlengine providers.google.cloud.operators.mlengine
* [AIP-21] Move gcp.operators.spanner providers.google.cloud.operators.spanner
* [AIP-21] Move gcp.operators.speech_to_text providers.google.cloud.operators.speech_to_text
* [AIP-21] Move gcp.operators.tasks providers.google.cloud.operators.tasks
* [AIP-21] Move gcp.operators.text_to_speech providers.google.cloud.operators.text_to_speech
* [AIP-21] Move gcp.operators.translate providers.google.cloud.operators.translate
* [AIP-21] Move gcp.operators.translate_speech providers.google.cloud.operators.translate_speech
* [AIP-21] Move gcp.operators.video_intelligence providers.google.cloud.operators.video_intelligence
* [AIP-21] Move operators.adls_to_gcs providers.google.cloud.operators.adls_to_gcs
* [AIP-21] Move operators.bigquery_to_bigquery providers.google.cloud.operators.bigquery_to_bigquery
* [AIP-21] Move operators.bigquery_to_gcs providers.google.cloud.operators.bigquery_to_gcs
* [AIP-21] Move operators.bigquery_to_mysql providers.google.cloud.operators.bigquery_to_mysql
* [AIP-21] Move operators.cassandra_to_gcs providers.google.cloud.operators.cassandra_to_gcs
* [AIP-21] Move operators.gcs_to_bq providers.google.cloud.operators.gcs_to_bigquery
* [AIP-21] Move operators.gcs_to_gcs providers.google.cloud.operators.gcs_to_gcs
* [AIP-21] Move operators.gcs_to_sftp providers.google.cloud.operators.gcs_to_sftp
* [AIP-21] Move operators.local_to_gcs providers.google.cloud.operators.local_to_gcs
* [AIP-21] Move operators.mssql_to_gcs providers.google.cloud.operators.mssql_to_gcs
* [AIP-21] Move operators.mysql_to_gcs providers.google.cloud.operators.mysql_to_gcs
* [AIP-21] Move operators.postgres_to_gcs providers.google.cloud.operators.postgres_to_gcs
* [AIP-21] Move operators.sql_to_gcs providers.google.cloud.operators.sql_to_gcs
* Update docs
* [AIP-21] Move gcp.hooks.base providers.google.cloud.hooks.base
* Move system tests to providers package
* Fix deprecation message
* Remove gcp.{operators,hooks,sensors}
* Move gcp.example_dags to providers.google.cloud.example_dags
* Move gcp.utils to providers.google.cloud.utils
* Remove airflow.gcp
* Fix import paths in tests
* Remove reference to airflow.gcp
* Fix utils path in test_mlengine_utils.py
2020-01-30 21:59:38 +01:00
057f3ae3a4
[AIRFLOW-6670][depends on AIRFLOW-6669] Move contrib operators to providers package ( #7286 )
2020-01-29 06:19:27 +01:00
1988a97e8f
[AIRFLOW-6659] Move AWS Transfer operators to providers package ( #7274 )
...
* [AIP-21] Move contrib.operators.dynamodb_to_s3 providers.amazon.aws.operators.dynamodb_to_s3
* [AIP-21] Move contrib.operators.hive_to_dynamodb providers.amazon.aws.operators.hive_to_dynamodb
* [AIP-21] Move contrib.operators.imap_attachment_to_s3_operator providers.amazon.aws.operators.imap_attachment_to_s3
* [AIP-21] Move contrib.operators.mongo_to_s3 providers.amazon.aws.operators.mongo_to_s3
* [AIP-21] Move contrib.operators.s3_to_sftp_operator providers.amazon.aws.operators.s3_to_sftp
* [AIP-21] Move contrib.operators.sftp_to_s3_operator providers.amazon.aws.operators.sftp_to_s3
* [AIP-21] Move operators.gcs_to_s3 providers.amazon.aws.operators.gcs_to_s3
* [AIP-21] Move operators.google_api_to_s3_transfer providers.amazon.aws.operators.google_api_to_s3_transfer
* [AIP-21] Move operators.redshift_to_s3_operator providers.amazon.aws.operators.redshift_to_s3
* [AIP-21] Move operators.s3_to_redshift_operator providers.amazon.aws.operators.s3_to_redshift
2020-01-28 22:34:38 +01:00
ab10443e96
[AIRFLOW-6424] Added a operator to modify EMR cluster ( #7213 )
...
* [AIRFLOW-6424] Added a operator to modify EMR cluster
* [AIRFLOW-6424] Updated docs for EMR modify cluster operator
* [AIRFLOW-6424] Removed sanity check
2020-01-28 16:14:11 +01:00
ceea293c16
[AIRFLOW-6656] Fix AIP-21 moving ( #7272 )
2020-01-28 15:37:30 +01:00
82c0e5aff6
[AIRFLOW-6655] Move AWS classes to providers ( #7271 )
2020-01-28 12:58:31 +01:00
9a04013b0e
[AIRFLOW-6646][AIP-21] Move protocols classes to providers package ( #7268 )
...
* [AIP-21] Move contrib.hooks.ftp_hook providers.ftp.hooks.ftp
* [AIP-21] Move contrib.hooks.grpc_hook providers.grpc.hooks.grpc
* [AIP-21] Move contrib.hooks.imap_hook providers.imap.hooks.imap
* [AIP-21] Move contrib.hooks.ssh_hook providers.ssh.hooks.ssh
* [AIP-21] Move contrib.hooks.winrm_hook providers.microsoft.winrm.hooks.winrm
* [AIP-21] Move contrib.operators.grpc_operator providers.grpc.operators.grpc
* [AIP-21] Move contrib.operators.ssh_operator providers.ssh.operators.ssh
* [AIP-21] Move contrib.operators.winrm_operator providers.microsoft.winrm.operators.winrm
* [AIP-21] Move contrib.sensors.imap_attachment_sensor providers.imap.sensors.imap_attachment
* [AIP-21] Move hooks.http_hook providers.http.hooks.http
* [AIP-21] Move hooks.jdbc_hook providers.jdbc.hooks.jdbc
* [AIP-21] Move contrib.sensors.ftp_sensor providers.ftp.sensors.ftp
* [AIP-21] Move operators.email_operator providers.email.operators.email
* [AIP-21] Move operators.http_operator providers.http.operators.http
* [AIP-21] Move operators.jdbc_operator providers.jdbc.operators.jdbc
* [AIP-21] Move sensors.http_sensor providers.http.sensors.http
* Update docs
2020-01-27 14:36:27 +01:00
c42a375e79
[AIRFLOW-6644][AIP-21] Move service classes to providers package ( #7265 )
...
* [AIP-21] Move contrib.hooks.cloudant_hook providers.cloudant.hooks.cloudant
* [AIP-21] Move contrib.hooks.databricks_hook providers.databricks.hooks.databricks
* [AIP-21] Move contrib.hooks.datadog_hook providers.datadog.hooks.datadog
* [AIP-21] Move contrib.hooks.dingding_hook providers.dingding.hooks.dingding
* [AIP-21] Move contrib.hooks.discord_webhook_hook providers.ddiscord.hooks.discord_webhook
* [AIP-21] Move contrib.hooks.gdrive_hook providers.google.suite.hooks.drive
* [AIP-21] Move contrib.hooks.jenkins_hook providers.jenking.hooks.jenkins
* [AIP-21] Move contrib.hooks.opsgenie_alert_hook providers.opsgenie.hooks.opsgenie_alert
* [AIP-21] Move contrib.hooks.pagerduty_hook providers.pagerduty.hooks.pagerduty
* [AIP-21] Move contrib.hooks.qubole_check_hook providers.qubole.hooks.qubole_check
* [AIP-21] Move contrib.hooks.qubole_hook providers.qubole.hooks.qubole
* [AIP-21] Move contrib.hooks.salesforce_hook providers.salesforce.hooks.salesforce
* [AIP-21] Move contrib.hooks.segment_hook providers.segment.hooks.segment
* [AIP-21] Move contrib.hooks.snowflake_hook providers.snowflake.hooks.snowflake
* [AIP-21] Move contrib.hooks.vertica_hook providers.vertica.hooks.vertica
* [AIP-21] Move gcp.hooks.gsheets providers.google.suite.hooks.sheets
* [AIP-21] Move hooks.slack_hook providers.slack.hooks.slack
* [AIP-21] Move hooks.zendesk_hook providers.zendesk.hooks.zendesk
* [AIP-21] Move contrib.operators.databricks_operator providers.databricks.operators.databricks
* [AIP-21] Move contrib.operators.dingding_operator providers.dindding.operators.dingding
* [AIP-21] Move contrib.operators.discord_webhook_operator providers.discord.operators.discord_webhook
* [AIP-21] Move contrib.operators.jenkins_job_trigger_operator providers.jenking.operators.jenkins_job_trigger
* [AIP-21] Move contrib.operators.opsgenie_alert_operator providers.opsgenie.operators.opsgenie_alert
* [AIP-21] Move contrib.operators.qubole_check_operator providers.qubole.operators.qubole_check
* [AIP-21] Move contrib.operators.qubole_operator providers.qubole.operators.qubole
* [AIP-21] Move contrib.operators.segment_track_event_operator providers.segment.operators.segment_track_event
* [AIP-21] Move contrib.operators.slack_webhook_operator providers.slack.operators.slack_webhook
* [AIP-21] Move contrib.operators.vertica_operator providers.vertica.operators.vertica
* [AIP-21] Move contrib.sensors.datadog_sensor providers.datadog.sensors.datadog
* [AIP-21] Move contrib.sensors.qubole_sensor providers.qubole.sensors.qubole
* [AIP-21] Move operators.slack_operator providers.slack.operators.slack
* Update docs
2020-01-27 01:33:15 +01:00
059eda05f8
[AIRFLOW-6610] Move software classes to providers package ( #7231 )
...
* [AIP-21] Move contrib.hooks.mongo_hook providers.mongo.hooks.mongo
* [AIP-21] Move contrib.hooks.openfaas_hook providers.openfass.hooks.openfaas
* [AIP-21] Move contrib.hooks.redis_hook providers.redis.hooks.redis
* [AIP-21] Move contrib.operators.docker_swarm_operator providers.docker.operators.docker_swarm
* [AIP-21] Move contrib.operators.redis_publish_operator providers.redis.operators.redis_publish
* [AIP-21] Move contrib.operators.kubernetes_pod_operator providers.cncf.kubernetes.operators.kubernetes_pod
* [AIP-21] Move contrib.sensors.bash_sensor sensors.bash
* [AIP-21] Move contrib.sensors.celery_queue_sensor providers.celery.sensors.celery_queue
* [AIP-21] Move contrib.sensors.mongo_sensor providers.mongo.sensors.mongo
* [AIP-21] Move contrib.sensors.python_sensor sensors.python
* [AIP-21] Move contrib.sensors.redis_key_sensor providers.redis.sensors.redis_key
* [AIP-21] Move contrib.sensors.redis_pub_sub_sensor providers.redis.sensors.redis_pub_sub
* [AIP-21] Move hooks.docker_hook providers.docker.hooks.docker
* [AIP-21] Move hooks.mssql_hook providers.microsoft.mssql.hooks.mssql
* [AIP-21] Move hooks.mysql_hook providers.mysql.hooks.mysql
* [AIP-21] Move hooks.oracle_hook providers.oracle.hooks.oracle
* [AIP-21] Move hooks.postgres_hook providers.postgres.hooks.postgres
* [AIP-21] Move hooks.presto_hook providers.presto.hooks.presto
* [AIP-21] Move hooks.samba_hook providers.samba.hooks.samba
* [AIP-21] Move hooks.sqlite_hook providers.sqlite.hooks.sqlite
* [AIP-21] Move operators.bash_operator operators.bash
* [AIP-21] Move operators.docker_operator providers.docker.operators.docker
* [AIP-21] Move operators.mssql_operator providers.microsoft.mssql.operators.mssql
* [AIP-21] Move operators.mysql_operator providers.mssql.operators.mysql
* [AIP-21] Move operators.oracle_operator providers.oracle.operators.oracle
* [AIP-21] Move operators.papermill_operator providers.papermill.operators.papermill
* [AIP-21] Move operators.postgres_operator providers.postgres.operators.postgres
* [AIP-21] Move operators.presto_check_operator providers.presto.operators.presto_check
* [AIP-21] Move operators.python_operator operators.python
* [AIP-21] Move operators.sqlite_operator providers.sqlite.operators.sqlite
* Update docs
2020-01-21 18:00:52 +01:00
17af3beea5
[AIRFLOW-5816] Add S3 to snowflake operator ( #6469 )
...
- move Snowflake from contrib to providers
- fix pylint issues
Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
2020-01-21 09:58:21 +01:00
2a819b11fb
[AIRFLOW-6296] add OdbcHook & deprecation warning for pymssql ( #6850 )
2020-01-19 07:54:56 +01:00
c319e81cae
[AIRFLOW-6572] Move AWS classes to providers.amazon.aws package ( #7178 )
...
* [AIP-21] Move contrib.hooks.aws_glue_catalog_hook airflow.contrib.hooks.aws_glue_catalog_hook
* [AIP-21] Move contrib.hooks.aws_logs_hook airflow.contrib.hooks.aws_logs_hook
* [AIP-21] Move contrib.hooks.emr_hook airflow.contrib.hooks.emr_hook
* [AIP-21] Move contrib.operators.ecs_operator airflow.contrib.operators.ecs_operator
* [AIP-21] Move contrib.operators.emr_add_steps_operator airflow.contrib.operators.emr_add_steps_operator
* [AIP-21] Move contrib.operators.emr_create_job_flow_operator airflow.contrib.operators.emr_create_job_flow_operator
* [AIP-21] Move contrib.operators.emr_terminate_job_flow_operator airflow.contrib.operators.emr_terminate_job_flow_operator
* [AIP-21] Move contrib.operators.s3_copy_object_operator airflow.contrib.operators.s3_copy_object_operator
* [AIP-21] Move contrib.operators.s3_delete_objects_operator airflow.contrib.operators.s3_delete_objects_operator
* [AIP-21] Move contrib.operators.s3_list_operator airflow.contrib.operators.s3_list_operator
* [AIP-21] Move contrib.operators.sagemaker_base_operator airflow.contrib.operators.sagemaker_base_operator
* [AIP-21] Move contrib.operators.sagemaker_endpoint_config_operator airflow.contrib.operators.sagemaker_endpoint_config_operator
* [AIP-21] Move contrib.operators.sagemaker_endpoint_operator airflow.contrib.operators.sagemaker_endpoint_operator
* [AIP-21] Move contrib.operators.sagemaker_model_operator airflow.contrib.operators.sagemaker_model_operator
* [AIP-21] Move contrib.operators.sagemaker_training_operator airflow.contrib.operators.sagemaker_training_operator
* [AIP-21] Move contrib.operators.sagemaker_transform_operator airflow.contrib.operators.sagemaker_transform_operator
* [AIP-21] Move contrib.operators.sagemaker_tuning_operator airflow.contrib.operators.sagemaker_tuning_operator
* [AIP-21] Move contrib.sensors.aws_glue_catalog_partition_sensor airflow.contrib.sensors.aws_glue_catalog_partition_sensor
* [AIP-21] Move contrib.sensors.emr_base_sensor airflow.contrib.sensors.emr_base_sensor
* [AIP-21] Move contrib.sensors.emr_job_flow_sensor airflow.contrib.sensors.emr_job_flow_sensor
* [AIP-21] Move contrib.sensors.emr_step_sensor airflow.contrib.sensors.emr_step_sensor
* [AIP-21] Move contrib.sensors.sagemaker_base_sensor airflow.contrib.sensors.sagemaker_base_sensor
* [AIP-21] Move contrib.sensors.sagemaker_endpoint_sensor airflow.contrib.sensors.sagemaker_endpoint_sensor
* [AIP-21] Move contrib.sensors.sagemaker_training_sensor airflow.contrib.sensors.sagemaker_training_sensor
* [AIP-21] Move contrib.sensors.sagemaker_transform_sensor airflow.contrib.sensors.sagemaker_transform_sensor
* [AIP-21] Move contrib.sensors.sagemaker_tuning_sensor airflow.contrib.sensors.sagemaker_tuning_sensor
* [AIP-21] Move operators.s3_file_transform_operator airflow.operators.s3_file_transform_operator
* [AIP-21] Move sensors.s3_key_sensor airflow.sensors.s3_key_sensor
* [AIP-21] Move sensors.s3_prefix_sensor airflow.sensors.s3_prefix_sensor
* [AIP-21] Move contrib.hooks.sagemaker_hook providers.amazon.aws.hooks.sagemaker
2020-01-17 09:39:04 +01:00
290330ba60
[AIRFLOW-6552] Move Azure classes to providers.microsoft package ( #7158 )
2020-01-15 21:22:10 +01:00
a0568b9536
[AIRFLOW-6543][AIP-21] Promotion of contrib classes to the core ( #7145 )
...
* [AIP-21] Move contrib.hooks.fs_hook hooks.filesystem
* [AIP-21] Move contrib.sensors.weekday_sensor sensors.weekday_sensor
* [AIP-21] Move contrib.sensors.file_sensor sensors.filesystem
2020-01-14 01:02:50 +01:00
0481b9a957
[AIRFLOW-6539][AIP-21] Move Apache classes to providers.apache package ( #7142 )
...
* [AIP-21] Move contrib.hooks.pinot_hook providers.apache.pinot.hooks.pinot
* [AIP-21] Move contrib.hooks.spark_jdbc_hook providers.apache.spark.hooks.spark_jdbc
* [AIP-21] Move contrib.hooks.spark_jdbc_script providers.apache.spark.hooks.spark_jdbc_script
* [AIP-21] Move contrib.hooks.spark_sql_hook providers.apache.spark.hooks.spark_sql
* [AIP-21] Move contrib.hooks.spark_submit_hook providers.apache.spark.hooks.spark_submit
* [AIP-21] Move contrib.hooks.sqoop_hook providers.apache.sqoop.hooks.sqoop
* [AIP-21] Move contrib.operators.druid_operator providers.apache.druid.operators.druid
* [AIP-21] Move contrib.operators.spark_jdbc_operator providers.apache.spark.operators.spark_jdbc
* [AIP-21] Move contrib.operators.spark_sql_operator providers.apache.spark.operators.spark_sql
* [AIP-21] Move contrib.operators.spark_submit_operator providers.apache.spark.operators.spark_submit
* [AIP-21] Move contrib.operators.sqoop_operator providers.apache.sqoop.operators.sqoop
* [AIP-21] Move contrib.sensors.hdfs_sensor providers.apache.hdfs.sensors.hdfs
* [AIP-21] Move hooks.druid_hook providers.apache.druid.hooks.druid
* [AIP-21] Move hooks.hdfs_hook providers.apache.hdfs.hooks.hdfs
* [AIP-21] Move hooks.hive_hooks providers.apache.hive.hooks.hive
* [AIP-21] Move hooks.pig_hook providers.apache.pig.hooks.pig
* [AIP-21] Move hooks.webhdfs_hook providers.apache.hdfs.hooks.webhdfs
* [AIP-21] Move operators.druid_check_operator providers.apache.druid.operators.druid_check
* [AIP-21] Move operators.hive_operator providers.apache.hive.operators.hive
* [AIP-21] Move operators.hive_stats_operator providers.apache.hive.operators.hive_stats
* [AIP-21] Move operators.pig_operator providers.apache.pig.operators.pig
* [AIP-21] Move sensors.hive_partition_sensor providers.apache.hive.sensors.hive_partition
* [AIP-21] Move sensors.metastore_partition_sensor providers.apache.hive.sensors.metastore_partition
* [AIP-21] Move sensors.named_hive_partition_sensor providers.apache.hive.sensors.named_hive_partition
* [AIP-21] Move sensors.web_hdfs_sensor providers.apache.hdfs.sensors.web_hdfs
* Update docs
* [AIP-21] Move sensors.hdfs_sensor providers.apache.hdfs.sensors.hdfs
2020-01-12 15:51:30 +01:00
78d8fe6944
[AIRFLOW-6245] Add custom waiters for AWS batch jobs ( #6811 )
...
- add AwsBatchWaiters
- the waiters are based on botocore, but not yet
available for AWS batch services
- refactor AwsBatchOperator:
- use an optional waiters object to wait for
batch job status indicators
- split execute into submit_job and monitor_job
- use job_id with an optional init-parameter;
discard jobId and jobName (already has job_name)
- inherit from AwsBatchClient
- add notes to UPDATING.md
- extract class for AwsBatchClient
- move responsibility for batch API calls and
response parsing to this client
- move responsibility for default wait and
polling to this client
- rename BatchProtocol to AwsBatchProtocol [AIP-21]
- test backward compatibility
- add PROTOCOLS to tests/test_core_to_contrib.py
- add notes to UPDATING.md
- split up polling for job status into steps:
- poll for a JobExists
- poll for a JobRunning
- poll for a JobComplete
- use random jitter for wait-polling delays for
high concurrency job polling
- modify the exponential backoff delay for the
existing polling functions
- revise and update unit tests
2020-01-08 11:52:28 +01:00
cf647c27e0
[AIRFLOW-6038] AWS DataSync reworked ( #6773 )
...
Added Amazon AWS how-to documentation scaffolding, plus example DAGs for AWS DataSync Operators with their respective how-to guides.
Reworked AWS DataSync operators into a single, logically idempotent operator.
2019-12-18 16:33:29 +01:00
7502cad284
[AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] ( #6764 )
...
- conform to AIP-21
- see https://issues.apache.org/jira/browse/AIRFLOW-4733
- see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
- use airflow.providers.amazon.aws.operators.batch.AwsBatchOperator
- deprecate airflow.contrib.operators.awsbatch_operator.AWSBatchOperator
- fix pylint for airflow/providers/amazon/aws/operators/batch.py
2019-12-17 16:15:47 +00:00
69629a5a94
[AIRFLOW-5807] Move SFTP from contrib to providers. ( #6464 )
...
* [AIRFLOW-5807] Move SFTP from contrib to core
2019-12-09 17:42:19 +01:00
37a1bd80f8
[AIRFLOW-6007] Check providers instead of provider package ( #6599 )
2019-12-04 13:11:44 +01:00
baae140847
[AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws ( #6588 )
2019-11-19 00:27:23 +01:00
504cfbac1a
[AIRFLOW-5783] AIP-21 Move aws redshift into providers structure ( #6539 )
2019-11-18 09:16:36 +01:00
5c4cfea8c0
[AIRFLOW-5718] Add SFTPToGoogleCloudStorageOperator ( #6393 )
2019-11-15 14:32:32 +01:00
4c9198017d
[AIRFLOW-XXX] Alphabetical table and remove duplicate ( #6487 )
2019-11-14 01:10:58 +01:00
992f0e3acf
AIRFLOW-5824: AWS DataSync Hook and Operators added ( #6512 )
2019-11-12 10:44:56 +01:00
d25ef2b2ab
[AIRFLOW-5832] Add pagerduty hook ( #6484 )
2019-11-11 21:48:20 +00:00
c015eb2f64
[AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws ( #6502 )
...
* [AIRFLOW-5786] Move AWS Sns hook and operator according to AIP-21
2019-11-10 13:57:11 +01:00
3d76fb4bf2
[AIRFLOW-5782] Migrate AWS Lambda to /providers/amazon/aws [AIP-21] ( #6518 )
...
* [AIRFLOW-5782] Migrate AWS Lambda to /providers/aws [AIP-21]
2019-11-09 16:03:41 +01:00
d985c02d9f
[AIRFLOW-XXX] Add How-To-Guide to GCP PubSub ( #6497 )
2019-11-05 22:16:02 -08:00
a296cdabdb
[AIRFLOW-5743] Move Google PubSub to providers package ( #6476 )
2019-11-04 22:27:03 +01:00
ba40be7a43
[AIRFLOW-XXX] Add How-To-Guide to GoogleCloudStorageToSFTPOperator ( #6488 )
...
* [AIRFLOW-XXX] Add How-To-Guide to GoogleCloudStorageToSFTPOperator
2019-11-04 14:24:39 +01:00
71cbe7e9bd
[AIRFLOW-5787] Moving AWS SQS to /providers/aws ( #6474 )
2019-10-31 09:48:16 +01:00
470b2a779d
[AIRFLOW-5741] Move Cloud Natural Language to providers ( #6421 )
2019-10-30 23:03:43 +01:00
db29639143
[AIRFLOW-5769] Move the S3_hook to /providers/aws/hooks ( #6443 )
2019-10-29 13:33:43 +01:00
Bas Harenslak
Michał Słowikowski
roitvt
Kamil Breguła
Jithin Sukumar
Vanessasaurus
Felix Uellendall
Tomek Urbaszek
Luca Cavazzana
Hossein Torabi
peter-volkov
Michael Spector
Aviem Zur
Gavin D'mello
SHAKEEL
dstandish
Darren Weber
Bjorn Olsen
TobKed
Jarek Potiuk
Rich Dean
Jiajie Zhong
Qingping Hou
Rich Dean
Cyril Shcherbin
Ishan Rastogi
MinJae Kwon