[Eventhub] TableCheckpointstore base branch (#19726)
adding checkingpointstore for table. This is my initial pull request.
This commit is contained in:
Jg1255
GitHub
Родитель
64128bae19
Коммит
03f14493ba
|
|
@ -0,0 +1,4 @@
|
|||
# Release History
|
||||
|
||||
## 1.0.0b1 (Unreleased)
|
||||
- Template package (can update this after implementation is done)
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
include *.md
|
||||
include azure/__init__.py
|
||||
include azure/eventhub/__init__.py
|
||||
include azure/eventhub/extensions/__init__.py
|
||||
recursive-include tests *.py
|
||||
recursive-include samples *.py
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
[](https://dev.azure.com/azure-sdk/public/_build/latest?definitionId=46?branchName=main)
|
||||
|
||||
# Azure EventHubs Checkpoint Store client library for Python using Tables
|
||||
|
||||
Azure EventHubs Checkpoint Store is used for storing checkpoints while processing events from Azure Event Hubs.
|
||||
This Checkpoint Store package works as a plug-in package to `EventHubConsumerClient`. It uses Azure Tables as the persistent store for maintaining checkpoints and partition ownership information.
|
||||
|
||||
|
||||
# Getting started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python2.7, Python 3.6 or later.
|
||||
- **Microsoft Azure Subscription:** To use Azure services, including Azure Event Hubs, you'll need a subscription. If you do not have an existing Azure account, you may sign up for a free trial or use your MSDN subscriber benefits when you [create an account](https://azure.microsoft.com/).
|
||||
|
||||
- **Event Hubs namespace with an Event Hub:** To interact with Azure Event Hubs, you'll also need to have a namespace and Event Hub available. If you are not familiar with creating Azure resources, you may wish to follow the step-by-step guide for [creating an Event Hub using the Azure portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create). There, you can also find detailed instructions for using the Azure CLI, Azure PowerShell, or Azure Resource Manager (ARM) templates to create an Event Hub.
|
||||
|
||||
- **Azure Storage Account:** You'll need to have an Azure Storage Account and create a Azure Table Storage to store the checkpoint data with entities. You may follow the guide [creating an Azure Table Storage Account]
|
||||
(https://docs.microsoft.com/azure/storage/tables/table-storage-overview).
|
||||
|
||||
# Key concepts
|
||||
|
||||
Bullet point list of your library's main concepts.
|
||||
|
||||
# Examples
|
||||
|
||||
Examples of some of the key concepts for your library.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
Running into issues? This section should contain details as to what to do there.
|
||||
|
||||
# Next steps
|
||||
|
||||
More sample code should go here, along with links out to the appropriate example tests.
|
||||
|
||||
# Contributing
|
||||
|
||||
If you encounter any bugs or have suggestions, please file an issue in the [Issues](<https://github.com/Azure/azure-sdk-for-python/issues>) section of the project.
|
||||
|
||||
|
||||
|
|
@ -0,0 +1 @@
|
|||
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
__path__ = __import__("pkgutil").extend_path(__path__, __name__) # type: ignore
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
__path__ = __import__("pkgutil").extend_path(__path__, __name__) # type: ignore
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
from ._version import VERSION
|
||||
|
||||
__version__ = VERSION
|
||||
|
||||
from ._tablestoragecs import TableCheckpointStore
|
||||
|
||||
__all__ = [
|
||||
"TableCheckpointStore",
|
||||
]
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# --------------------------------------------------------------------------------------------
|
||||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
class TableCheckpointStore:
|
||||
"""A CheckpointStore that uses Azure Table Storage to store the partition ownership and checkpoint data.
|
||||
This class implements methods list_ownership, claim_ownership, update_checkpoint and list_checkpoints.
|
||||
:param str table_account_url:
|
||||
The URI to the storage account.
|
||||
:param table_name:
|
||||
The name of the table for the tables.
|
||||
:type table_name: str
|
||||
:param credential:
|
||||
The credentials with which to authenticate. This is optional if the
|
||||
account URL already has a SAS token. The value can be a SAS token string, an account
|
||||
shared access key, or an instance of a TokenCredentials class from azure.identity.
|
||||
If the URL already has a SAS token, specifying an explicit credential will take priority.
|
||||
:keyword str api_version:
|
||||
The Storage API version to use for requests. Default value is '2019-07-07'.
|
||||
:keyword str secondary_hostname:
|
||||
The hostname of the secondary endpoint.
|
||||
"""
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
pass
|
||||
|
||||
def list_ownership(self, namespace, eventhub, consumergroup, **kwargs):
|
||||
pass
|
||||
|
||||
def list_checkpoints(self, namespace, eventhub, consumergroup, **kwargs):
|
||||
pass
|
||||
|
||||
def update_checkpoint(self, checkpoint, **kwargs):
|
||||
pass
|
||||
|
||||
def claim_ownership(self, ownershiplist, **kwargs):
|
||||
pass
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
# ------------------------------------
|
||||
# Copyright (c) Microsoft Corporation.
|
||||
# Licensed under the MIT License.
|
||||
# ------------------------------------
|
||||
|
||||
VERSION = "1.0.0b1"
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
-e ../../../tools/azure-sdk-tools
|
||||
../../core/azure-core
|
||||
-e ../../../tools/azure-devtools
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
[packaging]
|
||||
auto_update = false
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
[bdist_wheel]
|
||||
universal=1
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
from setuptools import setup, find_packages
|
||||
import os
|
||||
from io import open
|
||||
import re
|
||||
|
||||
# example setup.py Feel free to copy the entire "azure-template" folder into a package folder named
|
||||
# with "azure-<yourpackagename>". Ensure that the below arguments to setup() are updated to reflect
|
||||
# your package.
|
||||
|
||||
# this setup.py is set up in a specific way to keep the azure* and azure-mgmt-* namespaces WORKING all the way
|
||||
# up from python 2.7. Reference here: https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging
|
||||
|
||||
PACKAGE_NAME = "azure-eventhub-checkpointstoretable"
|
||||
PACKAGE_PPRINT_NAME = "Event Hubs checkpointer implementation with Azure Table Storage"
|
||||
|
||||
# a-b-c => a/b/c
|
||||
package_folder_path = PACKAGE_NAME.replace("-", "/")
|
||||
# a-b-c => a.b.c
|
||||
namespace_name = PACKAGE_NAME.replace("-", ".")
|
||||
|
||||
package_folder_path = "azure/eventhub/extensions/checkpointstoretable"
|
||||
|
||||
# Version extraction inspired from 'requests'
|
||||
with open(os.path.join(package_folder_path, "_version.py"), "r") as fd:
|
||||
version = re.search(r'^VERSION\s*=\s*[\'"]([^\'"]*)[\'"]', fd.read(), re.MULTILINE).group(1)
|
||||
if not version:
|
||||
raise RuntimeError("Cannot find version information")
|
||||
|
||||
with open("README.md", encoding="utf-8") as f:
|
||||
long_description = f.read()
|
||||
|
||||
setup(
|
||||
name=PACKAGE_NAME,
|
||||
version=version,
|
||||
description="Microsoft Azure {} Client Library for Python".format(PACKAGE_PPRINT_NAME),
|
||||
# ensure that these are updated to reflect the package owners' information
|
||||
long_description=long_description,
|
||||
long_description_content_type="text/markdown",
|
||||
url="https://github.com/Azure/azure-sdk-for-python",
|
||||
author="Microsoft Corporation",
|
||||
author_email="azuresdkengsysadmins@microsoft.com",
|
||||
license="MIT License",
|
||||
# ensure that the development status reflects the status of your package
|
||||
classifiers=[
|
||||
"Development Status :: 4 - Beta",
|
||||
"Programming Language :: Python",
|
||||
"Programming Language :: Python :: 2",
|
||||
"Programming Language :: Python :: 2.7",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.6",
|
||||
"Programming Language :: Python :: 3.7",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
],
|
||||
packages=find_packages(
|
||||
exclude=[
|
||||
"tests",
|
||||
# Exclude packages that will be covered by PEP420 or nspkg
|
||||
# This means any folder structure that only consists of a __init__.py.
|
||||
# For example, for storage, this would mean adding 'azure.storage'
|
||||
# in addition to the default 'azure' that is seen here.
|
||||
"azure",
|
||||
]
|
||||
),
|
||||
install_requires=[
|
||||
"azure-core<2.0.0,>=1.2.2",
|
||||
],
|
||||
extras_require={
|
||||
":python_version<'3.0'": ["azure-nspkg"],
|
||||
},
|
||||
project_urls={
|
||||
"Bug Reports": "https://github.com/Azure/azure-sdk-for-python/issues",
|
||||
"Source": "https://github.com/Azure/azure-sdk-python",
|
||||
},
|
||||
)
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
import pytest
|
||||
from azure.eventhub.extensions.checkpointstoretable import TableCheckpointStore
|
||||
|
||||
|
||||
def test_constructor():
|
||||
client = TableCheckpointStore()
|
||||
assert client is not None
|
||||
|
||||
|
|
@ -44,5 +44,7 @@ extends:
|
|||
safeName: azureeventhubcheckpointstoreblobaio
|
||||
- name: azure-eventhub-checkpointstoreblob
|
||||
safeName: azureeventhubcheckpointstoreblob
|
||||
- name: azure-eventhub-checkpointstoretable
|
||||
safeName: azureeventhubcheckpointstoretable
|
||||
- name: azure-mgmt-eventhub
|
||||
safeName: azuremgmteventhub
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче