зеркало из
1
0
Форкнуть 0

Add reference to "Release History" section in the "Major version specifics" section (which is moved from section 13 to 15)

This commit is contained in:
Yihezkel Schoenbrun 2020-11-12 21:05:10 +02:00
Родитель b99ec9a56a
Коммит b8a101e10b
1 изменённых файлов: 39 добавлений и 36 удалений

Просмотреть файл

@ -16,9 +16,9 @@ This repository contains the source code of the Kafka Connect Kusto sink connect
[10. Distributed deployment details](README.md#10-distributed-deployment-details)<br>
[11. Test drive the connector - distributed mode](README.md#11-test-drive-the-connector---distributed-mode)<br>
[12. Apache Kafka version - Docker Kafka Connect base image version - Confluent Helm chart version related](README.md#12-apache-kafka-version---docker-kafka-connect-base-image-version---confluent-helm-chart-version-related)<br>
[13. Major version specifics](README.md#13-major-version-specifics)<br>
[14. Other](README.md#14-other)<br>
[15. Need Support?](README.md#15-need-support)<br>
[13. Other](README.md#13-other)<br>
[14. Need Support?](README.md#14-need-support)<br>
[15. Major version specifics](README.md#15-major-version-specifics)<br>
[16. Release History](README.md#16-release-history)<br>
[17. Contribute](README.md#17-contribute)<br>
@ -279,7 +279,7 @@ This involves having the connector plugin jar in /usr/share/java of a Kafka Conn
### 7.2. Distributed Kafka Connect deployment mode
Distributed Kafka Connect essentially involves creation of a KafkaConnect worker cluster as shown in the diagram below.<br>
- Azure Kubernetes Service is a great infrastructure for the connect cluster, due to its managed and scalabale nature
- Azure Kubernetes Service is a great infrastructure for the connect cluster, due to its managed and scalable nature
- Kubernetes is a great platform for the connect cluster, due to its scalable nature and self-healing
- Each orange polygon is a Kafka Connect worker and each green polygon is a sink connector instance
- A Kafka Connect worker can have 1..many task instances which helps with scale
@ -461,7 +461,40 @@ Similarly, we recommend leveraging the right version of the Helm chart. [Conflu
<hr>
## 13. Major version specifics
## 13. Other
### 13.1. Feedback, issues and contribution
The connector plugin is open source. We welcome feedback, and contribution. Log an issue, in the issues tab as needed. See section 14.
### 13.2. Scaling out/in
- Connector tasks can be scaled out per Kafka Connect worker by pausing, editing and resuming the connector cluster
- When tasks per worker are maxed out, Azure Kubernetes Service cluster can be scaled out for more nodes, and Kafka Connect workers can be provisioned on the same
### 13.3. Sizing
- Confluent recommends Kafka Connect workers with minimum of 4 cores and 16 GB of RAM
- Start with 3 workers (3 AKS nodes), and scale horizontally as needed
- Number of tasks should ideally be equal to the number of Kafka topic partitions, not more
- Play with the number of tasks, wokers, nodes till you see the performance you desire
### 13.4. Performance tuning
- Kafka topic: number of partitions should be tuned for performance
- Connectors: AKS right-sizing, connector tasks right-sizing, configure the right values for flush.size.bytes and flush.interval.ms
- Kusto: Right-size Kusto cluster for ingestion (SKU and node count), tune the batch ingestion policy
- Format: Avro (with schema registry) and CSV perform more-or-less similarly from tests done
### 13.5. Upgrading to version 1.x from prior versions
To upgrade, you would have to stop the connector tasks, recreate your connect worker Docker image to include the latest jar, update the sink properties to leverage the renamed and latest sink properties, reprovision the connect workers, then launch the copy tasks. You can use the consumer.override* feature to manipulate offset to read from.
<hr>
## 14. Need Support?
- **Found a bug?** Please help us fix it by thoroughly documenting it and [filing an issue](https://github.com/Azure/kafka-sink-azure-kusto/issues/new).
- **Have a feature request?** Please post it on [User Voice](https://feedback.azure.com/forums/915733-azure-data-explorer) to help us prioritize
- **Have a technical question?** Ask on [Stack Overflow with tag "azure-data-explorer"](https://stackoverflow.com/questions/tagged/azure-data-explorer)
- **Need Support?** Every customer with an active Azure subscription has access to [support](https://docs.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request) with guaranteed response time. Consider submitting a ticket and get assistance from Microsoft support team
## 15. Major version specifics
With version 1.0, we overhauled the connector. The following are the changes-
1. We renamed some properties for consistency with standards
2. Added support for schema registry
@ -478,39 +511,9 @@ Here is our [blog post](https://techcommunity.microsoft.com/t5/azure-data-explor
To upgrade, you would have to stop the connector tasks, recreate your connect worker Docker image to include the latest jar, update the sink properties to leverage the renamed and latest sink properties, reprovision the connect workers, then launch the copy tasks.
<hr>
## 14. Other
For information about what changes are included in each release, please see the [Release History](README.md#16-release-history) section of this document.
### 14.1. Feedback, issues and contribution
The connector plugin is open source. We welcome feedback, and contribution. Log an issue, in the issues tab as needed. See section 15.
### 14.2. Scaling out/in
- Connector tasks can be scaled out per Kafka Connect worker by pausing, editing and resuming the connector cluster
- When tasks per worker are maxed out, Azure Kubernetes Service cluster can be scaled out for more nodes, and Kafka Connect workers can be provisioned on the same
### 14.3. Sizing
- Confluent recommends Kafka Connect workers with minimum of 4 cores and 16 GB of RAM
- Start with 3 workers (3 AKS nodes), and scale horizontally as needed
- Number of tasks should ideally be equal to the number of Kafka topic partitions, not more
- Play with the number of tasks, wokers, nodes till you see the performance you desire
### 14.4. Performance tuning
- Kafka topic: number of partitions should be tuned for performance
- Connectors: AKS right-sizing, connector tasks right-sizing, configure the right values for flush.size.bytes and flush.interval.ms
- Kusto: Right-size Kusto cluster for ingestion (SKU and node count), tune the batch ingestion policy
- Format: Avro (with schema registry) and CSV perform more-or-less similarly from tests done
### 14.5. Upgrading to version 1.x from prior versions
To upgrade, you would have to stop the connector tasks, recreate your connect worker Docker image to include the latest jar, update the sink properties to leverage the renamed and latest sink properties, reprovision the connect workers, then launch the copy tasks. You can use the consumer.override* feature to manipulate offset to read from.
<hr>
## 15. Need Support?
- **Found a bug?** Please help us fix it by thoroughly documenting it and [filing an issue](https://github.com/Azure/kafka-sink-azure-kusto/issues/new).
- **Have a feature request?** Please post it on [User Voice](https://feedback.azure.com/forums/915733-azure-data-explorer) to help us prioritize
- **Have a technical question?** Ask on [Stack Overflow with tag "azure-data-explorer"](https://stackoverflow.com/questions/tagged/azure-data-explorer)
- **Need Support?** Every customer with an active Azure subscription has access to [support](https://docs.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request) with guaranteed response time. Consider submitting a ticket and get assistance from Microsoft support team
## 16. Release History
| Release Version | Release Date | Changes Included |
| --------------- | ------------ | ---------------- |