Update documentation for the current release (2.3.15) (#505)
This commit is contained in:
SJ
GitHub
Родитель
5fa3f25d74
Коммит
99a0852a3d
|
|
@ -18,8 +18,7 @@ run all unit/integration tests and build a JAR.
|
|||
|
||||
### SBT Dependency
|
||||
|
||||
// https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark_2.11
|
||||
libraryDependencies += "com.microsoft.azure" %% "azure-eventhubs-spark" %% "2.3.13"
|
||||
libraryDependencies += "com.microsoft.azure" %% "azure-eventhubs-spark" %% "2.3.15"
|
||||
|
||||
## Filing Issues
|
||||
|
||||
|
|
|
|||
24
README.md
24
README.md
|
|
@ -30,17 +30,21 @@ By making Event Hubs and Spark easier to use together, we hope this connector ma
|
|||
#### Spark
|
||||
|Spark Version|Package Name|Package Version|
|
||||
|-------------|------------|----------------|
|
||||
|Spark 2.4|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.13%7Cjar)|
|
||||
|Spark 2.3|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.13%7Cjar)|
|
||||
|Spark 2.4|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|Spark 2.4|azure-eventhubs-spark_2.12|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.12%7C2.3.15%7Cjar)|
|
||||
|Spark 2.3|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|Spark 2.2|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.2.10%7Cjar)|
|
||||
|Spark 2.1|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.2.10%7Cjar)|
|
||||
|
||||
#### Databricks
|
||||
|Databricks Runtime Version|Artifact Id|Package Version|
|
||||
|-------------|------------|----------------|
|
||||
|Databricks Runtime 5.X|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.13%7Cjar)|
|
||||
|Databricks Runtime 4.X|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.13%7Cjar)|
|
||||
|Databricks Runtime 3.5|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.13%7Cjar)|
|
||||
|Databricks Runtime 6.X|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|Databricks Runtime 6.X|azure-eventhubs-spark_2.12|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.12%7C2.3.15%7Cjar)|
|
||||
|Databricks Runtime 5.X|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|Databricks Runtime 5.X|azure-eventhubs-spark_2.12|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.12%7C2.3.15%7Cjar)|
|
||||
|Databricks Runtime 4.X|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|Databricks Runtime 3.5|azure-eventhubs-spark_2.11|[](https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-eventhubs-spark_2.11%7C2.3.15%7Cjar)|
|
||||
|
||||
#### Roadmap
|
||||
|
||||
|
|
@ -51,11 +55,17 @@ There is an open issue for each planned feature/enhancement.
|
|||
### Linking
|
||||
|
||||
For Scala/Java applications using SBT/Maven project definitions, link your application with the artifact below.
|
||||
**Note:** See [Latest Releases](#latest-releases) to find the correct artifiact for your version of Apache Spark (or Databricks)!
|
||||
**Note:** See [Latest Releases](#latest-releases) to find the correct artifact for your version of Apache Spark (or Databricks)!
|
||||
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.11
|
||||
version = 2.3.13
|
||||
version = 2.3.15
|
||||
|
||||
or
|
||||
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.12
|
||||
version = 2.3.15
|
||||
|
||||
### Documentation
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,13 @@ Structured streaming integration for Azure Event Hubs is ultimately run on the J
|
|||
```
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.11
|
||||
version = 2.3.13
|
||||
version = 2.3.15
|
||||
|
||||
or
|
||||
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.12
|
||||
version = 2.3.15
|
||||
```
|
||||
|
||||
For Python applications, you need to add this above library and its dependencies when deploying your application.
|
||||
|
|
@ -49,6 +55,10 @@ connectionString = "YOUR.CONNECTION.STRING"
|
|||
|
||||
ehConf = {}
|
||||
ehConf['eventhubs.connectionString'] = connectionString
|
||||
|
||||
For 2.3.15 version and above, the configuration dictionary requires that connection string be encrypted.
|
||||
ehConf['eventhubs.connectionString'] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
|
||||
|
||||
```
|
||||
|
||||
Please read the [Connection String](#connection-string) subsection for more information on obtaining a valid
|
||||
|
|
@ -387,11 +397,11 @@ AMQP types need to be handled explicitly by the connector. Below we list the AMQ
|
|||
As with any Spark applications, `spark-submit` is used to launch your application. `azure-eventhubs-spark_2.11`
|
||||
and its dependencies can be directly added to `spark-submit` using `--packages`, such as,
|
||||
|
||||
./bin/spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 ...
|
||||
./bin/spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 ...
|
||||
|
||||
For experimenting on `spark-shell`, you can also use `--packages` to add `azure-eventhubs-spark_2.11` and its dependencies directly,
|
||||
|
||||
./bin/spark-shell --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 ...
|
||||
./bin/spark-shell --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 ...
|
||||
|
||||
See [Application Submission Guide](https://spark.apache.org/docs/latest/submitting-applications.html) for more details about submitting
|
||||
applications with external dependencies.
|
||||
|
|
|
|||
|
|
@ -18,12 +18,18 @@ partitions and Spark partitions, and access to sequence numbers and metadata.
|
|||
* [Deploying](#deploying)
|
||||
|
||||
## Linking
|
||||
For Scala/Java applications using SBT/Maven project defnitions, link your application with the following artifact:
|
||||
For Scala/Java applications using SBT/Maven project definitions, link your application with the following artifact:
|
||||
|
||||
```
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.11
|
||||
version = 2.3.13
|
||||
version = 2.3.15
|
||||
|
||||
or
|
||||
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.12
|
||||
version = 2.3.15
|
||||
```
|
||||
|
||||
For Python applications, you need to add this above library and its dependencies when deploying your application.
|
||||
|
|
|
|||
|
|
@ -18,12 +18,18 @@ Structured Streaming integration for Azure Event Hubs to read data from Event Hu
|
|||
* [Deploying](#deploying)
|
||||
|
||||
## Linking
|
||||
For Scala/Java applications using SBT/Maven project defnitions, link your application with the following artifact:
|
||||
For Scala/Java applications using SBT/Maven project definitions, link your application with the following artifact:
|
||||
|
||||
```
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.11
|
||||
version = 2.3.13
|
||||
version = 2.3.15
|
||||
|
||||
or
|
||||
|
||||
groupId = com.microsoft.azure
|
||||
artifactId = azure-eventhubs-spark_2.12
|
||||
version = 2.3.15
|
||||
```
|
||||
|
||||
For Python applications, you need to add this above library and its dependencies when deploying your application.
|
||||
|
|
@ -391,11 +397,11 @@ AMQP types need to be handled explicitly by the connector. Below we list the AMQ
|
|||
As with any Spark applications, `spark-submit` is used to launch your application. `azure-eventhubs-spark_2.11`
|
||||
and its dependencies can be directly added to `spark-submit` using `--packages`, such as,
|
||||
|
||||
./bin/spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 ...
|
||||
./bin/spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 ...
|
||||
|
||||
For experimenting on `spark-shell`, you can also use `--packages` to add `azure-eventhubs-spark_2.11` and its dependencies directly,
|
||||
|
||||
./bin/spark-shell --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 ...
|
||||
./bin/spark-shell --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 ...
|
||||
|
||||
See [Application Submission Guide](https://spark.apache.org/docs/latest/submitting-applications.html) for more details about submitting
|
||||
applications with external dependencies.
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче