33ee66e23a
Several bug fixes and improvements for 2.3.14 release ( #465 )
...
* [Bug fix] fix IO pipe issue that causes a blocking issue on send calls
* [Bug fix] fix broken TCP connection issue that causes prolonged delay in receiving events
* [Configuration] provide a configuration knob to specify whether an epoch receiver should be used. Default value is true.
* [Tracing] add a Spark task ID to log messages and emit latency data for receive calls
2019-10-23 09:44:13 -07:00
d5ad0d6caa
Update version number for new release (2.3.13) ( #451 )
2019-07-26 13:29:19 -07:00
f10a85f03e
add event hub name to the OffsetRange toString function so that it is present in the spark metadata. add simple unit test for validation of the string ( #450 )
2019-06-27 08:41:01 -07:00
64d3fb5eca
small misstype in EventPosition ( #449 )
2019-06-05 22:51:46 -07:00
1ad742d461
Update version number for new release (2.3.12) ( #446 )
2019-05-10 15:13:49 -07:00
0e909b8cc0
Fix typo Strucutred -> Structured ( #447 )
2019-05-10 12:10:43 -07:00
081e76b0cb
Update version numbers in the document and mark 2.3.11 as Preview release ( #445 )
2019-05-02 18:08:00 -07:00
d368d7c6b0
Update version numbers for new release (2.3.11 & 2.2.11) ( #443 )
2019-05-01 21:45:19 -07:00
d9f4fe02e5
performance improvement for EventHubs writer ( #441 )
2019-04-12 09:30:45 -07:00
d5e2734c47
Updated the doc - new settings (prefetchCount & threadPoolSize) ( #439 )
2019-04-08 17:43:46 -07:00
f5fe40d994
Prepare 2.3.10 & 2.2.10 release ( #438 )
2019-04-08 14:52:51 -07:00
08c0c58650
Update dpendency - Event Hubs client SDK (2.2.0 --> 2.3.0) ( #437 )
2019-04-08 09:36:54 -07:00
90bd0e8608
Update maximum prefetch count setting and improve logging ( #436 )
2019-04-01 19:58:28 -07:00
2a6a5b9f28
Update README.md ( #434 )
2019-03-19 17:30:32 -07:00
3986bae133
Update README.md ( #429 )
2019-01-28 10:58:18 -08:00
420e4bd2a2
Prepare 2.3.9 & 2.2.9 release ( #427 )
2019-01-18 15:53:32 -08:00
9f7c9d0818
Handle a case where messages in an offet range were already expired ( #426 )
2019-01-18 13:31:51 -08:00
cdea9ae746
Prepare 2.3.8 & 2.2.8 release and update Java client SDK dependency ( #425 )
2019-01-15 19:47:23 -08:00
0a59666dda
Provide a knob to specify the size of thread pool and recreate a receiver if receive call fails for an extended period of time ( #423 )
2019-01-14 09:47:44 -08:00
6932f56d0a
Release 2.2.7 for spark runtime 2.1 and 2.2 ( #421 )
2019-01-05 22:07:40 -08:00
86fb6db24d
Prepare for 2.3.7 release ( #420 )
...
* Prepare for 2.3.7 release
* Enable ReceiverTimeout and PrefetchCount configuraion
* Update Java client dependency
2019-01-05 18:25:23 -08:00
498746f308
Fix future timeout issue ( #419 )
2019-01-02 17:45:34 -08:00
53c1d79433
Provide a knob to specify prefetch count for the underlying receiver. ( #417 )
...
* Provide a knob to specify prefetch count for the underlying receiver
2018-12-11 18:08:26 -08:00
2bdfa15d40
Take the actual sequence numbers, instead of relying on Seq ( #409 )
...
For the tests we want to use the actual sequence number instead of
using the size of the underlying Seq. In real world, the consumer
group of the event hub might nog start at zero, but at a high number
2018-11-28 17:47:56 -08:00
0f75b0e9fe
Use OperationTimeout config value instead of fixed timeout when specifying timeout for internal API calls ( #415 )
...
* Use OperationTimeout instead of fixed timeout for retries
* fix format issue
* update comments for internalOperationTimeout API and declare it lazy val
* Update API doc
2018-11-28 17:36:40 -08:00
1c198ba39a
Prepare for 2.3.6 & 2.2.6 release ( #407 )
2018-11-08 09:06:05 -08:00
e0faa35fbc
Update logging for EventHubsConf class ( #406 )
2018-11-05 23:21:20 -08:00
58750b7b31
docs - updating default starting position ( #402 )
2018-10-19 13:56:42 -07:00
69fa95729a
spark 2.2.5 readme update ( #401 )
2018-10-19 13:48:57 -07:00
1c42c2f45b
prep for 2.3.5 release ( #399 )
2018-10-19 11:53:46 -07:00
57d083f2f8
add all system properties to schema ( #395 )
...
* add deviceID to schema
* add all properties
* Removing duplicated keys from SystemProperties map
2018-10-11 11:07:11 -07:00
45fb7b6595
Add foreachwriter docs ( #397 )
...
* Add foreachwriter docs
* update
2018-10-07 10:07:46 -07:00
c19dff2026
Add unit tests for RetryUtils ( #396 )
...
* Add unit tests for RetryUtils
* Address code review comments
2018-10-07 09:15:31 -07:00
7ceae44f3b
Receive calls one at a time to avoid overloading thread pool ( #394 )
...
* Receive calls one at a time to avoid overloading thread pool
* bumping to 1.2.0 of java client
2018-10-03 16:24:26 -07:00
22d7936c61
2.3.4 and 2.2.4 release ( #387 )
2018-09-10 15:06:55 -07:00
34f5488e56
Add consumer group to exception messaging in cached receiver ( #386 )
2018-09-10 07:59:18 -07:00
43717e0e90
Redo check cursor ( #384 )
...
* Redo check cursor
* Update to 2.3.4 snapshot
2018-09-07 14:32:41 -07:00
0733567c2b
2.3.3 and 2.2.3 release ( #381 )
2018-09-04 20:20:05 -07:00
4646ee2eb4
Increased strictness in receive calls ( #380 )
2018-09-02 16:25:02 -05:00
61aba38334
Properties can be added in EventHubsSink ( #375 )
...
* Properties can be added in EventHubsSink
* doc update
2018-08-20 14:34:05 -05:00
f1bb4ea7e9
Cached receivers are async ( #377 )
...
* Cached receivers are async
* avoid receiver creation and deletion if cursor isn't correct
2018-08-20 12:10:42 -05:00
334255e6a3
Retry receive when it returns null in translate ( #369 )
2018-08-07 23:06:05 -07:00
c9112b0420
Handled DescribedType in properties ( #373 )
2018-08-07 23:01:01 -07:00
68b91a21d0
Change the ehClient to a lazy variable ( #368 )
2018-08-01 10:10:58 -07:00
0fe7d8f5e0
Simplify configuration and creation of the client ( #359 )
...
* Dont read messages that are already pruned by EventHub
In the getPartitions method of the EventHubsRDD class we check if the
offsets are still valid. It is possible that the retention has kicked
in and the messages are no longer available on the bus.
For more info, refer to this issue:
https://github.com/Azure/azure-event-hubs-spark/issues/313
Did some minor refactoring:
- Made the clientFactory static so we don't need to pass this constructor
around
- Changed the signature of allBoundedSeqNos from a Seq to a Map, since
the partitionId is unique and later on in the code it is also converted
to a map.
- Removed the trim method, since passing EventHub config keys to Spark
does not do any harm. Without this change, the tests are failing since
they are not being switched to the simulator.
* Remove the offset calculation
* Bump version to 2.3.3-SNAPSHOT
* Restore trimmed config when creating a EventHubsRDD
2018-07-29 14:19:17 -07:00
47b6d31c77
CachedReceiver properly adjusts cursor ( #364 )
2018-07-25 15:01:32 -07:00
8c11923a24
Split the classes from EventHubsTestUtils.scala in seperate files ( #360 )
...
The EventHubsTestUtils.scala was very long, therefore it is better
to split the classes/objects into seperate files
2018-07-24 12:33:39 -07:00
599a979cf2
Make the code more Scala idomatic ( #356 )
...
There is a lot of mutable code in the source, we want to get rid of it
2018-07-23 10:15:16 -07:00
e46c031fb8
Typo fix in docs ( #358 )
...
Fix a typo in the Pyspark structured streaming docs
2018-07-22 15:08:22 -07:00
7ac76d8f49
update to 2.2 in readme ( #355 )
2018-07-06 14:29:02 -07:00
SJ
jgiardin
Dirceu Semighini Filho
gison93
Fokko Driesprong
Sabee Grewal
Quynh Anh "Emma" Nguyen
Sander Ardinois