Граф коммитов

69 Коммитов

Автор SHA1 Сообщение Дата
nyaghma 564267dd12
Update docs and add a Q/A in FAQ (#481)
* Update the current version in the README
* Add a question-answer to FAQ
* update version and default starting position in docs
2020-03-31 15:03:11 -07:00
SJ d5ad0d6caa
Update version number for new release (2.3.13) (#451) 2019-07-26 13:29:19 -07:00
Dirceu Semighini Filho 64d3fb5eca small misstype in EventPosition (#449) 2019-06-05 22:51:46 -07:00
gison93 0e909b8cc0 Fix typo Strucutred -> Structured (#447) 2019-05-10 12:10:43 -07:00
SJ 081e76b0cb
Update version numbers in the document and mark 2.3.11 as Preview release (#445) 2019-05-02 18:08:00 -07:00
SJ d368d7c6b0
Update version numbers for new release (2.3.11 & 2.2.11) (#443) 2019-05-01 21:45:19 -07:00
SJ d5e2734c47
Updated the doc - new settings (prefetchCount & threadPoolSize) (#439) 2019-04-08 17:43:46 -07:00
SJ f5fe40d994
Prepare 2.3.10 & 2.2.10 release (#438) 2019-04-08 14:52:51 -07:00
SJ 420e4bd2a2
Prepare 2.3.9 & 2.2.9 release (#427) 2019-01-18 15:53:32 -08:00
SJ cdea9ae746
Prepare 2.3.8 & 2.2.8 release and update Java client SDK dependency (#425) 2019-01-15 19:47:23 -08:00
SJ 86fb6db24d
Prepare for 2.3.7 release (#420)
* Prepare for 2.3.7 release
* Enable ReceiverTimeout and PrefetchCount configuraion
* Update Java client dependency
2019-01-05 18:25:23 -08:00
SJ 1c198ba39a
Prepare for 2.3.6 & 2.2.6 release (#407) 2018-11-08 09:06:05 -08:00
Sabee Grewal 58750b7b31
docs - updating default starting position (#402) 2018-10-19 13:56:42 -07:00
Sabee Grewal 1c42c2f45b
prep for 2.3.5 release (#399) 2018-10-19 11:53:46 -07:00
Sabee Grewal 57d083f2f8
add all system properties to schema (#395)
* add deviceID to schema

* add all properties

* Removing duplicated keys from SystemProperties map
2018-10-11 11:07:11 -07:00
Sabee Grewal 45fb7b6595
Add foreachwriter docs (#397)
* Add foreachwriter docs

* update
2018-10-07 10:07:46 -07:00
Sabee Grewal 22d7936c61 2.3.4 and 2.2.4 release (#387) 2018-09-10 15:06:55 -07:00
Sabee Grewal 0733567c2b
2.3.3 and 2.2.3 release (#381) 2018-09-04 20:20:05 -07:00
Sabee Grewal 61aba38334
Properties can be added in EventHubsSink (#375)
* Properties can be added in EventHubsSink

* doc update
2018-08-20 14:34:05 -05:00
Sander Ardinois e46c031fb8 Typo fix in docs (#358)
Fix a typo in the Pyspark structured streaming docs
2018-07-22 15:08:22 -07:00
Sabee Grewal 3682b30e7d
Prep for 2.3.2 release (#354) 2018-07-05 11:46:32 -07:00
Sabee Grewal 2c79b5f05f
Add checkpointing info to ss docs (#338)
* Add checkpointing info to ss docs

* Update spark-streaming-eventhubs-integration.md
2018-06-08 14:52:54 -07:00
Sabee Grewal 0e7c062608
add properties to schema (#328)
* add properties to schema

* make tasks serializable

* Cleaning up simulated send APIs

* add properties to simulated send APIs

* properties unit tests in relation and source

* support char and proton-j types in properties

* adding comments and docs

* updating toc in docs
2018-06-01 17:46:51 -07:00
Edouard Poitras c2abbd96b5 Update structured-streaming-pyspark.md (#318)
* Update structured-streaming-pyspark.md

Elaboration on the Connection String section.

Signed-off-by: Edouard Poitras <eddie@quantumcyberdefence.com>

* Update structured-streaming-pyspark.md
2018-05-17 15:04:25 -07:00
Sabee Grewal ad08e02fda
Update structured-streaming-pyspark.md 2018-04-18 08:18:58 -07:00
Ville Rantala 5d71c89058 Add note about connecting to IoT Hub (#310)
* Add note about connecting to IoT Hub

* Update structured-streaming-pyspark.md
2018-04-18 07:31:58 -07:00
Sabee Grewal 46bfd57168
Add partition to schema (#301)
* Add partition to schema

* Sort offset ranges by partition in RDD

* fix flaky EventPosition CIT
2018-03-27 13:43:22 -07:00
Basil Hariri f611ecf63a Creating new PySpark docs (#297)
* Adding python docs

* removed old python doc

* Fixed title, added linking, added r/w for batch queries to pyspark doc
2018-03-26 15:39:56 -07:00
Basil Hariri 22721a4911 Update structured-streaming-eventhubs-integration.md (#296) 2018-03-23 14:57:19 -07:00
Basil Hariri 1964ec34d1 Update structured-streaming-eventhubs-integration.md (#295) 2018-03-23 14:54:04 -07:00
Sabee Grewal ca17f585a2
Updating to 2.3.1 (#288) 2018-03-22 20:30:20 -07:00
Sabee Grewal 676b486acb
Add Managing Throughput section to docs (#290)
* Add Managing Throughput section to docs

* typo fixes, minor additions

* Update spark-streaming-eventhubs-integration.md

* Update spark-streaming-eventhubs-integration.md
2018-03-22 15:32:49 -07:00
Sabee Grewal 202b857bec
cleanup (#287)
* cleanup

* Update spark-streaming-eventhubs-integration.md

* Update spark-streaming-eventhubs-integration.md

* Update spark-streaming-eventhubs-integration.md
2018-03-21 19:21:57 -07:00
Basil Hariri 37495f7ca3 Created TOC for streaming docs (#283)
* Test toc for spark streaming

* Removed lowest subtopics from example TOC

* Added TOC to structured streaming, fixed links to point to Azure github in spark streaming TOC

* Update spark-streaming-eventhubs-integration.md

* Update structured-streaming-eventhubs-integration.md

* Testing link fix

* Fixed links
2018-03-21 10:57:15 -07:00
Sabee Grewal 0470588bca
Updating doc links (#276)
* Updating doc links

* Cleaning up wording
2018-03-15 10:08:46 -07:00
Sabee Grewal f8e3efe934
Minor cleanup and documentation edits (#270)
* Adjusting select expressions

* Contributor guide edits

* PR template edits

* README edits

* EventPosition cleanup

* README reflects support for 2.1 and 2.2

* Removing duplicate naming and unneeded comments

* Additional README and Contributing edits
2018-03-12 11:27:12 -07:00
Sabee Grewal 31b69adede
Adding new call to NameAndPartition (#267) 2018-03-08 12:59:19 -08:00
Sabee Grewal bea9075347
Bug fixes in docs (#264) 2018-03-08 11:45:47 -08:00
Sreeram Garlapati da175a86a5 Change offset column from Long Type to String Type (#253)
* Fix offset type in EventDataSchema

* fix type in readme
2018-03-06 16:37:34 -08:00
Sabee Grewal e16631bc4e
Changing body column to binary type (#246) 2018-03-05 14:05:17 -08:00
Basil Hariri 4bb7de5238 Removed references to EventData in structured streaming doc (#245) 2018-03-05 12:13:52 -08:00
Sreeram Garlapati 298056b089 nit doc fixes (#242) 2018-03-05 10:46:34 -08:00
Basil Hariri 57170a3983 Fixing typos in documentation (#241)
* Fixed minor typos in spark-streaming-eventhubs-integration.md

* Fixed minor typos in structured-streaming-eventhubs-integration.md
2018-03-02 15:43:41 -08:00
Sabee Grewal a3994ce6b6
Add default maxEventsPerTrigger (#237)
* Add default maxEventsPerTrigger

* Changing to 1000 per partition

* Change Some to Option
2018-03-02 12:21:05 -08:00
Sabee Grewal 73f486e535
Library re-write (For Spark 2.3) (#229)
* added EventHubsConf but haven't integrated it yet. build is stable!

* putting a pin in these EventHubsConf changes to focus on Spark 2.2

* WIP: implementation and tests complete. Need to fix issue related to Spark 2.2

* updating connector to work with Spark 2.2

* minor update to comments

* setting timeouts in EventHubClientWrapper

* change EventHubsConf.copy to EventHubsConf.clone

* temporarily disabling tests. progress tracker tests are being problematic and they are going to be removed in the next phase of cleanup

* driver-side translation added. dstream re-written. rdd re-written. configuration documentation added.

* EventHubsSource partial rewrite complete. Committing progress b/c need hit pause and fix a bug in an older version

* EventHubsSource re-write complete. Moving on to testing. Re-write was substantial, so I expect further changes will be needed as we fine tune the connector

* Fixed client, starting tests

* moving all client functionality into the client. added simulated eventhubs. gonna starting really reworking the tests now

* cleaned out old tests. updated code. everything is building, no tests yet.

* updated EventHubsConfSuite, all tests passing

* test utils set up, first RDD is passing

* adding RDD tests

* finalized sequence number support in eventhubsconf, dstream, and source

* basic stream tests done, moving to checkpointing tests

* finished DStream tests. moving to Source tests

* tests for EventHubsSourceOffset and JsonUtils

* removing excessive stack trace printing

* first few source tests. running into a cast exception due to EH Java Client, gonna take care of that now

* fixing how source handles EnqueueTime from EventData

* added maxSeqNoPerTrigger and corresponding source tests

* additional Source tests

* decoupled simulated client from simulated eventhubs. extended simulated eventhubs to allow sending events

* rdd, dstream, and source tests adapted to new simulated eventhubs

* adding AddEventHubsData integration tests. switching machines.

* modifying eventhubsclient to avoid false positive data loss reports

* additional structured streaming integration tests

* adding support for national and private clouds via setDomainName in EventHubsConf

* added final integration tests for struct streaming

* EnqueuedTime is converted to java.sql.Timestamp

* removing unused imports

* moving to eventhubs java client 1.0.0

* Remove isValid from EventHubsConf

* maxRatePerPartition refactoring

* Client refactoring - signature changes and removing unused methods

* EventHubsConf refactoring

* Common package is removed

* dropping default max rate

* Support for JavaRDD and JavaInputDStream

* Rename Position to EventPosition

* misc cleanup

* Support multiple simulated eventhubs at once

* remove sql containsProps and userDefinedKeys options

* parallelized all loops in EventHubsClient.translate

* removing unecessary comments

* adding javadoc comments

* conn str builder tests

* EventHubsConf tests added

* Minor bug fixes and EventPosition serialization issue is fixed

* Simulated client is enabled in tests

* Moving non-util files out of utils package

* ClientWrapper fix

* Minor bug fixes in tests

* Moved to Spark 2.3, all tests passing

* EventPosition bug fix

* Receive until we do get null, only make API call for partition count once

* moving defaults into package.scala

* removing out of date docs, adding structured streaming integration guide

* spark streaming integration guide

* Removing old information from docs

* Updated PySpark docs

* updating doc name

* Updating minor issues in docs. Added experimental tag to four apis in eventhubsconf

* Adding support for batch styled queries in structured streaming

* Update struct streaming docs to reflect new batch query support

* docs/README formatting

* doc fomratting

* add batch style query code sample in docs

* EventData: remove inclusive flag from public api. Starts are always inclusive, ends are always exclusive

* Updating public apis to take NameAndPartition instead of PartitionId

* Fixing javadoc issues in EventPosition

* updating readme

* updating templates for pull requests, issues, and contriubting

* moving test resource to test directory

* renaming EventHubsClientWrapper to EventHubsClient

* fixing access issues in NameandPartition

* reorganizing test resources

* Accomodating breaking changes in java client

* Additional tracing in translate method

* Client connection pooling and thread pooling first draft

* Minor bug fix to connection pool

* remove failOnDataLoss option

* Adding EventHubsSink

* Adding send functionality to TestUtils

* First batched writes passing

* More unit tests for EventHubsRelation and EventHubsSink

* Additional Sink tests

* Final Sink test updates

* Adding Sink documentation to integration guide

* Adding databricks docs

* remvoing concurrent jobs limit in spark streaming

* Check for EventData expiration each batch

* Rebase

* Adding preferred location in Spark Streaming and Struct Streaming

* concurrency bug fix in EVentHubsClient

* Minor logging fix

* retry client create until successful

* Update structured-streaming-eventhubs-integration.md

* Update azure_eventhubs_support.md

* Update spark-streaming-eventhubs-integration.md

* Update structured-streaming-eventhubs-integration.md

* Update azure_eventhubs_support.md

* Update README.md

* add toString for simulated eventhubs

* Update structured-streaming-eventhubs-integration.md

* Update spark-streaming-eventhubs-integration.md

* Update azure_eventhubs_support.md

* Updating docs - typo fixes and reorganizing

* fixing NPE in RDD

* Moving to proper Spark 2.3.0 release and Java client 1.0.0 release

* Enabling unit and integration tests in Travis

* Updating CONTRIBUTING.md

* additional traces in client pool
2018-03-02 10:33:18 -08:00
xavier geerinck 573c59c814 Fix links in IoT doc (#228)
We need to provide the protocol, since else github will show github.com/.../www.ms.portal.azure.com which is of course wrong
2018-02-02 12:32:18 -08:00
Lena d32ba18138 Removing a typo at the end of the code line (#227) 2018-01-31 10:29:10 -08:00
romitgirdhar 9ad814195a Sample for using the eventhubs-spark JAR in Jupyter notebook using PySpark3 (#215)
* Adding a sample for running a job using the eventhubs-spark JAR in Jupyter notebook using PySpark3

* Changing the location of the .md file.
2017-11-29 16:40:18 -08:00
Andrew Mills 6f26334717 Grammatical fixes to direct_stream.md (#209)
Made a few small changes to the grammar for clarification.
2017-11-08 11:56:14 -08:00
Sabee Grewal 20510f9f3e initial repo clean up: first draft of new README, unfinished docs/README and CONTRIBUTING, removed javadocs folder 2017-10-10 09:53:29 -07:00