9dc4674e93
Merge pull request #507 from xiongrenyi/DirectStream
...
unifying KafkaUtils.CreateDirectStream API
2016-07-20 18:58:53 -07:00
36e8fbbf2c
Update configuration-mobius.md
2016-07-20 14:59:44 -07:00
df00f417d7
Update configuration-mobius.md
2016-07-20 14:58:13 -07:00
2bceabb2a0
Update configuration-mobius.md
...
add spark.mobius.streaming.kafka.numPartitions.<topicName>[.<clusterId>]
2016-07-20 14:51:27 -07:00
66f6f60571
unifying KafkaUtils.CreateDirectStream API
2016-07-20 14:05:06 -07:00
3d541e942e
Change time unit of streaming back to millisecond ( #504 )
...
* Change time unit of streaming back to millisecond
* Update Microsoft.Spark.CSharp.Adapter.Doc.XML and Mobius_API_Documentation.md
* Change according to review comments
2016-07-19 19:16:40 -07:00
ae1494e547
Merge pull request #505 from hebinhuang/NativeCodeCov
...
Implement automatic code coverage reports for c++ code
2016-07-19 13:59:30 -07:00
38a9eb2069
Implement automatic code coverage reports for c++ code
2016-07-19 12:56:55 -07:00
bf210f47f0
Add CSharpInputDStream to enable control of data injection by C# code ( #500 )
...
* Add CSharpInputDStream
* Add Unit test for CSharpInputDStream
2016-07-17 18:46:26 -07:00
bf84818d46
Use log4net as default logger service as current default logger cannot suppress undesired lower level messages. ( #501 )
2016-07-16 22:24:48 +08:00
94420c1879
Merge pull request #502 from hebinhuang/RioSockBug
...
Fix bugs in RioSocketWrapper and SocketStream that cause vstest failed
2016-07-15 10:30:08 -07:00
41a91b03b0
Fix bugs in RioSocketWrapper and SocketStream that cause vstest failed
2016-07-14 21:08:52 -07:00
f31c92f7f0
Downgrade the log level of WeakObjectManager.cs to reduce logs. ( #499 )
2016-07-13 14:23:20 +08:00
46c08e45a5
Merge pull request #492 from tawan0109/local_checkpoint
...
Rebuild state for DStream UpdateStateByKey operation by replaying batches
2016-07-12 11:30:25 +08:00
ffe223c2f9
Merge pull request #483 from hebinhuang/ManagedRioSock
...
Add RIOSocketWrapper and SaeaSocketWrapper
2016-07-10 07:32:20 -07:00
b6ec24119e
Add RIOSocketWrapper and SaeaSocketWrapper
2016-07-09 22:00:10 -07:00
3b67b2a6a7
Avoid RDD checkpoint for DynamicPartitionKafkaInputDStream by replaying last N batches
2016-07-08 17:52:47 +08:00
d3999ca43d
Add option to turn on/off local checkpoint for CSharpStateDStream
2016-07-08 17:51:13 +08:00
e75eb189e0
Download big file to temp one to avoid possible download failure and use corrupted file. ( #490 )
2016-07-08 09:51:16 +08:00
db95eb5de8
Updating maven version
2016-07-05 12:27:00 -07:00
dd376179ac
updating maven version
2016-07-05 11:33:22 -07:00
d78493bc42
updating build version to reflect Spark version
2016-07-05 11:32:44 -07:00
754a530961
updating maven version
2016-07-05 11:31:56 -07:00
b15163657a
When check and release unused JVMObjectTracker objects, change the speed adaptively according to current objects count. ( #488 )
2016-07-04 00:12:10 -07:00
33b87b4b0f
Merge pull request #487 from xiongrenyi/DirectReceiver
...
add validation to tqin's for ReduceByKeyAndWindow
2016-07-02 08:46:56 -07:00
9515b44f26
in addition to tqin's fix of a typo and removing prevFunc in ReduceByKeyAndWindow, add numPartition explicitly and add validation to his sample
2016-07-01 16:49:24 -07:00
4004001732
fixes to run Mobius in Linux
...
* allowing to specify worker path for all modes - this will help specifying prefix script for worker exe in CentOS
* fixing issue 467 - with this change Mobius can run in YARN cluster mode
* updating notes since YARN cluster mode is supported with the fix to issue #467
* added check to run OS-specific path tests
2016-07-01 11:21:33 -07:00
2337e7cee2
upgrading to Spark 1.6.2
2016-07-01 09:59:26 -07:00
67609769ae
Merge pull request #472 from tawan0109/update_state_by_key
...
Operator updateStateByKey supports initial state RDD
2016-06-30 08:28:45 +08:00
1cde355f3b
Add unit test cases for UpdateStateByKey operator
2016-06-29 14:37:19 +08:00
64ee6a8437
Merge branch 'master' of https://github.com/microsoft/Mobius into update_state_by_key
2016-06-29 12:08:54 +08:00
4f39658e4e
updates to documentation for Mobius in Linux
2016-06-28 16:15:17 -07:00
66f0132c14
updating links
2016-06-27 22:35:32 -07:00
a71b525ce6
Merge pull request #466 from qintao1976/ParallelKafkaDataFetcher
...
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-24 11:08:33 +08:00
dd1122ff99
Solve the Unit test problem caused by concurrent access of StringBuilder
2016-06-23 19:18:26 -07:00
62474b104b
Merge pull request #482 from xiongrenyi/DirectReceiver
...
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 15:52:04 -07:00
7a24a79aa5
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 14:51:36 -07:00
7d6ea24a8d
Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side ( #476 )
...
* Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side
1.Java-side: https://github.com/Microsoft/Mobius/blob/master/scala/src/main/org/apache/spark/api/csharp/CSharpBackendHandler.scala#L269
JVMObjectTracker keep a HashMap[String, Object] which is [id, Java-object]
2.CSharp-side :
1) JvmObjectReference remember the id of the object: https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmObjectReference.cs#L20
2) So JvmBridge can call java object's method https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmBridge.cs#L69
Potential memory leak can happen in JVMObjectTracker.
To solve this, track the garbage collection in CSharp side, get the id, release JVMObjectTracker's HashMap.
2016-06-23 21:42:47 +08:00
0359d66548
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-22 22:40:00 -07:00
27916910ad
added code of conduct
2016-06-21 14:39:47 -07:00
e74253eb9d
Merge pull request #479 from tawan0109/dev
...
Disabled LogFileAppender for CSharpWorker
2016-06-21 19:37:42 +08:00
7d6f569c5e
Disabled LogFileAppender for CSharpWorker
2016-06-21 10:13:52 +08:00
6607411acf
Merge pull request #478 from jthelin/fix-run-samples-script-1
...
Fix comment line prefix in run_samples.cmd script.
2016-06-20 11:48:44 -07:00
38f42d2554
Fix comment line prefix in run_samples.cmd script.
2016-06-20 06:46:08 -07:00
47723f34f2
Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0). ( #473 )
...
* Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout).
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
2016-06-18 17:37:04 +08:00
945c59b1a8
Merge pull request #475 from hebinhuang/FixBuildScripts
...
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:58:13 -07:00
aa0540757e
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:26:04 -07:00
6cecf1c7b2
Cassandra example and other minor updates
2016-06-13 17:43:51 -07:00
6fdc415078
Operator updateStateByKey supports initial state RDD
2016-06-13 16:19:57 +08:00
a7f1360b7a
Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout)
2016-06-10 17:07:45 -07:00
Renyi Xiong
Tao Qin
Hebin Huang
Quanmao LIU
Tao Wang
Kaarthik Sivashanmugam
Daniel Li
jthelin