d3999ca43d
Add option to turn on/off local checkpoint for CSharpStateDStream
2016-07-08 17:51:13 +08:00
e75eb189e0
Download big file to temp one to avoid possible download failure and use corrupted file. ( #490 )
2016-07-08 09:51:16 +08:00
db95eb5de8
Updating maven version
2016-07-05 12:27:00 -07:00
dd376179ac
updating maven version
2016-07-05 11:33:22 -07:00
d78493bc42
updating build version to reflect Spark version
2016-07-05 11:32:44 -07:00
754a530961
updating maven version
2016-07-05 11:31:56 -07:00
b15163657a
When check and release unused JVMObjectTracker objects, change the speed adaptively according to current objects count. ( #488 )
2016-07-04 00:12:10 -07:00
33b87b4b0f
Merge pull request #487 from xiongrenyi/DirectReceiver
...
add validation to tqin's for ReduceByKeyAndWindow
2016-07-02 08:46:56 -07:00
9515b44f26
in addition to tqin's fix of a typo and removing prevFunc in ReduceByKeyAndWindow, add numPartition explicitly and add validation to his sample
2016-07-01 16:49:24 -07:00
4004001732
fixes to run Mobius in Linux
...
* allowing to specify worker path for all modes - this will help specifying prefix script for worker exe in CentOS
* fixing issue 467 - with this change Mobius can run in YARN cluster mode
* updating notes since YARN cluster mode is supported with the fix to issue #467
* added check to run OS-specific path tests
2016-07-01 11:21:33 -07:00
2337e7cee2
upgrading to Spark 1.6.2
2016-07-01 09:59:26 -07:00
67609769ae
Merge pull request #472 from tawan0109/update_state_by_key
...
Operator updateStateByKey supports initial state RDD
2016-06-30 08:28:45 +08:00
1cde355f3b
Add unit test cases for UpdateStateByKey operator
2016-06-29 14:37:19 +08:00
64ee6a8437
Merge branch 'master' of https://github.com/microsoft/Mobius into update_state_by_key
2016-06-29 12:08:54 +08:00
4f39658e4e
updates to documentation for Mobius in Linux
2016-06-28 16:15:17 -07:00
66f0132c14
updating links
2016-06-27 22:35:32 -07:00
a71b525ce6
Merge pull request #466 from qintao1976/ParallelKafkaDataFetcher
...
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-24 11:08:33 +08:00
dd1122ff99
Solve the Unit test problem caused by concurrent access of StringBuilder
2016-06-23 19:18:26 -07:00
62474b104b
Merge pull request #482 from xiongrenyi/DirectReceiver
...
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 15:52:04 -07:00
7a24a79aa5
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 14:51:36 -07:00
7d6ea24a8d
Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side ( #476 )
...
* Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side
1.Java-side: https://github.com/Microsoft/Mobius/blob/master/scala/src/main/org/apache/spark/api/csharp/CSharpBackendHandler.scala#L269
JVMObjectTracker keep a HashMap[String, Object] which is [id, Java-object]
2.CSharp-side :
1) JvmObjectReference remember the id of the object: https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmObjectReference.cs#L20
2) So JvmBridge can call java object's method https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmBridge.cs#L69
Potential memory leak can happen in JVMObjectTracker.
To solve this, track the garbage collection in CSharp side, get the id, release JVMObjectTracker's HashMap.
2016-06-23 21:42:47 +08:00
0359d66548
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-22 22:40:00 -07:00
27916910ad
added code of conduct
2016-06-21 14:39:47 -07:00
e74253eb9d
Merge pull request #479 from tawan0109/dev
...
Disabled LogFileAppender for CSharpWorker
2016-06-21 19:37:42 +08:00
7d6f569c5e
Disabled LogFileAppender for CSharpWorker
2016-06-21 10:13:52 +08:00
6607411acf
Merge pull request #478 from jthelin/fix-run-samples-script-1
...
Fix comment line prefix in run_samples.cmd script.
2016-06-20 11:48:44 -07:00
38f42d2554
Fix comment line prefix in run_samples.cmd script.
2016-06-20 06:46:08 -07:00
47723f34f2
Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0). ( #473 )
...
* Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout).
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
2016-06-18 17:37:04 +08:00
945c59b1a8
Merge pull request #475 from hebinhuang/FixBuildScripts
...
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:58:13 -07:00
aa0540757e
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:26:04 -07:00
6cecf1c7b2
Cassandra example and other minor updates
2016-06-13 17:43:51 -07:00
6fdc415078
Operator updateStateByKey supports initial state RDD
2016-06-13 16:19:57 +08:00
a7f1360b7a
Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout)
2016-06-10 17:07:45 -07:00
18668c3c58
adding instructions for Linux
2016-06-06 22:19:20 -07:00
513915fbfc
Merge pull request #469 from hebinhuang/DefaultSocketWrapper
...
Add a wrapper for Socket interface
2016-06-01 16:15:13 -07:00
3a16d09187
Add a wrapper for Socket interface
2016-06-01 13:45:28 -07:00
5512ccd48e
Merge pull request #462 from xiongrenyi/dev
...
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-29 23:20:33 -07:00
273551e254
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-28 08:51:34 -07:00
cc81df5cda
1. construct csharp worker path in CSharpRDD.compute; 2. set unix mode in zip files ( #461 )
2016-05-27 12:00:23 -07:00
cbdef13b94
Merge pull request #458 from xiongrenyi/dev
...
fix Duration type mismatch bug in Window API, changes streaming interval to seconds from mini seconds to be consistent with other APIs like python does.
2016-05-25 16:18:05 -07:00
9c7cd08c95
Merge branch 'dev' of https://github.com/xiongrenyi/SparkCLR into dev
2016-05-25 15:14:29 -07:00
abe1dfbbab
fix Duration type mismatch bug in
...
enable DStreamTextFileSample and add Window API
change duration type to seconds for streaming bacth interval, Window, Checkpoint and Remember APIs
fix Duration type mismatch bug in
enable DStreamTextFileSample and add Window API
Update JvmBridgeUtils.cs
use GetJavaDuration utility method
use GetJavaDuration utility method
include Window API in unit test
revert DStreamTextFileSample to experimental
Window API included in unit test
2016-05-25 15:10:10 -07:00
434bcbfd04
Merge branch 'master' of https://github.com/Microsoft/SparkCLR
2016-05-25 15:05:21 -07:00
78c3033f52
revert DStreamTextFileSample to experimental
...
Window API included in unit test
2016-05-25 10:30:49 -07:00
9909b7fc0d
include Window API in unit test
2016-05-25 10:28:48 -07:00
1711a2dfe1
use GetJavaDuration utility method
2016-05-25 10:27:46 -07:00
3776666a1a
use GetJavaDuration utility method
2016-05-25 10:25:51 -07:00
f7bf3cee5c
Update JvmBridgeUtils.cs
2016-05-25 10:23:06 -07:00
71e3c37616
Merge branch 'dev' of https://github.com/xiongrenyi/SparkCLR into dev
2016-05-20 13:06:04 -07:00
eab9a53366
change duration type to seconds for streaming bacth interval, Window, Checkpoint and Remember APIs
2016-05-20 13:04:14 -07:00
tawan0109
Quanmao LIU
Kaarthik Sivashanmugam
Tao Qin
Renyi Xiong
Daniel Li
jthelin
Hebin Huang
Kai Zeng