94420c1879
Merge pull request #502 from hebinhuang/RioSockBug
...
Fix bugs in RioSocketWrapper and SocketStream that cause vstest failed
2016-07-15 10:30:08 -07:00
41a91b03b0
Fix bugs in RioSocketWrapper and SocketStream that cause vstest failed
2016-07-14 21:08:52 -07:00
622242c5b5
Merge branch 'master' of https://github.com/microsoft/SparkCLR into REPL
2016-07-13 18:05:36 +08:00
f31c92f7f0
Downgrade the log level of WeakObjectManager.cs to reduce logs. ( #499 )
2016-07-13 14:23:20 +08:00
46c08e45a5
Merge pull request #492 from tawan0109/local_checkpoint
...
Rebuild state for DStream UpdateStateByKey operation by replaying batches
2016-07-12 11:30:25 +08:00
ffe223c2f9
Merge pull request #483 from hebinhuang/ManagedRioSock
...
Add RIOSocketWrapper and SaeaSocketWrapper
2016-07-10 07:32:20 -07:00
b6ec24119e
Add RIOSocketWrapper and SaeaSocketWrapper
2016-07-09 22:00:10 -07:00
470bb411fe
Merge branch 'master' of https://github.com/microsoft/Mobius into REPL
2016-07-09 18:08:17 +08:00
3b67b2a6a7
Avoid RDD checkpoint for DynamicPartitionKafkaInputDStream by replaying last N batches
2016-07-08 17:52:47 +08:00
d3999ca43d
Add option to turn on/off local checkpoint for CSharpStateDStream
2016-07-08 17:51:13 +08:00
e75eb189e0
Download big file to temp one to avoid possible download failure and use corrupted file. ( #490 )
2016-07-08 09:51:16 +08:00
db95eb5de8
Updating maven version
2016-07-05 12:27:00 -07:00
dd376179ac
updating maven version
2016-07-05 11:33:22 -07:00
d78493bc42
updating build version to reflect Spark version
2016-07-05 11:32:44 -07:00
754a530961
updating maven version
2016-07-05 11:31:56 -07:00
b15163657a
When check and release unused JVMObjectTracker objects, change the speed adaptively according to current objects count. ( #488 )
2016-07-04 00:12:10 -07:00
33b87b4b0f
Merge pull request #487 from xiongrenyi/DirectReceiver
...
add validation to tqin's for ReduceByKeyAndWindow
2016-07-02 08:46:56 -07:00
9515b44f26
in addition to tqin's fix of a typo and removing prevFunc in ReduceByKeyAndWindow, add numPartition explicitly and add validation to his sample
2016-07-01 16:49:24 -07:00
4004001732
fixes to run Mobius in Linux
...
* allowing to specify worker path for all modes - this will help specifying prefix script for worker exe in CentOS
* fixing issue 467 - with this change Mobius can run in YARN cluster mode
* updating notes since YARN cluster mode is supported with the fix to issue #467
* added check to run OS-specific path tests
2016-07-01 11:21:33 -07:00
2337e7cee2
upgrading to Spark 1.6.2
2016-07-01 09:59:26 -07:00
67609769ae
Merge pull request #472 from tawan0109/update_state_by_key
...
Operator updateStateByKey supports initial state RDD
2016-06-30 08:28:45 +08:00
1cde355f3b
Add unit test cases for UpdateStateByKey operator
2016-06-29 14:37:19 +08:00
64ee6a8437
Merge branch 'master' of https://github.com/microsoft/Mobius into update_state_by_key
2016-06-29 12:08:54 +08:00
4f39658e4e
updates to documentation for Mobius in Linux
2016-06-28 16:15:17 -07:00
66f0132c14
updating links
2016-06-27 22:35:32 -07:00
a71b525ce6
Merge pull request #466 from qintao1976/ParallelKafkaDataFetcher
...
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-24 11:08:33 +08:00
dd1122ff99
Solve the Unit test problem caused by concurrent access of StringBuilder
2016-06-23 19:18:26 -07:00
62474b104b
Merge pull request #482 from xiongrenyi/DirectReceiver
...
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 15:52:04 -07:00
7a24a79aa5
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 14:51:36 -07:00
7d6ea24a8d
Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side ( #476 )
...
* Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side
1.Java-side: https://github.com/Microsoft/Mobius/blob/master/scala/src/main/org/apache/spark/api/csharp/CSharpBackendHandler.scala#L269
JVMObjectTracker keep a HashMap[String, Object] which is [id, Java-object]
2.CSharp-side :
1) JvmObjectReference remember the id of the object: https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmObjectReference.cs#L20
2) So JvmBridge can call java object's method https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmBridge.cs#L69
Potential memory leak can happen in JVMObjectTracker.
To solve this, track the garbage collection in CSharp side, get the id, release JVMObjectTracker's HashMap.
2016-06-23 21:42:47 +08:00
0359d66548
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-22 22:40:00 -07:00
27916910ad
added code of conduct
2016-06-21 14:39:47 -07:00
e74253eb9d
Merge pull request #479 from tawan0109/dev
...
Disabled LogFileAppender for CSharpWorker
2016-06-21 19:37:42 +08:00
7d6f569c5e
Disabled LogFileAppender for CSharpWorker
2016-06-21 10:13:52 +08:00
6607411acf
Merge pull request #478 from jthelin/fix-run-samples-script-1
...
Fix comment line prefix in run_samples.cmd script.
2016-06-20 11:48:44 -07:00
38f42d2554
Fix comment line prefix in run_samples.cmd script.
2016-06-20 06:46:08 -07:00
47723f34f2
Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0). ( #473 )
...
* Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout).
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
2016-06-18 17:37:04 +08:00
945c59b1a8
Merge pull request #475 from hebinhuang/FixBuildScripts
...
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:58:13 -07:00
aa0540757e
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:26:04 -07:00
6cecf1c7b2
Cassandra example and other minor updates
2016-06-13 17:43:51 -07:00
6fdc415078
Operator updateStateByKey supports initial state RDD
2016-06-13 16:19:57 +08:00
a7f1360b7a
Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout)
2016-06-10 17:07:45 -07:00
18668c3c58
adding instructions for Linux
2016-06-06 22:19:20 -07:00
513915fbfc
Merge pull request #469 from hebinhuang/DefaultSocketWrapper
...
Add a wrapper for Socket interface
2016-06-01 16:15:13 -07:00
3a16d09187
Add a wrapper for Socket interface
2016-06-01 13:45:28 -07:00
5512ccd48e
Merge pull request #462 from xiongrenyi/dev
...
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-29 23:20:33 -07:00
273551e254
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-28 08:51:34 -07:00
cc81df5cda
1. construct csharp worker path in CSharpRDD.compute; 2. set unix mode in zip files ( #461 )
2016-05-27 12:00:23 -07:00
cbdef13b94
Merge pull request #458 from xiongrenyi/dev
...
fix Duration type mismatch bug in Window API, changes streaming interval to seconds from mini seconds to be consistent with other APIs like python does.
2016-05-25 16:18:05 -07:00
9c7cd08c95
Merge branch 'dev' of https://github.com/xiongrenyi/SparkCLR into dev
2016-05-25 15:14:29 -07:00
Hebin Huang
tawan0109
Quanmao LIU
Kaarthik Sivashanmugam
Tao Qin
Renyi Xiong
Daniel Li
jthelin
Kai Zeng