2337e7cee2
upgrading to Spark 1.6.2
2016-07-01 09:59:26 -07:00
67609769ae
Merge pull request #472 from tawan0109/update_state_by_key
...
Operator updateStateByKey supports initial state RDD
2016-06-30 08:28:45 +08:00
1cde355f3b
Add unit test cases for UpdateStateByKey operator
2016-06-29 14:37:19 +08:00
64ee6a8437
Merge branch 'master' of https://github.com/microsoft/Mobius into update_state_by_key
2016-06-29 12:08:54 +08:00
4f39658e4e
updates to documentation for Mobius in Linux
2016-06-28 16:15:17 -07:00
66f0132c14
updating links
2016-06-27 22:35:32 -07:00
a71b525ce6
Merge pull request #466 from qintao1976/ParallelKafkaDataFetcher
...
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-24 11:08:33 +08:00
dd1122ff99
Solve the Unit test problem caused by concurrent access of StringBuilder
2016-06-23 19:18:26 -07:00
62474b104b
Merge pull request #482 from xiongrenyi/DirectReceiver
...
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 15:52:04 -07:00
7a24a79aa5
add kafka cluster id and refactor UpdateStateByKey API
2016-06-23 14:51:36 -07:00
7d6ea24a8d
Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side ( #476 )
...
* Release JVM object reference in Java(Scala) side JVMObjectTracker from C# side
1.Java-side: https://github.com/Microsoft/Mobius/blob/master/scala/src/main/org/apache/spark/api/csharp/CSharpBackendHandler.scala#L269
JVMObjectTracker keep a HashMap[String, Object] which is [id, Java-object]
2.CSharp-side :
1) JvmObjectReference remember the id of the object: https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmObjectReference.cs#L20
2) So JvmBridge can call java object's method https://github.com/Microsoft/Mobius/blob/master/csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc/JvmBridge.cs#L69
Potential memory leak can happen in JVMObjectTracker.
To solve this, track the garbage collection in CSharp side, get the id, release JVMObjectTracker's HashMap.
2016-06-23 21:42:47 +08:00
0359d66548
support parallelly pre-fetch data for DynamicPartitionKafkaInputDStream
2016-06-22 22:40:00 -07:00
27916910ad
added code of conduct
2016-06-21 14:39:47 -07:00
e74253eb9d
Merge pull request #479 from tawan0109/dev
...
Disabled LogFileAppender for CSharpWorker
2016-06-21 19:37:42 +08:00
7d6f569c5e
Disabled LogFileAppender for CSharpWorker
2016-06-21 10:13:52 +08:00
6607411acf
Merge pull request #478 from jthelin/fix-run-samples-script-1
...
Fix comment line prefix in run_samples.cmd script.
2016-06-20 11:48:44 -07:00
38f42d2554
Fix comment line prefix in run_samples.cmd script.
2016-06-20 06:46:08 -07:00
47723f34f2
Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0). ( #473 )
...
* Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout).
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
* Fix needless quotes to arguments (well on Spark 1.6 but causes "input line too long" error on Spark 2.0).
Error repro : (quotes "Pi*" cause error)
set HADOOP_HOME=d:\msgit\skaarthikMobius\build\tools\winutils
set SPARKCLR_HOME=d:\msgit\skaarthikMobius\build\runtime
set SPARK_HOME=d:\msgit\skaarthikMobius\build\tools\spark-2.0.0-preview-bin-hadoop2.6
cd /d d:\msgit\skaarthikMobius\build\runtime\samples
%SPARKCLR_HOME%\scripts\sparkclr-submit.cmd --exe SparkCLRSamples.exe %CD% --torun Pi* --data d:\tmp\skdata
2016-06-18 17:37:04 +08:00
945c59b1a8
Merge pull request #475 from hebinhuang/FixBuildScripts
...
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:58:13 -07:00
aa0540757e
Fix build scripts to comsume msbuild v12.0
2016-06-14 17:26:04 -07:00
6cecf1c7b2
Cassandra example and other minor updates
2016-06-13 17:43:51 -07:00
6fdc415078
Operator updateStateByKey supports initial state RDD
2016-06-13 16:19:57 +08:00
a7f1360b7a
Fix AwaitTermination(int timeout) error, and change to AwaitTerminationOrTimeout(long timeout)
2016-06-10 17:07:45 -07:00
18668c3c58
adding instructions for Linux
2016-06-06 22:19:20 -07:00
513915fbfc
Merge pull request #469 from hebinhuang/DefaultSocketWrapper
...
Add a wrapper for Socket interface
2016-06-01 16:15:13 -07:00
3a16d09187
Add a wrapper for Socket interface
2016-06-01 13:45:28 -07:00
5512ccd48e
Merge pull request #462 from xiongrenyi/dev
...
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-29 23:20:33 -07:00
273551e254
fix DynamicPartitionKafkaInputDStream checkpint and add C# func support
2016-05-28 08:51:34 -07:00
cc81df5cda
1. construct csharp worker path in CSharpRDD.compute; 2. set unix mode in zip files ( #461 )
2016-05-27 12:00:23 -07:00
cbdef13b94
Merge pull request #458 from xiongrenyi/dev
...
fix Duration type mismatch bug in Window API, changes streaming interval to seconds from mini seconds to be consistent with other APIs like python does.
2016-05-25 16:18:05 -07:00
9c7cd08c95
Merge branch 'dev' of https://github.com/xiongrenyi/SparkCLR into dev
2016-05-25 15:14:29 -07:00
abe1dfbbab
fix Duration type mismatch bug in
...
enable DStreamTextFileSample and add Window API
change duration type to seconds for streaming bacth interval, Window, Checkpoint and Remember APIs
fix Duration type mismatch bug in
enable DStreamTextFileSample and add Window API
Update JvmBridgeUtils.cs
use GetJavaDuration utility method
use GetJavaDuration utility method
include Window API in unit test
revert DStreamTextFileSample to experimental
Window API included in unit test
2016-05-25 15:10:10 -07:00
434bcbfd04
Merge branch 'master' of https://github.com/Microsoft/SparkCLR
2016-05-25 15:05:21 -07:00
78c3033f52
revert DStreamTextFileSample to experimental
...
Window API included in unit test
2016-05-25 10:30:49 -07:00
9909b7fc0d
include Window API in unit test
2016-05-25 10:28:48 -07:00
1711a2dfe1
use GetJavaDuration utility method
2016-05-25 10:27:46 -07:00
3776666a1a
use GetJavaDuration utility method
2016-05-25 10:25:51 -07:00
f7bf3cee5c
Update JvmBridgeUtils.cs
2016-05-25 10:23:06 -07:00
71e3c37616
Merge branch 'dev' of https://github.com/xiongrenyi/SparkCLR into dev
2016-05-20 13:06:04 -07:00
eab9a53366
change duration type to seconds for streaming bacth interval, Window, Checkpoint and Remember APIs
2016-05-20 13:04:14 -07:00
d8f96a9fe5
enable DStreamTextFileSample and add Window API
2016-05-20 13:04:09 -07:00
69f5673741
fix Duration type mismatch bug in
2016-05-20 13:04:07 -07:00
0ee69f65a1
enable DStreamTextFileSample and add Window API
2016-05-20 11:17:46 -07:00
bd1eb4354d
fix Duration type mismatch bug in
2016-05-20 11:12:52 -07:00
e35a903155
Merge pull request #453 from kai-zeng/master
...
Fix for issue #285
2016-05-20 10:25:29 -07:00
4def779f7f
Merge pull request #449 from xiongrenyi/dev
...
add parallel job to UpdateStateByKey
2016-05-20 08:38:35 -07:00
7e93c05352
hacky check to work with previous releases
2016-05-20 00:11:09 -07:00
9c491397c6
trigger pull request since last one failed randomly
2016-05-19 12:25:49 -07:00
5f4e589154
add conf.set("spark.testing", "true")
2016-05-19 12:08:39 -07:00
ef65a37a8b
update jobExecutor from protected to private[streaming] so that accessable from unit test
2016-05-19 11:19:40 -07:00
Kaarthik Sivashanmugam
Tao Wang
Tao Qin
Renyi Xiong
Quanmao LIU
Daniel Li
jthelin
Hebin Huang
Kai Zeng
kai