9aa97b98c6
Dev/sutyag/upgrade mobius ( #697 )
...
* basic
* Add extractor and outputter
* Add reducer not done
* Add procedure
* kill node, broadcast, upload executable error feed to cosmos, specify avro or parquet syntax
* Add more functions to HDFS.
Add submitter heartbeat
Update doc
* Redesign cosmos download, add replication setting for hdfs
* Improve executable runner to deal with bad lines
* MERGE MOBIUS
* change dependency path
* Add registration method to mobius
* Major refactoring to add ISparkosmosModule to modulize everything
Start supporting streaming
Fixed a couple of Mobius bugs
Added integration tests
Reenabled unit tests
Added DatedPath
* Make sparkcontext settable, fix setjobgroup
* Expose more interface from Mobius
* Mobius change for Spark 2.3
* fix version conflict, remove unused files
* Added support for multiple UDFs
* Fixed non sql udf issue
* 1. Upgarde mobius to spark 2.3.1 2. Fixed UDF bugs 3. Added support for multipe UDFs
* 1. Added sample testcases 2.Updated referece for examples
* Removed stashed files
* Fixed review comments
* Fixed review comments
* Fixed failed unit test cases
* Deleting all the things
* Updated version in appveyor
* Updated tartool download path
* Fixed java process terminate issue
* Revert access modifier to internal from public for JvmBridge
2018-11-20 17:23:43 -08:00
1a27f42dc1
updating TarTool URL
2017-04-08 21:07:10 -07:00
a402cb4471
Update TarTool direct link. ( #639 )
...
* Update TarTool direct link.
A more permanent solution here needs to be found so that
we do not need to update this link. In the meantime, this
will unblock users from successfully building
* Trigger new build
2017-03-19 12:23:59 -04:00
92c281ffc2
Update default spark version in Windows build to 2.0.2
2017-02-09 14:51:25 -05:00
8aad6b483f
upgrading to Spark 2.0.2
2017-01-28 01:48:49 -08:00
6e2b820524
Fix Spark version build problems
...
* Check that Download-File function got a real file.
- Check in the `Download-File` function that the current downloaded copy of the file is not empty [zero length].
* archive.apache.org
- Use `archive.apache.org` as the default server location for downloading Spark [and Maven] binary distributions.
- Previous location was `www.us.apache.org` but that only supports the most recent distributions for each major release
[ie currently `2.0.1` but not `2.0.0` which Mobius is using at the moment.]
* The Apache distribution server is used during both build and run phases.
2016-11-01 22:41:17 -07:00
3c767bddd4
Add Python version performance benchmark test; Add usage + example and RIOSocket option support for C# version benchmark; Update Scala version benchmark; Update csv package version. ( #565 )
...
Besides normal spark-submit , there's a convenient way using https://github.com/qualiu/testMobius
1.Cluster mode examples as following:
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 10 hdfs:///perf/data/deletions/*
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 10
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 10
2.Local mode examples :
(1) First set SparkOptions : just copy the "Local Mode set SparkOptions=" after running the bat without arguments.
(2) Then submit local test : (you should have downloaded or copied the test data, like d:\mobius\deletions)
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
2016-09-20 13:41:29 +08:00
3da5beb91e
Store external dependencies in build before copying to runtime folder
2016-09-16 18:28:42 -04:00
427ce52e8d
Download external dependencies from the build.sh shell script
2016-09-16 16:32:06 -04:00
67046cef3e
Update run-samples.sh to use unix path separators
2016-09-16 14:13:48 -04:00
9f4276e338
[Mobius-102] Add missing license text in batch and powershell scripts ( #557 )
2016-08-31 11:33:41 -07:00
09d8a2f215
updating examples to use Mobius 2.0 preview release and minor updates
2016-08-25 09:04:19 -07:00
cec7737060
upgrade to Spark 2.0
2016-08-24 12:55:19 -07:00
42a0c3561d
Escaped^Cpecial characters in build.cmd
2016-08-12 12:31:59 +08:00
04f642eed8
documentation updates
2016-08-11 18:18:38 -07:00
2c4ea6e07e
Code clean up & add document
2016-07-27 16:31:22 +08:00
45ef6a5291
Merge branch 'master' of https://github.com/microsoft/SparkCLR into REPL
2016-07-25 20:03:58 +08:00
1eb7d98158
Rename sparkclr-repl.cmd to sparkclr-shell.cmd
2016-07-25 15:00:02 +08:00
38a9eb2069
Implement automatic code coverage reports for c++ code
2016-07-19 12:56:55 -07:00
622242c5b5
Merge branch 'master' of https://github.com/microsoft/SparkCLR into REPL
2016-07-13 18:05:36 +08:00
46c08e45a5
Merge pull request #492 from tawan0109/local_checkpoint
...
Rebuild state for DStream UpdateStateByKey operation by replaying batches
2016-07-12 11:30:25 +08:00
b6ec24119e
Add RIOSocketWrapper and SaeaSocketWrapper
2016-07-09 22:00:10 -07:00
470bb411fe
Merge branch 'master' of https://github.com/microsoft/Mobius into REPL
2016-07-09 18:08:17 +08:00
3b67b2a6a7
Avoid RDD checkpoint for DynamicPartitionKafkaInputDStream by replaying last N batches
2016-07-08 17:52:47 +08:00
e75eb189e0
Download big file to temp one to avoid possible download failure and use corrupted file. ( #490 )
2016-07-08 09:51:16 +08:00
db95eb5de8
Updating maven version
2016-07-05 12:27:00 -07:00
754a530961
updating maven version
2016-07-05 11:31:56 -07:00
2337e7cee2
upgrading to Spark 1.6.2
2016-07-01 09:59:26 -07:00
38f42d2554
Fix comment line prefix in run_samples.cmd script.
2016-06-20 06:46:08 -07:00
b97ed0a37d
Fixing issue that build did not stop when mvn build failed
2016-05-19 12:16:06 +08:00
0c91852baa
Update download path of winutils.exe to fix running localmode example problem (such as: permission of /tmp/hive)
2016-05-10 10:58:01 +08:00
07b704f8aa
Sample word count project in F#
2016-05-07 11:07:19 -07:00
f8dea1f0e7
limiting executionpolicy change to one script instead of current user
2016-05-02 16:06:26 -07:00
1268b20906
limiting executionpolicy change to one script instead of current user
2016-05-01 22:43:15 -07:00
2fbd2b0cfe
merge from branch-1.6 and version bump
2016-05-01 14:47:30 -07:00
7982c8e4a6
updated doc and packaging step
2016-04-26 21:37:36 -07:00
d7f528bc22
fix bug in copyjar that happens in tagged releases due to multiple files that match pattern ( #424 )
2016-04-26 00:38:22 -07:00
cc27f3a31a
Copies the dependencies of Examples from runtime/bin to use latest Mobius bits
...
* Copies the dependencies of Examples from runtime/bin to use latest Mobius bits
* Change the copy order - copy Mobius bits first
2016-04-25 17:27:11 -07:00
80832ba9f0
doc updates and using Mobius name in scripts
2016-04-25 13:06:18 -07:00
cd4cfec562
updating scripts to download & use spark-csv and its dependency
2016-04-24 01:56:58 -07:00
1d1a31873c
adding powershell script to copy non-uber jar to runtime folder
2016-04-22 15:16:15 -07:00
e2eaa6faa3
Fix build.sh to check whether Examples build failed
2016-04-22 07:42:01 -07:00
9342171946
Fix Build.cmd to stop building when Examples compiling failed
2016-04-21 10:08:35 -07:00
7cb61f0b5d
Fix Build.cmd to include examples binaries in release zip file
2016-04-20 20:34:30 -07:00
0a855ce640
Implement a Daemon process to take charge of launching Worker processes
2016-03-24 16:24:02 +08:00
3a4c938619
upgrade to Spark 1.6.1
2016-03-18 21:55:02 -07:00
c0ba9c59b6
Fix license
2016-03-16 11:53:26 -07:00
601ff40a41
including examples in build
2016-02-04 16:44:50 -08:00
04ac532e81
Rename class name and Assembly name to 'Repl'
2016-02-04 21:22:24 +08:00
1f5b6dddc2
Rename project from shell to Repl
2016-02-04 16:51:18 +08:00
sutyag
Kaarthik Sivashanmugam
David Nichols
skaarthik
Jorgen Thelin
Quanmao LIU
Yun Tang
tawan0109
Hebin Huang
Isaac Abraham
guiling