* Check that Download-File function got a real file.
- Check in the `Download-File` function that the current downloaded copy of the file is not empty [zero length].
* archive.apache.org
- Use `archive.apache.org` as the default server location for downloading Spark [and Maven] binary distributions.
- Previous location was `www.us.apache.org` but that only supports the most recent distributions for each major release
[ie currently `2.0.1` but not `2.0.0` which Mobius is using at the moment.]
* The Apache distribution server is used during both build and run phases.
Besides normal spark-submit , there's a convenient way using https://github.com/qualiu/testMobius
1.Cluster mode examples as following:
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 10 hdfs:///perf/data/deletions/*
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 10
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 10
2.Local mode examples :
(1) First set SparkOptions : just copy the "Local Mode set SparkOptions=" after running the bat without arguments.
(2) Then submit local test : (you should have downloaded or copied the test data, like d:\mobius\deletions)
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
* Add read and write buffer for CSharpWorker
* Fix test coverage decrease issue
* Adjust read buffer
* Fix failed unit test
* use --conf option to pass buffer size
* Fix flaky unit test
* Address review comments
* Fix option name renaming issue
dwnichols
Attila Szucs
Kaarthik Sivashanmugam
Jorgen Thelin
Kai Zeng
skaarthik
Shakib Bawa
shakdoesgithub
Quanmao LIU
Yun Tang
Hebin Huang
Tao Wang