Besides normal spark-submit , there's a convenient way using https://github.com/qualiu/testMobius
1.Cluster mode examples as following:
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 10 hdfs:///perf/data/deletions/*
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 10
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 10
2.Local mode examples :
(1) First set SparkOptions : just copy the "Local Mode set SparkOptions=" after running the bat without arguments.
(2) Then submit local test : (you should have downloaded or copied the test data, like d:\mobius\deletions)
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-python-perf-test.bat d:\msgit\lqmMobius\python\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-scala-perf-test.bat d:\msgit\lqmMobius\scala\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
D:\msgit\lqmMobius\testMobius\scripts\perf\submit-mobius-perf-test.bat d:\msgit\lqmMobius\csharp\perf 3 d:\mobius\deletions\deletions.csv-00000-of-00020
3c767bddd4