doc updates and using Mobius name in scripts
This commit is contained in:
Kaarthik Sivashanmugam
Родитель
a01ea174c9
Коммит
80832ba9f0
29
README.md
29
README.md
|
|
@ -89,11 +89,11 @@ Refer to [Mobius C# API documentation](csharp/Adapter/documentation/Mobius_API_D
|
|||
|
||||
Mobius API usage samples are available at:
|
||||
|
||||
* [Examples folder](./examples) which contains standalone [C# projects](/notes/running-mobius-app.md#running-mobius-examples-in-local-mode) that can be used as templates to start developing Mobius applications
|
||||
|
||||
* [Samples project](csharp/Samples/Microsoft.Spark.CSharp/) which uses a comprehensive set of Mobius APIs to implement samples that are also used for functional validation of APIs
|
||||
|
||||
* [Examples folder](./examples) which contains standalone C# projects that can be used as templates to start developing Mobius applications
|
||||
|
||||
* Performance test scenarios implemented in [C#](csharp/Perf/Microsoft.Spark.CSharp) and [Scala](scala/perf) for side by side comparison of Spark driver code
|
||||
|
||||
* Mobius performance test scenarios implemented in [C#](csharp/Perf/Microsoft.Spark.CSharp) and [Scala](scala/perf) for side by side comparison of Spark driver code
|
||||
|
||||
## Documents
|
||||
|
||||
|
|
@ -108,16 +108,20 @@ Refer to the [docs folder](docs) for design overview and other info on Mobius
|
|||
## Getting Started
|
||||
|
||||
| |Windows |Linux |
|
||||
|---|:------:|:----:|
|
||||
|Build & run unit tests |[windows-instructions.md](notes/windows-instructions.md#building-mobius) |[linux-instructions.md](notes/linux-instructions.md#building-mobius) |
|
||||
|Run samples (functional tests) in local mode |[windows-instructions.md](notes/windows-instructions.md#running-samples) |[linux-instructions.md](notes/linux-instructions.md#running-samples) |
|
||||
|Run examples in local mode |[running-mobius-app.md](notes/running-mobius-app.md#running-examples-in-local-mode) |[running-mobius-app.md](notes/running-mobius-app.md#linux-instructions) |
|
||||
|Run Mobius app in standalone cluster |[running-mobius-app.md](notes/running-mobius-app.md#standalone-cluster) |[running-mobius-app.md](notes/running-mobius-app.md#linux-instructions) |
|
||||
|Run Mobius app in YARN cluster |[running-mobius-app.md](notes/running-mobius-app.md#yarn-cluster) |[running-mobius-app.md](notes/running-mobius-app.md#linux-instructions) |
|
||||
|---|:------|:----|
|
||||
|Build & run unit tests |[Build in Windows](notes/windows-instructions.md#building-mobius) |[Build in Linux](notes/linux-instructions.md#building-mobius) |
|
||||
|Run samples (functional tests) in local mode |[Samples in Windows](notes/windows-instructions.md#running-samples) |[Samples in Linux](notes/linux-instructions.md#running-samples) |
|
||||
|Run examples in local mode |[Examples in Windows](/notes/running-mobius-app.md#running-mobius-examples-in-local-mode) |[Examples in Linux](notes/running-mobius-app.md#linux-instructions) |
|
||||
|Run Mobius app |<ul><li>[Standalone cluster](notes/running-mobius-app.md#standalone-cluster)</li><li>[YARN cluster](notes/running-mobius-app.md#yarn-cluster)</li></ul> |[Mobius app in Linux](notes/running-mobius-app.md#linux-instructions) |
|
||||
|
||||
### Useful Links
|
||||
* [Configuration parameters in Mobius](/notes/configuration-mobius.md)
|
||||
* [Troubleshoot errors in Mobius](/notes/troubleshooting-mobius.md)
|
||||
* [Debug Mobius apps](/notes/running-mobius-app.md#debug-mode)
|
||||
|
||||
## Supported Spark Versions
|
||||
|
||||
Mobius is built and tested with [Spark 1.4.1](https://github.com/Microsoft/Mobius/tree/branch-1.4), [Spark 1.5.2](https://github.com/Microsoft/Mobius/tree/branch-1.5) and [Spark 1.6.*](https://github.com/Microsoft/Mobius/tree/branch-1.6).
|
||||
Mobius is built and tested with Apache Spark [1.4.1](https://github.com/Microsoft/Mobius/tree/branch-1.4), [1.5.2](https://github.com/Microsoft/Mobius/tree/branch-1.5) and [1.6.*](https://github.com/Microsoft/Mobius/tree/branch-1.6).
|
||||
|
||||
## Releases
|
||||
|
||||
|
|
@ -125,7 +129,7 @@ Mobius releases are available at https://github.com/Microsoft/Mobius/releases. R
|
|||
|
||||
[](https://www.nuget.org/packages/Microsoft.SparkCLR)
|
||||
|
||||
Refer to [mobius-release-info.md](notes/mobius-release-info.md) for the details on verioning policy and the contents of the release.
|
||||
Refer to [mobius-release-info.md](notes/mobius-release-info.md) for the details on versioning policy and the contents of the release.
|
||||
|
||||
## License
|
||||
|
||||
|
|
@ -148,4 +152,3 @@ Mobius is licensed under the MIT license. See [LICENSE](LICENSE) file for full l
|
|||
* create post with "sparkclr" tag in [Stack Overflow](https://stackoverflow.com/questions/tagged/sparkclr)
|
||||
* join chat at [Mobius room in Gitter](https://gitter.im/Microsoft/Mobius)
|
||||
* tweet [@MobiusForSpark](http://twitter.com/MobiusForSpark)
|
||||
* send email to sparkclr-user@googlegroups.com
|
||||
|
|
|
|||
|
|
@ -37,14 +37,14 @@ if NOT EXIST "%SPARKCLR_HOME%\data" mkdir "%SPARKCLR_HOME%\data"
|
|||
if NOT EXIST "%SPARKCLR_HOME%\lib" mkdir "%SPARKCLR_HOME%\lib"
|
||||
if NOT EXIST "%SPARKCLR_HOME%\samples" mkdir "%SPARKCLR_HOME%\samples"
|
||||
|
||||
@echo Assemble SparkCLR Scala components
|
||||
@echo Assemble Mobius Scala components
|
||||
pushd "%CMDHOME%\..\scala"
|
||||
|
||||
@rem clean the target directory first
|
||||
call mvn.cmd %MVN_QUIET% clean
|
||||
|
||||
@rem
|
||||
@rem Note: Shade-plugin helps creates an uber-package to simplify SparkCLR job submission;
|
||||
@rem Note: Shade-plugin helps creates an uber-package to simplify running samples during CI;
|
||||
@rem however, it breaks debug mode in IntellJ. So enable shade-plugin
|
||||
@rem only in build.cmd to create the uber-package.
|
||||
@rem
|
||||
|
|
@ -85,12 +85,12 @@ IF "%APPVEYOR_REPO_TAG%" == "true" (goto :sign)
|
|||
copy /y %temp%\pom.xml.original pom.xml
|
||||
|
||||
if %ERRORLEVEL% NEQ 0 (
|
||||
@echo Build SparkCLR Scala components failed, stop building.
|
||||
@echo Build Mobius Scala components failed, stop building.
|
||||
popd
|
||||
goto :eof
|
||||
)
|
||||
|
||||
@echo SparkCLR Scala binaries
|
||||
@echo Mobius Scala binaries
|
||||
@rem copy non-uber jar to runtime\lib folder
|
||||
powershell -f ..\build\copyjar.ps1
|
||||
popd
|
||||
|
|
@ -105,7 +105,7 @@ if EXIST "%CMDHOME%\lib" (
|
|||
)
|
||||
|
||||
:buildCSharp
|
||||
@echo Assemble SparkCLR C# components
|
||||
@echo Assemble Mobius C# components
|
||||
pushd "%CMDHOME%\..\csharp"
|
||||
|
||||
@rem clean any possible previous build first
|
||||
|
|
@ -113,20 +113,20 @@ call Clean.cmd
|
|||
call Build.cmd
|
||||
|
||||
if %ERRORLEVEL% NEQ 0 (
|
||||
@echo Build SparkCLR C# components failed, stop building.
|
||||
@echo Build Mobius C# components failed, stop building.
|
||||
popd
|
||||
goto :eof
|
||||
)
|
||||
|
||||
@echo SparkCLR C# binaries
|
||||
@echo Mobius C# binaries
|
||||
copy /y Worker\Microsoft.Spark.CSharp\bin\Release\* "%SPARKCLR_HOME%\bin\"
|
||||
|
||||
@echo SparkCLR C# Samples binaries
|
||||
@echo Mobius C# Samples binaries
|
||||
@rem need to include CSharpWorker.exe.config in samples folder
|
||||
copy /y Worker\Microsoft.Spark.CSharp\bin\Release\* "%SPARKCLR_HOME%\samples\"
|
||||
copy /y Samples\Microsoft.Spark.CSharp\bin\Release\* "%SPARKCLR_HOME%\samples\"
|
||||
|
||||
@echo SparkCLR Samples data
|
||||
@echo Mobius Samples data
|
||||
copy /y Samples\Microsoft.Spark.CSharp\data\* "%SPARKCLR_HOME%\data\"
|
||||
popd
|
||||
|
||||
|
|
@ -139,13 +139,13 @@ powershell -f localmode\downloadtools.ps1 dependencies
|
|||
@echo Assemble dependencies
|
||||
xcopy /e /y "%DEPENDENCIES_HOME%" "%SPARKCLR_HOME%\dependencies\"
|
||||
|
||||
@echo Assemble SparkCLR examples
|
||||
@echo Assemble Mobius examples
|
||||
pushd "%CMDHOME%\..\examples"
|
||||
call Clean.cmd
|
||||
call Build.cmd
|
||||
|
||||
if %ERRORLEVEL% NEQ 0 (
|
||||
@echo Build SparkCLR C# examples failed, stop building.
|
||||
@echo Build Mobius C# examples failed, stop building.
|
||||
popd
|
||||
goto :eof
|
||||
)
|
||||
|
|
@ -177,7 +177,7 @@ goto :copyscripts
|
|||
:copyscripts
|
||||
popd
|
||||
|
||||
@echo Assemble SparkCLR script components
|
||||
@echo Assemble Mobius script components
|
||||
xcopy /e /y "%CMDHOME%\..\scripts" "%SPARKCLR_HOME%\scripts\"
|
||||
|
||||
@echo Make distribution
|
||||
|
|
|
|||
|
|
@ -18,14 +18,14 @@ fi
|
|||
[ ! -d "$SPARKCLR_HOME/samples" ] && mkdir "$SPARKCLR_HOME/samples"
|
||||
[ ! -d "$SPARKCLR_HOME/scripts" ] && mkdir "$SPARKCLR_HOME/scripts"
|
||||
|
||||
echo "Assemble SparkCLR Scala components"
|
||||
echo "Assemble Mobius Scala components"
|
||||
pushd "$FWDIR/../scala"
|
||||
|
||||
# clean the target directory first
|
||||
mvn clean -q
|
||||
[ $? -ne 0 ] && exit 1
|
||||
|
||||
# Note: Shade-plugin helps creates an uber-package to simplify SparkCLR job submission;
|
||||
# Note: Shade-plugin helps creates an uber-package to simplify running samples during CI;
|
||||
# however, it breaks debug mode in IntellJ. So enable shade-plugin
|
||||
# only in build.cmd to create the uber-package.
|
||||
# build the package
|
||||
|
|
@ -33,11 +33,11 @@ mvn package -Puber-jar -q
|
|||
|
||||
if [ $? -ne 0 ]
|
||||
then
|
||||
echo "Build SparkCLR Scala components failed, stop building."
|
||||
echo "Build Mobius Scala components failed, stop building."
|
||||
popd
|
||||
exit 1
|
||||
fi
|
||||
echo "SparkCLR Scala binaries"
|
||||
echo "Mobius Scala binaries"
|
||||
cp target/spark*.jar "$SPARKCLR_HOME/lib/"
|
||||
popd
|
||||
|
||||
|
|
@ -52,7 +52,7 @@ then
|
|||
done
|
||||
fi
|
||||
|
||||
echo "Assemble SparkCLR C# components"
|
||||
echo "Assemble Mobius C# components"
|
||||
pushd "$FWDIR/../csharp"
|
||||
|
||||
# clean any possible previous build first
|
||||
|
|
@ -62,23 +62,23 @@ pushd "$FWDIR/../csharp"
|
|||
|
||||
if [ $? -ne 0 ];
|
||||
then
|
||||
echo "Build SparkCLR C# components failed, stop building."
|
||||
echo "Build Mobius C# components failed, stop building."
|
||||
popd
|
||||
exit 1
|
||||
fi
|
||||
echo "SparkCLR C# binaries"
|
||||
echo "Mobius C# binaries"
|
||||
cp Worker/Microsoft.Spark.CSharp/bin/Release/* "$SPARKCLR_HOME/bin/"
|
||||
|
||||
echo "SparkCLR C# Samples binaries"
|
||||
echo "Mobius C# Samples binaries"
|
||||
# need to include CSharpWorker.exe.config in samples folder
|
||||
cp Worker/Microsoft.Spark.CSharp/bin/Release/* "$SPARKCLR_HOME/samples/"
|
||||
cp Samples/Microsoft.Spark.CSharp/bin/Release/* "$SPARKCLR_HOME/samples/"
|
||||
|
||||
echo "SparkCLR Samples data"
|
||||
echo "Mobius Samples data"
|
||||
cp Samples/Microsoft.Spark.CSharp/data/* "$SPARKCLR_HOME/data/"
|
||||
popd
|
||||
|
||||
echo "Assemble SparkCLR examples"
|
||||
echo "Assemble Mobius examples"
|
||||
pushd "$FWDIR/../examples"
|
||||
# clean any possible previous build first
|
||||
./clean.sh
|
||||
|
|
@ -86,13 +86,13 @@ pushd "$FWDIR/../examples"
|
|||
|
||||
if [ $? -ne 0 ];
|
||||
then
|
||||
echo "Build SparkCLR C# Examples failed, stop building."
|
||||
echo "Build Mobius C# Examples failed, stop building."
|
||||
popd
|
||||
exit 1
|
||||
fi
|
||||
popd
|
||||
|
||||
echo "Assemble SparkCLR script components"
|
||||
echo "Assemble Mobius script components"
|
||||
pushd "$FWDIR/../scripts"
|
||||
cp *.sh "$SPARKCLR_HOME/scripts/"
|
||||
popd
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ if "%1" == "" (
|
|||
@rem TODO: this check will fail if "--exe" only exists in the argument list of user application.
|
||||
if "%1" == "--exe" (
|
||||
set USER_EXE="true"
|
||||
@echo [RunSamples.cmd] Run user specified application, instead of SparkCLR samples.
|
||||
@echo [RunSamples.cmd] Run user specified application, instead of Mobius samples.
|
||||
)
|
||||
|
||||
rem - shift the arguments and examine %1 again
|
||||
|
|
|
|||
|
|
@ -257,7 +257,7 @@ function Download-BuildTools
|
|||
$gpgZip = "$toolsDir\gpg4win-vanilla-2.3.0.zip"
|
||||
if (!(test-path $gpgZip))
|
||||
{
|
||||
$url = "https://github.com/SparkCLR/build/blob/master/tools/gpg4win-vanilla-2.3.0.zip?raw=true"
|
||||
$url = "https://github.com/MobiusForSpark/build/blob/master/tools/gpg4win-vanilla-2.3.0.zip?raw=true"
|
||||
$output=$gpgZip
|
||||
Download-File $url $output
|
||||
# Unzip-File $output $toolsDir
|
||||
|
|
@ -495,8 +495,8 @@ function Print-Usage
|
|||
Write-Output ''
|
||||
Write-Output ' This script takes one input parameter ("stage"), which can be either [build | run].'
|
||||
Write-Output ''
|
||||
Write-Output ' Build: Download tools required in building SparkCLR;'
|
||||
Write-Output ' Run: Download Apache Spark and related binaries, required to run SparkCLR samples locally.'
|
||||
Write-Output ' Build: Download tools required in building Mobius;'
|
||||
Write-Output ' Run: Download Apache Spark and related binaries, required to run Mobius samples locally.'
|
||||
Write-Output ''
|
||||
Write-Output '====================================================================================================='
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ if not exist "%JAVA_HOME%\bin\java.exe" (
|
|||
@echo.
|
||||
@echo ============================================================================================
|
||||
@echo.
|
||||
@echo WARNING!!! %~nx0 detected JAVA_HOME is not set properly. SparkCLR requires JDK 7u85 and above,
|
||||
@echo WARNING!!! %~nx0 detected JAVA_HOME is not set properly. Mobius requires JDK 7u85 and above,
|
||||
@echo or JDK 8u60 and above. You can either download OpenJDK available at
|
||||
@echo http://www.azul.com/downloads/zulu/zulu-windows/, or use Oracle JDK.
|
||||
@echo.
|
||||
|
|
@ -33,7 +33,7 @@ goto :eof
|
|||
@echo ============================================================================================
|
||||
@echo.
|
||||
@echo WARNING!!! %~nx0 detected version of Visual Studio in current command prompt as %version%.
|
||||
@echo SparkCLR %~nx0 requires "Developer Command Prompt for VS2013" and above, or
|
||||
@echo Mobius %~nx0 requires "Developer Command Prompt for VS2013" and above, or
|
||||
@echo "MSBuild Command Prompt for VS2015" and above.
|
||||
@echo.
|
||||
@echo ============================================================================================
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ function Update-SparkClrSubmit($targetDir, $version)
|
|||
Write-Output "[SetSparkClrJarVersion.Update-SparkClrSubmit] Start setting sparkclr-submit.cmd under $targetDir to version=$version"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version to this release. The version string in sparkclr-submit.cmd -
|
||||
# Update Mobius package version to this release. The version string in sparkclr-submit.cmd -
|
||||
#
|
||||
# if not defined SPARKCLR_JAR (set SPARKCLR_JAR=spark-clr_2.10-1.5.2-SNAPSHOT.jar)
|
||||
#
|
||||
|
|
@ -48,7 +48,7 @@ function Update-SparkClrSubmit($targetDir, $version)
|
|||
Write-Output "[SetSparkClrJarVersion.Update-SparkClrSubmit] Start setting sparkclr-submit.sh under $targetDir to version=$version"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version to this release. The version string in sparkclr-submit.sh -
|
||||
# Update Mobius package version to this release. The version string in sparkclr-submit.sh -
|
||||
#
|
||||
# export SPARKCLR_JAR=spark-clr_2.10-1.6.0-SNAPSHOT.jar
|
||||
#
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
#
|
||||
# This script takes in and "nuspecDir" and "version" parameters, update SparkCLR Nuget package
|
||||
# This script takes in and "nuspecDir" and "version" parameters, update Mobius Nuget package
|
||||
# version
|
||||
#
|
||||
Param([string]$nuspecDir, [string]$version)
|
||||
|
|
@ -16,7 +16,7 @@ $validNugetVersion = $version -replace "(\s*\S*-\S*)(\.)(\S*)",'$1-$3'
|
|||
Write-Output "[SetSparkClrNugetPackageVersion] Start setting SparkCLR.nuspec under $nuspecDir to version=$validNugetVersion"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version. Example in SparkCLR.nuspec:
|
||||
# Update Mobius package version. Example in SparkCLR.nuspec:
|
||||
# <version>1.6.0-SNAPSHOT-1</version>
|
||||
#
|
||||
Get-ChildItem $nuspecDir -filter "SparkCLR.nuspec" | % {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
#
|
||||
# This script takes in "version" and "targetDir" (optional) parameters, update SparkCLR Nuget package
|
||||
# This script takes in "version" and "targetDir" (optional) parameters, update Mobius Nuget package
|
||||
# version reference in all *.csproj and packages.config under "dir".
|
||||
#
|
||||
# "targetDir" parameter is default to current directory where this script is located, when not provided.
|
||||
|
|
@ -17,7 +17,7 @@ function Update-Csproj($targetDir, $version)
|
|||
Write-Output "[SetSparkClrPackageVersion.Update-Csproj] Start setting *.csproj under $targetDir to version=$version"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version to this release. Example in *.csproj:
|
||||
# Update Mobius package version to this release. Example in *.csproj:
|
||||
# <HintPath>..\packages\Microsoft.SparkCLR.1.5.2-SNAPSHOT\lib\net45\CSharpWorker.exe</HintPath>
|
||||
#
|
||||
Get-ChildItem $targetDir -filter "*.csproj" -recurs | % {
|
||||
|
|
@ -39,7 +39,7 @@ function Update-PackageConfig($targetDir, $version)
|
|||
Write-Output "[SetSparkClrPackageVersion.Update-PackageConfig] Start setting packages.config under $targetDir to version=$version"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version to this release. Example in packages.config:
|
||||
# Update Mobius package version to this release. Example in packages.config:
|
||||
# <package id="Microsoft.SparkCLR" version="1.5.2-SNAPSHOT" targetFramework="net45" />
|
||||
#
|
||||
Get-ChildItem $targetDir -filter "packages.config" -recurs | % {
|
||||
|
|
@ -61,7 +61,7 @@ function Update_NuSpec($nuspecDir, $version)
|
|||
Write-Output "[SetSparkClrPackageVersion.Update-NuSpec] Start setting SparkCLR.nuspec under $nuspecDir to version=$version"
|
||||
|
||||
#
|
||||
# Update SparkCLR package version to this release. Example in SparkCLR.nuspec:
|
||||
# Update Mobius package version to this release. Example in SparkCLR.nuspec:
|
||||
# <version>1.5.2-SNAPSHOT</version>
|
||||
#
|
||||
Get-ChildItem $nuspecDir -filter "SparkCLR.nuspec" | % {
|
||||
|
|
|
|||
Двоичные данные
docs/SparkCLR.docx → docs/Mobius.docx
Двоичные данные
docs/SparkCLR.docx → docs/Mobius.docx
Двоичный файл не отображается.
Двоичный файл не отображается.
Двоичные данные
docs/SparkCLR.pptx
Двоичные данные
docs/SparkCLR.pptx
Двоичный файл не отображается.
|
|
@ -0,0 +1,8 @@
|
|||
# Mobius Configuration
|
||||
|Type |Property Name |Usage |
|
||||
|-----|-----|-----|
|
||||
|Worker |spark.mobius.CSharpWorker.maxProcessCount |Sets max number of C# worker processes in Spark executors |
|
||||
|Streaming (Kafka) |spark.mobius.streaming.kafka.CSharpReader.enabled |Enables use of C# Kafka reader in Mobius streaming applications |
|
||||
|Streaming (Kafka) |spark.mobius.streaming.kafka.maxMessagesPerTask.<topicName> |Sets the max number of messages per RDD partition created from specified Kafka topic to uniformly spread load across tasks that process them |
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
# Troubleshoot Errors in Mobius Apps
|
||||
|
||||
|Error |Error Details | Possible Fix |
|
||||
|----|------|------|
|
||||
|Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD |Exception java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0 | Try explicitly setting spark.local.dir command line parameter to some location like C:\Temp\SparkCLRTemp. If the default temp location (C:\Users\userName\AppData\Local\Temp) is used, Windows Defender might delete CSharpWorker.exe (thinking it is a malicious exe) when it gets downloaded by Spark into this temp location.|
|
||||
Загрузка…
Ссылка в новой задаче