Mirror of Apache Spark
Перейти к файлу
Matei Zaharia efc5423210 Made compression configurable separately for shuffle, broadcast and RDDs 2012-10-07 11:30:53 -07:00
bagel/src Removed the need to sleep in tests due to waiting for Akka to shut down 2012-10-07 00:17:59 -07:00
bin Update license info on deploy scripts 2012-09-25 14:43:47 -07:00
conf Settings variables and bugfix for stop script. 2012-08-02 15:59:39 -07:00
core/src Made compression configurable separately for shuffle, broadcast and RDDs 2012-10-07 11:30:53 -07:00
docs Made compression configurable separately for shuffle, broadcast and RDDs 2012-10-07 11:30:53 -07:00
ec2 Made region used by spark-ec2 configurable. 2012-08-28 22:40:00 -07:00
examples/src/main More updates to documentation 2012-09-25 19:31:07 -07:00
project Removes the included mesos-0.9.0.jar and adds a libraryDependency to 2012-10-03 08:58:05 -07:00
repl Removed the need to sleep in tests due to waiting for Akka to shut down 2012-10-07 00:17:59 -07:00
sbt Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216 2012-10-07 04:19:16 +00:00
.gitignore Ignore file spark-tests.log in git 2012-10-01 15:08:20 -07:00
LICENSE Added BSD license 2010-12-07 10:32:17 -08:00
README.md Simplify README even further 2012-10-02 22:14:40 -07:00
kmeans_data.txt Fixed bugs 2012-01-09 11:59:52 -08:00
lr_data.txt Test commit 2012-02-06 09:58:06 -08:00
run Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216 2012-10-07 04:19:16 +00:00
run.cmd Add spark-shell.cmd 2012-09-25 07:26:29 -07:00
run2.cmd Don't check for JARs in core/lib anymore 2012-10-04 15:11:43 -07:00
spark-executor Further refactoring, and start of a standalone scheduler backend 2012-07-06 17:56:44 -07:00
spark-shell More work to allow Spark to run on the standalone deploy cluster. 2012-07-08 14:00:04 -07:00
spark-shell.cmd Add spark-shell.cmd 2012-09-25 07:26:29 -07:00

README.md

Spark

Lightning-Fast Cluster Computing - http://www.spark-project.org/

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project webpage at http://spark-project.org/documentation.html. This README file only contains basic setup instructions.

Building

Spark requires Scala 2.9.2. The project is built using Simple Build Tool (SBT), which is packaged with it. To build Spark and its example programs, run:

sbt/sbt compile

To run Spark, you will need to have Scala's bin in your PATH, or you will need to set the SCALA_HOME environment variable to point to where you've installed Scala. Scala must be accessible through one of these methods on Mesos slave nodes as well as on the master.

To run one of the examples, use ./run <class> <params>. For example:

./run spark.examples.SparkLR local[2]

will run the Logistic Regression example locally on 2 CPUs.

Each of the example programs prints usage help if no params are given.

All of the Spark samples take a <host> parameter that is the Mesos master to connect to. This can be a Mesos URL, or "local" to run locally with one thread, or "local[N]" to run locally with N threads.

A Note About Hadoop

Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported storage systems. Because the HDFS API has changed in different versions of Hadoop, you must build Spark against the same version that your cluster runs. You can change the version by setting the HADOOP_VERSION variable at the top of project/SparkBuild.scala, then rebuilding Spark.

Configuration

Spark can be configured through two files: conf/java-opts and conf/spark-env.sh.

In java-opts, you can add flags to be passed to the JVM when running Spark.

In spark-env.sh, you can set any environment variables you wish to be available when running Spark programs, such as PATH, SCALA_HOME, etc. There are also several Spark-specific variables you can set:

  • SPARK_CLASSPATH: Extra entries to be added to the classpath, separated by ":".

  • SPARK_MEM: Memory for Spark to use, in the format used by java's -Xmx option (for example, -Xmx200m means 200 MB, -Xmx1g means 1 GB, etc).

  • SPARK_LIBRARY_PATH: Extra entries to add to java.library.path for locating shared libraries.

  • SPARK_JAVA_OPTS: Extra options to pass to JVM.

  • MESOS_NATIVE_LIBRARY: Your Mesos library, if you want to run on a Mesos cluster. For example, this might be /usr/local/lib/libmesos.so on Linux.

Note that spark-env.sh must be a shell script (it must be executable and start with a #! header to specify the shell to use).