microsoft
/
spark
зеркало из https://github.com/microsoft/spark.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
938 коммитов 13 Ветки 28 Теги 178 MiB
Граф коммитов

938 Коммитов

Автор SHA1 Сообщение Дата
Denny 24b9b37314 Subclass URLClassLoader instead of using reflection 2012-09-11 16:51:08 -07:00
Denny 31c53e917d Use stageId as index for fileSet caches. 2012-09-11 16:10:45 -07:00
Denny 4d3471dd07 Fix serialization bugs and added local cluster tests 2012-09-10 15:39:58 -07:00
Denny 94a7e82ba1 Merge branch 'dev' of git://github.com/mesos/spark into feature/fileserver 2012-09-10 12:54:20 -07:00
Denny b864c36a30 Dynamically adding jar files and caching fileSets. 2012-09-10 12:49:09 -07:00
Denny f275fb07da General FileServer 
A general fileserver for both JARs and regular files.
 2012-09-10 12:48:59 -07:00
Matei Zaharia a13780670d Added a unit test for local-cluster mode and simplified some of the code involved in that 2012-09-10 12:48:58 -07:00
Denny f2ac55840c Add shutdown hook to Executor Runner and execute code to shutdown local cluster in Scheduler Backend 2012-09-10 12:48:58 -07:00
Denny 9ead8ab14e Set SPARK_LAUNCH_WITH_SCALA=0 in Executor Runner 2012-09-10 12:48:58 -07:00
Denny 8bb3c73977 Renamed spark-cluster to spark-local. 2012-09-10 12:48:58 -07:00
Denny a367c20f49 Fix wrong counting 2012-09-10 12:48:57 -07:00
Denny 93fe331e6d Delete old DeployUtils. 2012-09-10 12:48:57 -07:00
Denny cf074f9c96 Renamed class. 2012-09-10 12:48:57 -07:00
Denny 3749f94184 Start a standalone cluster locally. 2012-09-10 12:48:57 -07:00
Matei Zaharia 995982b3c9 Added a unit test for local-cluster mode and simplified some of the code involved in that 2012-09-07 17:08:36 -07:00
Matei Zaharia 8d2fcc2832 Merge pull request #189 from dennybritz/feature/localcluster 
Simulating a Spark standalone cluster locally
 2012-09-07 15:43:43 -07:00
Denny 7ff9311add Add shutdown hook to Executor Runner and execute code to shutdown local cluster in Scheduler Backend 2012-09-07 14:09:12 -07:00
Denny 4e7b264cf7 Set SPARK_LAUNCH_WITH_SCALA=0 in Executor Runner 2012-09-07 11:39:44 -07:00
root c2da64409a Randomize the order of block fetches in getMultiple 2012-09-06 23:16:26 +00:00
Denny 886183e591 Renamed spark-cluster to spark-local. 2012-09-05 17:10:54 -07:00
Matei Zaharia 53a5681c8a Merge pull request #190 from rxin/dev 
Log cache add/remove messages in block manager.
 2012-09-05 16:41:52 -07:00
Reynold Xin c308fbcb79 Removed cache add/remove log messages from CacheTracker. 
Added log messages on BlockManagerMaster to reflect block add/remove.
Also did some minor cleanup of storage package code.
 2012-09-05 15:59:48 -07:00
Denny babbca0a2f Fix wrong counting 2012-09-04 22:04:18 -07:00
Denny 9326509f66 Delete old DeployUtils. 2012-09-04 21:15:23 -07:00
Denny 1588d4dbe6 Renamed class. 2012-09-04 21:13:25 -07:00
Denny 22dde6e020 Start a standalone cluster locally. 2012-09-04 20:56:30 -07:00
Matei Zaharia a842c63044 Minor formatting fixes 2012-09-03 16:24:00 -07:00
Matei Zaharia 2d6a629f8c Merge pull request #182 from HarveyFeng/dev-fetch 
Add a limit on the number of parallel fetches in the reduce stage
 2012-09-03 16:14:57 -07:00
Harvey 3076b038f4 Start fetching a remote block when a received remote block has been passed 
to the reduce function
 2012-09-01 12:01:35 -07:00
Matei Zaharia 389fb4cc54 End runJob() with a SparkException when a task fails too many times in 
one of the cluster schedulers.
 2012-08-31 17:47:43 -07:00
Matei Zaharia a480dec6b2 Deserialize multi-get results in the caller's thread. This fixes an 
issue with shared buffers in the KryoSerializer.
 2012-08-30 20:01:06 -07:00
Matei Zaharia bf3212615a Merge pull request #184 from rxin/dev 
Disable running combiners on map tasks when mergeCombiners function is not specified by the user.
 2012-08-30 14:12:40 -07:00
Reynold Xin a8a2a08a1a Added a test for testing map-side combine on/off switch. 2012-08-30 12:34:28 -07:00
Reynold Xin 5945bcdcc5 Added a new flag in Aggregator to indicate applying map side combiners. 2012-08-29 23:32:08 -07:00
Reynold Xin c68e820b2a Merge branch 'dev' of github.com:mesos/spark into dev 2012-08-29 23:01:19 -07:00
Reynold Xin 940869dfda Disable running combiners on map tasks when mergeCombiners function is 
not specified by the user.
 2012-08-29 23:00:02 -07:00
Matei Zaharia 47507d69d9 Made region used by spark-ec2 configurable. 2012-08-28 22:40:00 -07:00
Matei Zaharia bf2e9cb08e Fault tolerance and block store fixes discovered through streaming tests. 2012-08-27 23:07:50 -07:00
Matei Zaharia 291abc2c28 Merge pull request #181 from rxin/dev 
Removed the deserialization cache for ShuffleMapTask
 2012-08-27 22:38:22 -07:00
Reynold Xin 3a6a95dc24 Removed the deserialization cache for ShuffleMapTask because it was 
causing concurrency problems (some variables in Shark get set to null).
The cost of task deserialization on slaves is trivial compared with the
execution time of the task anyway.
 2012-08-27 22:33:15 -07:00
Matei Zaharia 51453eb87b Merge pull request #179 from JoshRosen/fix/sparklr-caching 
Cache points in SparkLR example
 2012-08-26 15:32:50 -07:00
Josh Rosen 566feafe1d Cache points in SparkLR example. 2012-08-26 15:24:43 -07:00
Matei Zaharia 2c16ae36d7 Set log level in tests to WARN 2012-08-23 20:38:14 -07:00
Matei Zaharia deedb9e7b7 Fix further issues with tests and broadcast. 
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
 2012-08-23 20:31:49 -07:00
Matei Zaharia 59b831b9d1 Fixed test failures due to broadcast not stopping correctly 2012-08-23 19:59:55 -07:00
Matei Zaharia 7310a6f499 Merge pull request #147 from mosharaf/dev 
Broadcast refactoring/cleaning up
 2012-08-23 19:38:28 -07:00
Matei Zaharia 25a6a39e6d Added other SparkContext constructors to JavaSparkContext 2012-08-19 18:59:16 -07:00
Matei Zaharia 9a0c128fec Merge pull request #172 from dennybritz/dev 
Rsync root directory in EC2 script
 2012-08-14 13:05:22 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00