зеркало из https://github.com/microsoft/spark.git
[SPARK-4620] Add unpersist in Graph and GraphImpl
Add an IF to uncache both vertices and edges of Graph/GraphImpl. This IF is useful when iterative graph operations build a new graph in each iteration, and the vertices and edges of previous iterations are no longer needed for following iterations. Author: Takeshi Yamamuro <linguin.m.s@gmail.com> This patch had conflicts when merged, resolved by Committer: Ankur Dave <ankurdave@gmail.com> Closes #3476 from maropu/UnpersistInGraphSpike and squashes the following commits: 77a006a [Takeshi Yamamuro] Add unpersist in Graph and GraphImpl
This commit is contained in:
Родитель
2e6b736b0e
Коммит
8817fc7fe8
|
@ -104,6 +104,12 @@ abstract class Graph[VD: ClassTag, ED: ClassTag] protected () extends Serializab
|
|||
*/
|
||||
def checkpoint(): Unit
|
||||
|
||||
/**
|
||||
* Uncaches both vertices and edges of this graph. This is useful in iterative algorithms that
|
||||
* build a new graph in each iteration.
|
||||
*/
|
||||
def unpersist(blocking: Boolean = true): Graph[VD, ED]
|
||||
|
||||
/**
|
||||
* Uncaches only the vertices of this graph, leaving the edges alone. This is useful in iterative
|
||||
* algorithms that modify the vertex attributes but reuse the edges. This method can be used to
|
||||
|
|
|
@ -70,6 +70,12 @@ class GraphImpl[VD: ClassTag, ED: ClassTag] protected (
|
|||
replicatedVertexView.edges.checkpoint()
|
||||
}
|
||||
|
||||
override def unpersist(blocking: Boolean = true): Graph[VD, ED] = {
|
||||
unpersistVertices(blocking)
|
||||
replicatedVertexView.edges.unpersist(blocking)
|
||||
this
|
||||
}
|
||||
|
||||
override def unpersistVertices(blocking: Boolean = true): Graph[VD, ED] = {
|
||||
vertices.unpersist(blocking)
|
||||
// TODO: unpersist the replicated vertices in `replicatedVertexView` but leave the edges alone
|
||||
|
|
Загрузка…
Ссылка в новой задаче