See Spark crash for the first time: the phenomenon of Spark Shell memory OOM!

The first time I saw Spark crash.

The phenomenon of Spark Shell memory OOM

To do Spark diagram calculation, so the use of Google web-Google.txt, size 71.8MB.

To command:

Val graph = GraphLoader.edgeListFile (sc, "hdfs://192.168.0.10:9000/input/graph/web-Google.txt")

When building the graph, it was returned directly to the console after a long time of operation.

Interface xian

Scala > val graph = GraphLoader.edgeListFile (sc, "hdfs://192.168.0.10:9000/input/graph/web-Google.txt")

[Stage 0: > (0 + 2) / 2]. / bin/spark-shell: line 44: 3592 Killed "${SPARK_HOME}" / bin/spark-submit-- class org.apache.spark.repl.Main-- name "Spark shell"$@"

The second Spark crash!

Perform the pagerank algorithm test, using the data web-Google.txt of google, the size 71.8MB.

Val rank = graph.pageRank. Vertices

Spark is a cluster of master,2 worker. As a result, slave2 is down.

09:51:44 INFO cluster.SparkDeploySchedulerBackend on 16-11-14: Granted executor ID app-20161114084026-0000 on hostPort 1333 on hostPort 192.168.0.5 with 4 cores, 1024.0 MB RAM

09:51:44 on 16-11-14 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161114084026-0000Universe 1333 is now RUNNING

09:51:44 on 16-11-14 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161114084026-0000Universe 1333 is now EXITED (Command exited with code 1)

09:51:44 on 16-11-14 INFO cluster.SparkDeploySchedulerBackend: Executor app-20161114084026-0000Compact 1333 removed: Command exited with code 1

16-11-14 09:51:44 INFO cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 1333

09:51:44 on 16-11-14 INFO client.AppClient$ClientEndpoint: Executor added: app-20161114084026-0000max 1334 on worker-20161114083607-192.168.0.5-42898 (192.168.0.5 with 42898) with 4 cores

As a result, I have been trying to connect to slave2, hehe.