Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

21/06/01 15:03:28 INFO ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (6) reached)

Possible Causes

The option spark.yarn.max.executor …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

spark-submit and spark-shell

Overwrite the PATH environment variable before invoking spark-submit and/or spark-shell often resolves the issue.

Spark in Jupyter/Lab Notebooks

Remove or reset the environment variable HADOOP_CONF_DIR resolves …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

OutOfMemoryError

Cause

java.lang.OutOfMemoryError is thrown when there is not enough heap memory (for JVM to allocating new objects).

Solution

Increase executor memory.

--executor-memory=20G

Reference:

http://stackoverflow.com …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

15/12/24 02:04:21 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker endpoint = AkkaRpcEndpointRef(Actor[akka.tcp://sparkDriver@10.115.58.55:52077/user/MapOutputTracker#-1937024516])

15/12/24 …

** Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue

Mac spark-shell Error initializing SparkContext

Possible Causes

Solution

One possible solution is to simplfy restart Mac. Other solutions exists in the discussion on StackOverflow

Reference

https://stackoverflow.com/questions/34601554 …