Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Docker has a limit on maximum apt cache size on the host system. The issue can be fixed by running the following command.

docker image prune -f

References

Repository is not …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

OutOfMemoryError

Cause

java.lang.OutOfMemoryError is thrown when there is not enough heap memory (for JVM to allocating new objects).

Solution

Increase executor memory.

--executor-memory=20G

Reference:

http://stackoverflow.com …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

15/12/24 02:04:21 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker endpoint = AkkaRpcEndpointRef(Actor[akka.tcp://sparkDriver@10.115.58.55:52077/user/MapOutputTracker#-1937024516])

15/12/24 …

** Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue

Mac spark-shell Error initializing SparkContext

Possible Causes

Solution

One possible solution is to simplfy restart Mac. Other solutions exists in the discussion on StackOverflow

Reference

https://stackoverflow.com/questions/34601554 …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symtom

High disk and memory spill when doing shuffle.

Cause

Insufficient executor memory (you can monitor this spill metrics from Spark UI).

Solution

1. Increase executor memory.

   --executor-memory=4G
   

2. For jobs that …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom 1

org.apache.spark.sql.AnalysisException: Table not found

Symptom 2

java.lang.RuntimeException: Table Not Found: my_rdd

Cause 1

Miss-spelled a table name.

Solution 1

Correct miss-spelling.

Cause 2 …