Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
15/12/24 02:04:21 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker endpoint = AkkaRpcEndpointRef(Actor[akka.tcp://sparkDriver@10.115.58.55:52077/user/MapOutputTracker#-1937024516])
15/12/24 …
** Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Mac spark-shell Error initializing SparkContext
One possible solution is to simplfy restart Mac. Other solutions exists in the discussion on StackOverflow
https://stackoverflow.com/questions/34601554 …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
High disk and memory spill when doing shuffle.
Insufficient executor memory (you can monitor this spill metrics from Spark UI).
Increase executor memory.
--executor-memory=4G
For jobs that …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: Table not found
java.lang.RuntimeException: Table Not Found: my_rdd
Miss-spelled a table name.
Correct miss-spelling.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: cannot resolve ...
Miss-spell a column name or refer to a column which does not exist in the DataFrame.
Correct the column name or …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: Path does not exist ...
A specified HDFS path does not exist.
Use the correct HDFS path.