Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
High disk and memory spill when doing shuffle.
Insufficient executor memory (you can monitor this spill metrics from Spark UI).
Increase executor memory.
--executor-memory=4G
For jobs that …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: Table not found
java.lang.RuntimeException: Table Not Found: my_rdd
Miss-spelled a table name.
Correct miss-spelling.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
16/04/17 11:17:36 ERROR scheduler.TaskSetManager: Total size of serialized results of 126 tasks (1137.3 MB) is bigger than spark.driver.maxResultSize (1024.0 …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
15/12/10 07:44:21 ERROR shuffle.OneForOneBlockFetcher: Failed while starting block fetches
java.lang.RuntimeException: java.io.FileNotFoundException: /hadoop/1/scratch/local/usercache/dclong/appcache/application_1447357188616_340392 …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to Spark Issue: _Pickle.Picklingerror: Args[0] from Newobj Args Has the Wrong Class for a similar serialization issue in PySpark.
org.apache.spark.SparkException: Job aborted …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.shuffle.FetchFailedException: Too large frame: 2200180718 Caused by: java.lang.IllegalArgumentException: Too large frame: 2200289525 at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119)
There …