Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
High disk and memory spill when doing shuffle.
Insufficient executor memory (you can monitor this spill metrics from Spark UI).
Increase executor memory.
--executor-memory=4G
For jobs that …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: Table not found
java.lang.RuntimeException: Table Not Found: my_rdd
Miss-spelled a table name.
Correct miss-spelling.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: cannot resolve ...
Miss-spell a column name or refer to a column which does not exist in the DataFrame.
Correct the column name or …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.sql.AnalysisException: Path does not exist ...
A specified HDFS path does not exist.
Use the correct HDFS path.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.hadoop.security.AccessControlException: Permission denied ...
The user of the Spark application has no permission to the query a table or HDFS path.
Apply to access to …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
org.apache.spark.InsertOperationConflictException: Failed to hold insert operation lock ...
Multiple Spark applications attempts to write to the same directory at the same time.
Resubmit your Spark application …