Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Table Not Found

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom 1

org.apache.spark.sql.AnalysisException: Table not found

Symptom 2

java.lang.RuntimeException: Table Not Found: my_rdd

Cause 1

Miss-spelled a table name.

Solution 1

Correct miss-spelling.

Cause 2 …

Spark Issue: AnalysisException: Cannot Resolve

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

org.apache.spark.sql.AnalysisException: cannot resolve ...

Cause

Miss-spell a column name or refer to a column which does not exist in the DataFrame.

Solution

Correct the column name or …

Spark Issue: AnalysisException: Path Does Not Exist

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

org.apache.spark.sql.AnalysisException: Path does not exist ...

Cause

A specified HDFS path does not exist.

Solution

Use the correct HDFS path.

Spark Issue: AccessControlException: Permission Denied

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

org.apache.hadoop.security.AccessControlException: Permission denied ...

Cause

The user of the Spark application has no permission to the query a table or HDFS path.

Solution

Apply to access to …

Spark Issue: Container Killed by Yarn for Exceeding Memory Limits

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Symptom 1

Container killed by YARN for exceeding memory limits.
22.0 GB of 19 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled …