Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: InvalidInputException for Some Hive Data Partitions

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

15/12/29 17:22:27 ERROR yarn.ApplicationMaster: User class …

Spark Issue: a Master URL Must Be Set in Your Configuration

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

Error initializzing SparkContext: A master URL must be set in your configuration.

Possible Causes

The master of Spark cluster is not specified.

Solutions

Add .master("yarn") into the following …

Spark Issue: Spark Application Submission Is Not Finished

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

Application submission is not finished, submitted application application__1524215324275_0081 is still …

Spark Issue: Cannot Create a Path from An Empty String

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue

java.lang.IllegalArgumentException: Can not create a Path from an empty string

Possible Causes

The error you are seeing could be from number of things:

  1. parameters , check for ${param} in …

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …