Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …