Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …

Alternatives to Docker Containers

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. LXD and Multipass are alternatives to Docker container. Docker is more lightweight than LXD which is more lightweight than Multipass (Docker < LXD < Multipass).

  2. Neither Docker nor LXD requires a CPU which …

Check the CPU Limit of a Docker Container

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Get the number in the file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us. Denote it as cfs_quota_us.

  2. Get the number in the file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us. Denote …

Check the Memory Limit of a Docker Container

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Docker 1.13+

Docker 1.13+ mounts the container's cgroup to /sys/fs/cgroup (this could change in future versions). You can check the limit using

cat /sys/fs/cgroup/memory …