Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Container Killed by Yarn for Exceeding Memory Limits

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Symptom 1

Container killed by YARN for exceeding memory limits.
22.0 GB of 19 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled …

Spark Issue: Data Skew on Shuffle Phase

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

org.apache.spark.shuffle.FetchFailedException: Too large frame: 2200180718 Caused by: java.lang.IllegalArgumentException: Too large frame: 2200289525 at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119)

Reason

There …

Poker Game in Casinos

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Vegas

Best place for gambling games in the US.

Washington

Max bet is 300 in non-tribals, 500 on tribal casinos. There is a 4-bet cap (3 raises) in both types of …