Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Too Large Table for Auto BroadcastHashJoin

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Symptom 1

16/04/17 11:17:36 ERROR scheduler.TaskSetManager: Total size of serialized results of 126 tasks (1137.3 MB) is bigger than spark.driver.maxResultSize (1024.0 …