Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Caused by: org.apache.spark.SparkException: Could not execute broadcast in 600 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
16/04/17 11:17:36 ERROR scheduler.TaskSetManager: Total size of serialized results of 126 tasks (1137.3 MB) is bigger than spark.driver.maxResultSize (1024.0 …
BroadcastHashJoin, i.e., map-side join is fast. Use BroadcastHashJoin if possible. Notice that Spark will automatically use BroacastHashJoin if a table in inner join has a size less then the configured BroadcastHashJoin limit.
Notice that BroadcastJoin only works for inner joins. If you have a outer join, BroadcastJoin won't happend even if you explicitly Broadcast a DataFrame.
The broadcast concept in numpy
is essentially a way to "virtually" duplicate data in a numpy array
so that it is "virtually" reshaped to be compatible with another numpy array for a certain operation.
Do not confused yourself about it with the broadcast concept in Spark
which sends a full copy of a (small) DataFrame to each work node for BroadCastJoin