Tips and Traps¶

1. BroadcastHashJoin, i.e., map-side join is fast. Use BroadcastHashJoin if possible. Notice that Spark will automatically use BroacastHashJoin if a table in inner join has a size less then the configured BroadcastHashJoin limit.

2. Notice that BroadcastJoin only works for inner joins. If you have a outer join, BroadcastJoin won't happend even if you explicitly Broadcast a DataFrame.

Reference¶

https://pandas.pydata.org/pandas-docs/stable/merging.html

http://stackoverflow.com/questions/22676081/pandas-the-difference-between-join-and-merge

Comment¶

1. You are able to specify (via left_on and right_on) which columns to join in each data frame.

2. Columns that appear in both data frames but not used in joining are distinguished using suffixes.