Tips and Traps¶

1. Bucketed column is only supported in Hive table at this time.

2. A Hive table can have both partition and bucket columns.

3. Suppose t1 and t2 are 2 bucketed tables and with the number of buckets b1 and b2 respecitvely. For bucket optimization to kick in when joining them:

    - The 2 tables must be bucketed on the same keys/columns.
    - Must joining on the bucket keys/columns.
    - `b1` is a multiple of `b2` or `b2` is a multiple of `b1`.
   
   

Window with orderBy¶

It is tricky!!!

If you provide ORDER BY clause then the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW:

https://stackoverflow.com/questions/52273186/pyspark-spark-window-function-first-last-issue

1. Avoid using last and use first with descending order by instead. This gives less surprisings.

2. Do NOT use order by if not necessary. It introduces unnecessary ...