Tips and Traps¶

1. If you use PySpark instead of Spark/Scala, pandas udf is a great alternative to all those (complicated) collections functions discussed here. Leveraging pandas udf, each partition of a Spark DataFrame can be converted to a pandas DataFrame without copying the underlying data, you can then do transforms on pandas DataFrames which will be converted back to partitons of a Spark DataFrame.

https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html

https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html#approxQuantile-java.lang.String:A-double:A-double-

https://spark.apache.org/docs/latest/api/sql/index.html#approx_percentile

Window with orderBy¶

It is tricky!!!

If you provide ORDER BY clause then the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW:

https://stackoverflow.com/questions/52273186/pyspark-spark-window-function-first-last-issue

1. Avoid using last and use first with descending order by instead. This gives less surprisings.

2. Do NOT use order by if not necessary. It introduces unnecessary ...