https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html

https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html#approxQuantile-java.lang.String:A-double:A-double-

https://spark.apache.org/docs/latest/api/sql/index.html#approx_percentile

Window with orderBy¶

It is tricky!!!

If you provide ORDER BY clause then the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW:

https://stackoverflow.com/questions/52273186/pyspark-spark-window-function-first-last-issue

1. Avoid using last and use first with descending order by instead. This gives less surprisings.

2. Do NOT use order by if not necessary. It introduces unnecessary ...

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

%time¶

Measue the execuation time of the code ONCE.

%timeit¶

Measure the execuation time of the code (accurately) by running it MULTIPLE TIMES and taking the average.

%prun | %%prun¶

-D: output the profiling results into a file so that you can other tools (e.g., snakeviz) to visualize it.

Noticd that %prun and %%prun are based on cProfile