Comments¶

1. After sorting, rows in a DataFrame are sorted according to partition ID. And within each partition, rows are sorted. This property can be leverated to implement global ranking of rows. For more details, please refer to Computing global rank of a row in a DataFrame with Spark SQL. However, notice that multi-layer ranking is often more efficiency than a global ranking in big data applications.

AQE (Adaptive Query Execution)¶

To enable AQE, you have to set spark.sql.adaptive.enabled to true (using --conf spark.sql.adaptive.enabled=true in spark-submit or using `spark.config("spark.sql.adaptive,enabled", "true") in Spark/PySpark code.)

Pandas UDFs¶

Pandas UDFs are user defined functions that are executed by Spark using Arrow to transfer data to Pandas to work with the data, which allows vectorized operations. A Pandas UDF is defined using pandas_udf

framequery¶

References¶

https://github.com/apache/arrow

https://stackoverflow.com/questions/54582073/sharing-objects-across-workers-using-pyarrow

https://github.com/pytorch/pytorch/issues/13039

https://issues.apache.org/jira/browse/ARROW-5130

https://uwekorn.com/2019/09/15/how-we-build-apache-arrows-manylinux-wheels.html

Comments¶

1. As long as the class name is not need at definition time of the class, it is OK to use it.

You cannot use a class in default values of the __init__ function of the class.