Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Tips and Traps¶

1. LazyCsvReader is more limited compared to CsvReader. CsvReader support specifying schema while LazyCsvReader does not.

2. An empty filed is parsed as null instead of an empty string by default. And there is no way to change this behavior at this time. Please refer to this issue

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Tips and Traps¶

1. You can use the method Series.astype to cast the type of a series.

2. Series.astype(str) converts NaNs to the string literal nan. This is often NOT what people want. A better way is to use Series.astype(object)

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

References¶

https://stackoverflow.com/questions/42171499/get-current-number-of-partitions-of-a-dataframe

coalesce vs repartition¶

https://hackernoon.com/managing-spark-partitions-with-coalesce-and-repartition-4050c57ad5c4m

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Tips and Traps¶

Alternatives to pandas for Small Data¶

1. Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow as memory model. It is the best replacement of pandas for small data at this time. Notice that Polars support multithreading and lazy computation but it cannot handle data larger than memory at this time.

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!