Comments¶

Spark DataFrame is an alias to Dataset[Row]. Even though a Spark DataFrame is stored as Rows in a Dataset, built-in operations/functions (in org.apache.spark.sql.functions) for Spark DataFrame are Column-based. Sometimes, there might be transformations on a DataFrame that is hard to express as Column expressions but rather evey convenient to express as Row expressions. The traditional way to resolve this issue is to wrap the row-based function into a UDF. It is worthing knowing that Spark DataFrame supports map/flatMap APIs which works on Rows. They are still experimental as Spark 2.4.3. It is suggested that you stick to Column-based operations/functions until the Row-based methods mature.

https://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/

Comment¶

1. The order of elements within each group are preserved (as the original order).

2. groupby works exactly the same on index if the index is named.

3. The order of columns in groupby matters if you want unstack the results later.

4. groupby works on columns too and it can group by some level of a MultiIndex.