Change the Index of a pandas DataFrame

Jul 31, 2020

Assign Index¶

Jul 19, 2020

You can use withColumnRenamed to rename a column in a DataFrame. You can also do renaming using alias when select columns.

Nov 14, 2016

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

The reason is due to missing SSH.

The solution is simply to install openssh-client.

sudo apt-get install openssh

Jul 14, 2017

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

It is hard to figure out the version of Python from the version of the Docker image.

Jul 04, 2020

After sorting, rows in a DataFrame are sorted according to partition ID. And within each partition, rows are sorted. This property can be leverated to implement global ranking of rows. For more details, please refer to Computing global rank of a row in a DataFrame with Spark SQL. However, notice that multi-layer ranking is often more efficiency than a global ranking in big data applications.