Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Comments¶

1. A PySpark DataFrame can be converted to a pandas DataFrame by calling the method DataFrame.toPandas, and a pandas DataFrame can be converted to a PySpark DataFrame by calling SparkSession.createDataFrame. Notice that when you call DataFrame.toPandas to convert a Spark DataFrame to a pandas DataFrame, the whole Spark DataFrame is collected to the driver machine! This means that you should only call the method DataFrame.toPandas

https://medium.com/@mrpowers/manually-creating-spark-dataframes-b14dae906393

Seq.toDF¶

toDF() provides a concise syntax for creating DataFrames and can be accessed after importing Spark implicits.

import spark.implicits._

SparkSession.createDataFrame¶