Ben Chuanlong Du's Blog

It is never too late to learn.

Conversion Between PySpark DataFrames and pandas DataFrames

Comments

  1. A PySpark DataFrame can be converted to a pandas DataFrame by calling the method DataFrame.toPandas, and a pandas DataFrame can be converted to a PySpark DataFrame by calling SparkSession.createDataFrame. Notice that when you call DataFrame.toPandas to convert a Spark DataFrame to a pandas DataFrame, the whole Spark DataFrame is collected to the driver machine! This means that you should only call the method DataFrame.toPandas

Conda Build Issue

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

I encountered an issue building the package sqlalchemy-teradata. There are a few related issues: issues/2790, issues/2826. It seems that the recipe uses an outdated URL. Change the URL to …

Python Developing in PyCharm

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Change Scheme

  1. File -> Settings... -> Editor -> Colors & Fonts

  2. Choose a scheme right to "Scheme name"

Change Font of Terminal

  1. Search for Console Font

  2. Set the desired font size.

  3. Restart terminal (the font …

Tips on ANTLR4

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Install ANTLR4 in Ubuntu

wajig install antlr4
sudo pip3 install antlr4-python3-runtime==4.5

References