Ben Chuanlong Du's Blog

It is never too late to learn.

Handling Complicated Data Types in Python and PySpark

Tips and Traps

  1. An element in a pandas DataFrame can be any (complicated) type in Python. To save a padnas DataFrame with arbitrary (complicated) types as it is, you have to use the pickle module . The method pandas.DataFrame.to_pickle (which is simply a wrapper over pickle.dump) serialize the DataFrame to a pickle file while the method pandas.read_pickle

Null, None and Alike in Scala

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Nil represents an emptry List of anything of zero length. It is not that it refers to nothing but it refers to List which has no contents. In other words, Nil …