Ben Chuanlong Du's Blog

It is never too late to learn.

String Functions in Spark

Tips and Traps

  1. You can use the split function to split a delimited string into an array. It is suggested that removing trailing separators before you apply the split function. Please refer to the split section before for more detailed discussions.

  2. Some string functions (e.g., right, etc.) are available in the Spark SQL APIs but not available as Spark DataFrame APIs.

The Case Statement and the when Function in Spark

Tips and Traps

  1. Watch out for NaNs ..., behave might not what you expect ...

  2. None can be used for otherwise and yield null in DataFrame.

Column alias and postional columns can be used in group by in Spark SQL!!!

Notice the function when behaves like if-else.

Date and Time in Java and Scala

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Use Joda time if you are using JDK <= 7 and java.time if you are using JDK8 and above.

If you do prefer Scala libraries (when working in Scala), https://github …