The code below returns the execution time in seconds.

Fold vs Reduce¶

fold takes an initial value, and the first invocation of the lambda you pass to it will receive that initial value and the first element of the collection as parameters. reduce doesn't take an initial value, but instead starts with the first element of the collection as the accumulator (called sum in the following example).

Tips and Traps¶

1. A static method cannot be abstract in Java. For refer to this discussion on Stackoverflow for rationale behind it. If you need the two together, what you are looking is probably singleton object with inheritance.

2. Singleton objects are more flexible and useful than static methods in Java. Please refer to Singleton Pattern Versus Static Class

An implementation of the singleton pattern must:

• ensure that only one instance of the singleton class ever exists;
• and provide global access to that instance.

Typically, this is done by:

• declaring all constructors of the class to be private;
• and providing a static method that returns a reference to the instance.

The instance is usually stored as a private static variable; the instance is created when the variable is initialized, at some point before the static method is first called. The following is a sample implementation written in Java.

Comments¶

1. String is a immutable class in Java. Extensive operations on strings (e.g., + in a big loop) is usually very slow before Java 7 (the + operator is optimized by the compiler automatically starting from Java 7). To avoid this problem (in older versions of Java), you can use the StringBuilder

Aggregation Without Grouping¶

1. You can aggregate all values in Columns of a DataFrame. Just use aggregation functions in select without groupBy, which is very similar to SQL syntax.

2. The aggregation functions all and any are available since Spark 3.0. However, they can be achieved using other aggregation functions such as sum