Tips & Traps¶

1. Optimus requires Python 3.6+.

Comments¶

1. multiprocess is a fork of the Python standard libary multiprocessing . multiprocess extends multiprocessing to provide enhanced serialization, using dill. multiprocess leverages multiprocessing to support the spawning of processes using the API of the python standard library's threading module.

2. multiprocessing.Pool.map does not work with lambda functions due to the fact that lambda functions cannot be pickled. There are multiple approaches to avoid the issue. You can define a function or use functools.partial

https://docs.python.org/3/library/threading.html#module-threading

https://docs.python.org/3/library/_thread.html#module-_thread

Comemnt¶

1. The methods BufferedRead.readLine and BufferedRead.lines are very helpful for reading text Files.

public String BufferedRead.readLine¶

Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.

public Stream\<String> BufferedRead.lines¶

Returns a Stream, the elements of which are lines read from this BufferedReader. The Stream is lazily populated, i.e., read only occurs during the terminal stream operation. The reader must not be operated on during the execution of the terminal stream operation. Otherwise, the result of the terminal stream operation is undefined.

Python has built-in functions and object that users can use directly (no need to import). However, if you import another module which hide a built-in function or object, you cannot use it anymore. For example, sum is a built-in function in Python which can be used directly. However, if you use PySpark import SQL functions (from pyspark.sql.functions import *