Ben Chuanlong Du's Blog

It is never too late to learn.

Hands on the Python module Multiprocessing

Comments

  1. multiprocess is a fork of the Python standard libary multiprocessing . multiprocess extends multiprocessing to provide enhanced serialization, using dill. multiprocess leverages multiprocessing to support the spawning of processes using the API of the python standard library's threading module.

  2. multiprocessing.Pool.map does not work with lambda functions due to the fact that lambda functions cannot be pickled. There are multiple approaches to avoid the issue. You can define a function or use functools.partial

Parallel Computing in Bash

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. & and wait

  2. parallel

  3. xargs

parallel is a cool bash command which makes parallel computing easy in Bash. It is a parallel version replacement of xargs.