Ben Chuanlong Du's Blog

It is never too late to learn.

Logging in PySpark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Excessive logging is better than no logging! This is generally true in distributed big data applications.

  2. Use loguru if it is available. If you have to use the logging module, be aware of traps in using it. For more details, please refer to Hands on the logging Module in Python.

Comments