Ben Chuanlong Du's Blog

It is never too late to learn.

Ensure Capturing Log of Applications

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Logging is critical for debugging applications. For production applications, it is best to send log information into a file instead of the standard output so that the log information is persisted …

General Tips on Logging

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Many logging libraries support sending logs as emails or as attachments of emails. As a poor man's way of subscribing to errors and warnings (if there's no engineering teams support parsing …

Logging in PySpark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Excessive logging is better than no logging! This is generally true in distributed big data applications.

  2. Use loguru if it is available. If you have to use the logging module, be …