Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue Libc Not Found

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

/lib64/libc.so.6: version `GLIBC_2.18' not found (required by ...)

Cause

The required version of GLIBC by the binary executor is not found on Spark nodes.

Solution

Recompile your …

General Tips on Programming

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Do NOT chase the latest versions of libraries/software/tools. Wait for some time for them to be tested thoroughly before adopting them.

  2. Follow a good Semantic Versioning if you release …

Data Quality

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  • Upper and lower bounds tests and Inter Quartile Range Checks(IQR) and standard deviations

  • Aggregate level checks (after manipulating data, there should still be the ability to explain how the data …