Ben Chuanlong Du's Blog

It is never too late to learn.

Sum Type in Rust

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Enum is the preferred way to constrcut a sum type of several types (which does not implemente the same trait).

The Rust crate either provides an enum Either (with variants Left …

Spark Issue: Pure Python Code Errors

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

This post collects some typical pure Python errors in PySpark applications.

Symptom 1

object has no attribute

Solution 1

Fix the attribute name.

Symptom 2

No such file or directory

Solution …

Spark Issue: TypeError WithReplacement

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

TypeError: withReplacement (optional), fraction (required) and seed (optional) should be a bool, float and number; however, got [].

Causes

An integer number (e.g., 1) is passed to the fraction parameter …

Computer Vision Libraries in Rust

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

resize

resize Image resampling library in pure Rust. - Fast, with support for many pixel formats - No encoders/decoders, meant to be used with some external library - Tuned for resizing to the …

Configure Log4J for Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Show Error Messages Only

When you run Spark or PySpark in a Jupyter/Lab notebook, it is recommended that you show ERROR messages only. Otherwise, there might be too much logging information polluting your notebook. You can set the log level of Spark to ERROR using the following line of code.