Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: ArrowTypeError: Expect a Type but Got a Different Type

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64

Possible Causes

A pandas_udf tag specifies a return type of String but the corresponding pandas udf returns a different …