Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
A pandas_udf tag specifies a return type of String
but the corresponding pandas udf returns a different …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
RuntimeError: Arrow legacy IPC format is not supported in PySpark, please unset ARROW_PRE_0_15_IPC_FORMAT
You are using PySpark 3.0+ with one (or both) of the following options.
--conf …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
pyspark.sql.utils.AnalysisException: Found duplicate column(s) when inserting into ...
As the error message says, there are duplicated columns in your Spark SQL code.
Fix …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
py4j.protocol.Py4JJavaError: An error occurred while calling o156.getQuotaUsage.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): The quota system is disabled in Router.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
When you run Spark or PySpark in a Jupyter/Lab notebook, it is recommended that you show ERROR messages only. Otherwise, there might be too much logging information polluting your notebook. You can set the log level of Spark to ERROR using the following line of code.