Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
java.lang.IllegalArgumentException: Wrong FS: hdfs://..., expected: viewfs://...
The Spark cluster has migrated to Router-based Federation (RBF) namenodes,
and viewfs:// (instead of hdfs://) is required to access HDFS …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://cluster-name-ns02/
As the error message says,
viewfs://cluster-name-ns02 is not configured.
It is possible that …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
A pandas_udf tag specifies a return type of String
but the corresponding pandas udf returns a different …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
RuntimeError: Arrow legacy IPC format is not supported in PySpark, please unset ARROW_PRE_0_15_IPC_FORMAT
You are using PySpark 3.0+ with one (or both) of the following options.
--conf …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
pyspark.sql.utils.AnalysisException: Found duplicate column(s) when inserting into ...
As the error message says, there are duplicated columns in your Spark SQL code.
Fix …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
py4j.protocol.Py4JJavaError: An error occurred while calling o156.getQuotaUsage.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): The quota system is disabled in Router.