Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

pyspark.sql.utils.AnalysisException: Found duplicate column(s) when inserting into ...

Possible Causes

As the error message says, there are duplicated columns in your Spark SQL code.

Possible Solutions

Fix …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom I

py4j.protocol.Py4JJavaError: An error occurred while calling o156.getQuotaUsage.

Symptom II

org.apache.hadoop.ipc.RemoteException(java.io.IOException): The quota system is disabled in Router.

Possible Causes …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://clustername-router/

Possible Causes

This is due to missing clustername-router settings in the property dfs.nameservices in …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs::/cluster-name/user/dclong/feature_example/features/train/2022-03-11

Possible Causes

As the error message points out, there's a syntax …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

CalledProcessError: Command './pine' died with .

Possible Causes

SIGBUS (bus error) is a signal that happens when you try to access memory that has not been physically mapped . There are several …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

Block rdd_123_456 could not be removed as it was not found on disk or in memory.

Cause

1. The execution plan of a DataFrame is too complicated.

2. Not enough memory to …