Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://clustername-router/

Possible Causes

This is due to missing clustername-router settings in the property dfs.nameservices in …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs::/cluster-name/user/dclong/feature_example/features/train/2022-03-11

Possible Causes

As the error message points out, there's a syntax …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

CalledProcessError: Command './pine' died with .

Possible Causes

SIGBUS (bus error) is a signal that happens when you try to access memory that has not been physically mapped . There are several …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

Block rdd_123_456 could not be removed as it was not found on disk or in memory.

Cause

1. The execution plan of a DataFrame is too complicated.

2. Not enough memory to …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

Caused by: org.apache.spark.SparkException: Could not execute broadcast in 600 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

This post collects some typical pure Python errors in PySpark applications.

Symptom 1

object has no attribute

Solution 1

Fix the attribute name.

Symptom 2

No such file or directory

Solution …