Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: Duplicated Partitions

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

There seems to be an issue in Spark that it might fail to overwrite files even if mode of spark.write is set to be "overwrite".

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …

Spark Issue: Unable to Find Encoder Type

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue Unable to find encoder for type stored in a Dataset

Solution …

Proxychains-Ng Issues on Mac

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

ProxyChains-ng does not work on macOS due to System Integration Protection (SIP). There are basically 3 ways to fix/avoid it.

  1. The issue only happens if you execute a system binary …

Common Errors Encountered in Scala and Solutions

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Java Version Issue

    Unsupported major minor version

    https://stackoverflow.com/questions/22489398/unsupported-major-minor-version-52-0

java.lang.NoSuchMethodError: scala.Product.\(init\)

Fixing the Scala error: java.lang.NoSuchMethodError: scala.Product.\(init\)

It probably …