Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
TypeError: withReplacement (optional), fraction (required) and seed (optional) should be a bool, float and number; however, got [
].
An integer number (e.g., 1) is passed to the fraction parameter …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=16 …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to http://www.legendu.net/misc/tag/spark-issue.html for a comprehensive list of Spark Issues and (possible) causes and solutions.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by ...)
The required version of GLIBC by the binary executor is not found on Spark nodes.
Recompile your …