Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
py4j.protocol.Py4JJavaError: An error occurred while calling o156.getQuotaUsage.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): The quota system is disabled in Router.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Define command as ["/busybox/sh", "-c", "tail -f /dev/null"]
instead of
["/busybox/sh", "-c", "tail", "-f", "/dev/null"]
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The error message "Modulenotfounderror: No Module Named _Ctypes" is thrown when intalling packages in Python 3.7.
libffi-dev is needed to cleanly build Python 3.7.
If you need trackback information when throwing an exception use raise ExceptionClass(msg),
otherwise, use sys.exit(msg) instead.
The assert statement (which raises AssertionError if the assertion is not met)
is a very good way to ensure conditions to be met.
:::python
assert some_conditionThings on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Logging is critical for debugging applications. For production applications, it is best to send log information into a file instead of the standard output so that the log information is persisted …