Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
It is suggested that you never use the -skipTrash option
unless you are absolutely aware of what you are doing.
I made mistakes a couple of times in …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Since a Spark DataFrame is immutable, you cannot update or delete records from a physical table (e.g., a Hive table) directly using Spark DataFrame/SQL API. However …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to Spark SQL for tips specific to Spark SQL.
It is almost always a good idea to filter out null value in the joinining columns before joining …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to http://www.legendu.net/misc/tag/spark-issue.html for a comprehensive list of Spark Issues and (possible) causes and solutions.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The simplest and best way is to leverage pandas_udf in PySpark.
In the pandas UDF,
you can call subprocess.run to run any shell command
and capture its output.
from pathlib …