Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Block rdd_123_456 could not be removed as it was not found on disk or in memory.
The execution plan of a DataFrame is too complicated.
Not enough memory to …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
It is suggested that you never use the -skipTrash option
unless you are absolutely aware of what you are doing.
I made mistakes a couple of times in …
Bucketed column is only supported in Hive table at this time.
A Hive table can have both partition and bucket columns.
Suppose t1 and t2 are 2 bucketed tables and with the number of buckets b1 and b2 respecitvely.
For bucket optimization to kick in when joining them:
- The 2 tables must be bucketed on the same keys/columns.
- Must joining on the bucket keys/columns.
- `b1` is a multiple of `b2` or `b2` is a multiple of `b1`.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Since a Spark DataFrame is immutable, you cannot update or delete records from a physical table (e.g., a Hive table) directly using Spark DataFrame/SQL API. However …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to Spark SQL for tips specific to Spark SQL.
It is almost always a good idea to filter out null value in the joinining columns before joining …