Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Caused by: org.apache.spark.SparkException: Could not execute broadcast in 600 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting …
Bucketed column is only supported in Hive table at this time.
A Hive table can have both partition and bucket columns.
Suppose t1 and t2 are 2 bucketed tables and with the number of buckets b1 and b2 respecitvely.
For bucket optimization to kick in when joining them:
- The 2 tables must be bucketed on the same keys/columns.
- Must joining on the bucket keys/columns.
- `b1` is a multiple of `b2` or `b2` is a multiple of `b1`.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
This post collects some typical pure Python errors in PySpark applications.
object has no attribute
Fix the attribute name.
No such file or directory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Since a Spark DataFrame is immutable, you cannot update or delete records from a physical table (e.g., a Hive table) directly using Spark DataFrame/SQL API. However …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
TypeError: withReplacement (optional), fraction (required) and seed (optional) should be a bool, float and number; however, got [
].
An integer number (e.g., 1) is passed to the fraction parameter …