Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

There seems to be an issue in Spark that it might fail to overwrite files even if mode of spark.write is set to be "overwrite".

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue

Total size of serialized results is bigger than spark.driver.maxResultSize

Solutions

1. Eliminate unnecessary broadcast or collect.

2. If one of the tables for joining contains too large number of partitions …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue Unable to find encoder for type stored in a Dataset

Solution …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Grant Permission to Users

GRANT
    priv_type [, priv_type ] ...
    ON database_table_or_view_name
    TO principal_specification [, principal_specification] ...
    [WITH GRANT OPTION];

Examples:

GRANT SELECT ON table1 TO USER user1;
GRANT SELECT ON DATABASE db1 TO USER user1 …