Ben Chuanlong Du's Blog

It is never too late to learn.

Spark SQL

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Spark SQL Guide

  1. Since a Spark DataFrame is immutable, you cannot update or delete records from a physical table (e.g., a Hive table) directly using Spark DataFrame/SQL API. However …

Hive SQL

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Hive is case-insensitive, both keywords and functions

  2. You can use both double and single quotes for strings

  3. use = rather than == for equality comparison but it seems that == also works

  4. use % rather …

Spark Issue: InvalidInputException for Some Hive Data Partitions

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

15/12/29 17:22:27 ERROR yarn.ApplicationMaster: User class …