Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
List all Spark applications.
yarn application --list
Show status of a Spark application.
yarn application -status application_1459542433815_0002
view logs of a Spark application.
yarn logs -applicationId application_1459542433815_0002
kill a Spark application …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by ...)
The required version of GLIBC by the binary executor is not found on Spark nodes.
Recompile your …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Upper and lower bounds tests and Inter Quartile Range Checks(IQR) and standard deviations
Aggregate level checks (after manipulating data, there should still be the ability to explain how the data …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Please refer to Spark Issue: Task Not Serializable for a similar serialization issue in Spark/Scala.
For example, if you have the following import
from nltk.corpus import stopwords …Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
convert to delta [db_name.]table_name [partitioned by ...] [vacuum [retain number hours]]
vaccum
describe history db_name.table_name
can select from historical snapshot can also rollback to a historical snapshot rollback …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Hive is case-insensitive, both keywords and functions
You can use both double and single quotes for strings
use = rather than == for equality comparison
but it seems that == also works
use % rather …