Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Improve Network Speed
软路由让你的翻墙速度快十倍！LEDE软路由v2ray对比梅林固件(v2ray/LEDE/谷歌云)
https://zhuanlan.zhihu.com/p/129139369
https://www.zhihu …
Configure Log4J for Spark
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Show Error Messages Only¶
When you run Spark or PySpark in a Jupyter/Lab notebook, it is recommended that you show ERROR messages only. Otherwise, there might be too much logging information polluting your notebook. You can set the log level of Spark to ERROR using the following line of code.
Process Big Data Using Spark
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
General Tips
-
Please refer to Spark SQL for tips specific to Spark SQL.
-
It is almost always a good idea to filter out null value in the joinining columns before joining …
Docker Images for Remote Desktop
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Tips and Traps
Make Date Show Time of a Specific Timezone in Linux
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
For example, to show the current Pacific time,
TZ=America/Los_Angeles date
Spark Issue: InvalidResourceRequestException
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Symptoms
Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=16 …