Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Computer Vision Libraries in Rust
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
resize
resize Image resampling library in pure Rust. - Fast, with support for many pixel formats - No encoders/decoders, meant to be used with some external library - Tuned for resizing to the …
Configure Log4J for Spark
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Show Error Messages Only¶
When you run Spark or PySpark in a Jupyter/Lab notebook, it is recommended that you show ERROR messages only. Otherwise, there might be too much logging information polluting your notebook. You can set the log level of Spark to ERROR using the following line of code.
Process Big Data Using Spark
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
General Tips
-
Please refer to Spark SQL for tips specific to Spark SQL.
-
It is almost always a good idea to filter out null value in the joinining columns before joining …
Spark Issue: InvalidResourceRequestException
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Symptoms
Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=16 …