Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Using os.sysconf¶
Notice that this ways only works on Linux but not on macOS or Windows.
Get physical memory in bytes.
Spark Issue: Block Could Not Be Removed as It Was Not Found on Disk or in Memory
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Symptom
Block rdd_123_456 could not be removed as it was not found on disk or in memory.
Cause
-
The execution plan of a DataFrame is too complicated.
-
Not enough memory to …
Memory in JVM
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Stack, heap and off-heap are all memory that stored in a computer's RAM.
Stack
Stack is used for static memory allocation. Variables allocated on the stack are stored directly to the …
Spark Issue: IllegalArgumentException: System Memory Must Be At Least
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Symptom
Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration …
Understand Memory Usage in PyTorch
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!