Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Different Types of Memory in Rust¶

Rust has 3 different types of memory: static memory, stack memory and heap memory.

1. Static variables live in static memory and is determined at compile time. It is suggested that you define large data variables as static so that they live in the static memory instead of stack memory to avoid stack overflow problems. Of course, another way is to put those variables into heap memory (but at the cost of slight performance loss).

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

The HDFS Way¶

You can use the hdfs dfs -du /path/to/table command or hdfs dfs -count -q -v -h /path/to/table to get the size of an HDFS path (or table). However, this only works if the cluster supports HDFS. If a Spark cluster exposes only JDBC/ODBC APIs, this method does not work.