Bundle Resource Files into a Rust Application
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The Rust macro include_str includes a UTF-8 encoded file as a string; the Rust macro include_bytes includes a file as a reference to a byte array. Both of those 2 macros load files into the Rust binary at compile time and consume memory at run time. It might not be a good idea to use them for large files!
Data Sources
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Data Platforms
https://github.com/quiltdata/quilt
https://registry.opendata.aws/
https://www.google.com/publicdata/directory
https://proxycrawl.com
Data for Computer Vison
Data for NLP
Data for Learning to …
Spark Issue: Too Many Containers Asked
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Error Message
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.
Possible Causes
"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …
Hands on the Resource Module in Python
Tips and Traps¶
This module provides basic mechanisms for measuring and controlling system resources utilized by a process and its subprocesses. It cannot be used to check resource usage of other processes.
Check Resources Used by Processes Using Python
Use the resource
Library¶
Please refer to Hands on the resource Module in Python for more details.
Use the tracemalloc
Library¶
https://docs.python.org/3/library/tracemalloc.html#module-tracemalloc
The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information:
- Traceback where an object was allocated
- Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks