Ben Chuanlong Du's Blog

It is never too late to learn.

Bundle Resource Files into a Rust Application

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. The Rust macro include_str includes a UTF-8 encoded file as a string; the Rust macro include_bytes includes a file as a reference to a byte array. Both of those 2 macros load files into the Rust binary at compile time and consume memory at run time. It might not be a good idea to use them for large files!

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …

Hands on the Resource Module in Python

Tips and Traps

This module provides basic mechanisms for measuring and controlling system resources utilized by a process and its subprocesses. It cannot be used to check resource usage of other processes.

Check Resources Used by Processes Using Python

Use the tracemalloc Library

https://docs.python.org/3/library/tracemalloc.html#module-tracemalloc

The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information:

  • Traceback where an object was allocated
  • Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks