Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
ydata-profiling (successor to pandas-profiling) is tool for profiling pandas and Spark DataFrames. One possible way to work with large data is to do simple profiling on the large DataFrame and …
It is suggested that you use multiprocessing
(e.g., pool_size=8)
to speed up data profiling.
Note: It seems to me that currently multiprocessing only works
when minimal=True.
minimal=True helps reuce consumed memory.
profile = ProfileReport(
df, title="Data Profiling Report",
explorative=True, minimal=True, pool_size=8
)