Ben Chuanlong Du's Blog

It is never too late to learn.

Get the Dependencies of a Python Package Without Installing It

After installing a Python package, you can use the command pip3 show pkg to check detailed information about it. What if you want to get information about a Python package without installing? The RESTAPI https://pypi.org/pypi/pkg/json returns detailed information of the Python package pkg.

Note: The recursive version of dependencies implemented bellow does not take into consideration of versions, so it might not return the correct dependencies of a Python packages.

Access Control in Spark SQL

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Grant Permission to Users

GRANT
    priv_type [, priv_type ] ...
    ON database_table_or_view_name
    TO principal_specification [, principal_specification] ...
    [WITH GRANT OPTION];

Examples:

GRANT SELECT ON table1 TO USER user1;
GRANT SELECT ON DATABASE db1 TO USER user1 …

Tips on the Linux Command ln

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

It is suggest that you avoid using the trailing slash when you use ln to create symbolic link for a directory. This is different from the command rsync which a trailing …

Debugging Tools for Java

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

arthas

arthas

JDK commands

Thread dump: jstack jvm_pid > (use -F if not responding)

Memory dump: jmap -dump:live,format=b,file=.hprof jvm_pid

Heap status: jmap -heap jvm_pid

GC status: jstat …

Check the CPU Limit of a Docker Container

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Get the number in the file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us. Denote it as cfs_quota_us.

  2. Get the number in the file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us. Denote …

Check the Memory Limit of a Docker Container

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Docker 1.13+

Docker 1.13+ mounts the container's cgroup to /sys/fs/cgroup (this could change in future versions). You can check the limit using

cat /sys/fs/cgroup/memory …