Ben Chuanlong Du's Blog

It is never too late to learn.

Check Whether a File Exists in Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

org.apache.hadoop.fs.FileSystem

val conf = sc.hadoopConfiguration
val fs = org.apache.hadoop.fs.FileSystem.get(conf)
val exists = fs.exists(new org.apache.hadoop.fs.Path("/path/on/hdfs …

Parallel Computing in Shell

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. & and wait

  2. parallel

  3. xargs

parallel is a cool bash command which makes parallel computing easy in Bash. It is a parallel version replacement of xargs.

Add Method to a Python Class

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://www.ianlewis.org/en/dynamically-adding-method-classes-or-class-instanc

http://block.arch.ethz.ch/blog/2016/07/adding-methods-to-python-classes/

Java Reflection

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://www.oracle.com/technetwork/articles/java/javareflection-1536171.html

https://stackoverflow.com/questions/37628/what-is-reflection-and-why-is-it-useful