Ben Chuanlong Du's Blog

It is never too late to learn.

Check Whether a File Exists in Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

org.apache.hadoop.fs.FileSystem

val conf = sc.hadoopConfiguration
val fs = org.apache.hadoop.fs.FileSystem.get(conf)
val exists = fs.exists(new org.apache.hadoop.fs.Path("/path/on/hdfs …

Change File Permission on Linux

R: 4 W: 2 X: 1

Make a directory readable to other people.

chmod 755 dir  

Make a file readable to other people.

chmod 644 file 

Make a directory and all its subcontents readable to other people.

# make the dir and its subcontents readable and executable
chmod 755 -R dir …