Comments¶

Even though Spark DataFrame/SQL APIs do not distinguish cases of column names, the columns saved into HDFS are case-sensitive!

DataFrameReader APIs

DataFrameWriter APIs

https://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources

Comments¶

Writing a customized Spark SQL Parser

Writing custom optimization in Apache Spark SQL - parser

SparkSqlParser.scala

What part of Spark SQL parse SQL statements and create execution plan?

Static Analyzer¶

If we get the execuation plan, then it is quite easy to analyze ...

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-lineage.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-dependencies.html

http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html

Spark Testing Frameworks/Tools¶

You can use Scala testing frameworks ScalaTest (recommended) and Specs, or you can use frameworks/tools developed based on them for Spark specifically. Various discussions suggests that Spark Testing Base is a good one.

https://www.slideshare.net/SparkSummit/beyond-parallelize-and-collect-by-holden-karau

Spark Unit Testing¶