Comments¶

Writing a customized Spark SQL Parser

Writing custom optimization in Apache Spark SQL - parser

SparkSqlParser.scala

What part of Spark SQL parse SQL statements and create execution plan?

Static Analyzer¶

If we get the execuation plan, then it is quite easy to analyze ...

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-lineage.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-dependencies.html

http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html

Spark Testing Frameworks/Tools¶

You can use Scala testing frameworks ScalaTest (recommended) and Specs, or you can use frameworks/tools developed based on them for Spark specifically. Various discussions suggests that Spark Testing Base is a good one.

https://www.slideshare.net/SparkSummit/beyond-parallelize-and-collect-by-holden-karau

Spark Unit Testing¶

Construct an element with initial values.

https://www.scala-lang.org/api/2.13.0/scala/jdk/CollectionConverters$.html

https://stackoverflow.com/questions/8301947/what-is-the-difference-between-javaconverters-and-javaconversions-in-scala

Short Story¶

Just import scala.collection.JavaConverters._.