DataFrameReader APIs

DataFrameWriter APIs

https://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources

Comments¶

Writing a customized Spark SQL Parser

Writing custom optimization in Apache Spark SQL - parser

SparkSqlParser.scala

What part of Spark SQL parse SQL statements and create execution plan?

Static Analyzer¶

If we get the execuation plan, then it is quite easy to analyze ...

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-lineage.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-dependencies.html

http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html

Spark Testing Frameworks/Tools¶

You can use Scala testing frameworks ScalaTest (recommended) and Specs, or you can use frameworks/tools developed based on them for Spark specifically. Various discussions suggests that Spark Testing Base is a good one.

https://www.slideshare.net/SparkSummit/beyond-parallelize-and-collect-by-holden-karau

Spark Unit Testing¶

https://docs.dask.org/en/latest/

Installation¶

1. You have to install the complete version of Dask (using the command pip3 install dask[complete]) if you need support of extended memory (for handling big data) and schedulers (for performance). The default installation version (pip3 install dask) of Dask does not include those features out-of-box.

Comments¶

1. Forget to return is a common and tricky mistake in Kotlin!

Unpack Values¶