Comments¶

Writing a customized Spark SQL Parser

Writing custom optimization in Apache Spark SQL - parser

SparkSqlParser.scala

What part of Spark SQL parse SQL statements and create execution plan?

Static Analyzer¶

If we get the execuation plan, then it is quite easy to analyze ...

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-lineage.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-dependencies.html

http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html

Spark Testing Frameworks/Tools¶

You can use Scala testing frameworks ScalaTest (recommended) and Specs, or you can use frameworks/tools developed based on them for Spark specifically. Various discussions suggests that Spark Testing Base is a good one.

https://www.slideshare.net/SparkSummit/beyond-parallelize-and-collect-by-holden-karau

Spark Unit Testing¶

https://docs.dask.org/en/latest/

Installation¶

1. You have to install the complete version of Dask (using the command pip3 install dask[complete]) if you need support of extended memory (for handling big data) and schedulers (for performance). The default installation version (pip3 install dask) of Dask does not include those features out-of-box.

Comments¶

1. Forget to return is a common and tricky mistake in Kotlin!

Unpack Values¶

Function Overload and Named Arguments¶

1. Function overload might cause tricky invoking bugs if you change the signature of an overloaded function. You should always be careful when you change the signature of an overloaded function. There is a graceful way of resolving this kind of issues in Kotlin (and similarly in Scala and Python), which is to use named arguments. If you invoke an function with named arguments, it is relatively robust to change of order of arguments and it fails fast if you change the names of arguments.

Java Operator Precedence

1. The Bitwise operators has relative low precedence. They have lower precedence than arithmatical operators. It is suggested that you use parentheses when you mix lower precendenc (bitwise opertors, ternary opertor, etc.) and high precendenc operators together.

2. There is no unsigned left shift operator in Java. https://www.quora.com/Why-is-there-no-unsigned-left-shift-operator-in-Java