Ben Chuanlong Du's Blog

It is never too late to learn.

Tips on Nox

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

It is suggested that you leverage profession CICD tools instead of nox for testing.

https://github.com/theacodes/nox

https://nox.thea.codes/en/stable/index.html

https://cjolowicz.github.io …

Unit Testing for Spark

Spark Testing Frameworks/Tools

You can use Scala testing frameworks ScalaTest (recommended) and Specs, or you can use frameworks/tools developed based on them for Spark specifically. Various discussions suggests that Spark Testing Base is a good one.

https://www.slideshare.net/SparkSummit/beyond-parallelize-and-collect-by-holden-karau

Spark Unit Testing