The select Function in Spark DataFrame
Construct Simple Spark DataFrames Using Seq
Seq.toDF¶
toDF() provides a concise syntax for creating DataFrames and can be accessed after importing Spark implicits.
import spark.implicits._SparkSession.createDataFrame¶
Read/Write CSV in Spark
Tips on Maven
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
$HOME/.m2
maven: shade plugin, scala compile plugin, exclude manifest files, ...
import as maven project ...
find an updated version of scala archetype for maven
mvn package
mvn clean …Tips on sbt Assembly
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Add sbt-assembly as a dependency in project/assembly.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")