This notebook present a minimum example on how to use Spark with the Almond Scala kernel in Jupyterlab. Notice that Spark 2.4.2 is used since it is the only stable Spark version that supports Scala 2.12 as of now. Please refer to almond-sh/examples for more examples.
In [6]:
scala.util.Properties.versionString
Out[6]:
res5: String = "version 2.12.10"
In [2]:
interp.load.ivy("org.apache.spark" % "spark-core_2.12" % "2.4.2")
interp.load.ivy("org.apache.spark" % "spark-sql_2.12" % "2.4.2")
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.pom Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.pom Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.pom Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.pom Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5.pom Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.pom Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.pom Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.pom Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.pom Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.pom Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.pom Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5.pom Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.pom Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet/1.10.1/parquet-1.10.1.pom Downloading https://repo1.maven.org/maven2/org/apache/orc/orc/1.5.5/orc-1.5.5.pom Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-java-root/0.10.0/arrow-java-root-0.10.0.pom Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-java-root/0.10.0/arrow-java-root-0.10.0.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet/1.10.1/parquet-1.10.1.pom Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc/1.5.5/orc-1.5.5.pom Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.pom Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.pom Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.pom Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.pom Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.pom Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.pom Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.pom Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.pom Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.pom Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.pom Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.pom Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.pom Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.pom Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.pom Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.pom Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.pom Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.pom Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.pom Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.pom Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.pom Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.pom Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.pom Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.pom Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.pom Downloading https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/scala-reflect-2.11.12.pom Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.pom Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.pom Downloaded https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/scala-reflect-2.11.12.pom Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.pom Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.pom Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc-parent/0.7.2/hppc-parent-0.7.2.pom Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino-parent/3.0.9/janino-parent-3.0.9.pom Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.7/antlr4-master-4.7.pom Downloading https://repo1.maven.org/maven2/io/airlift/airbase/78/airbase-78.pom Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.25/slf4j-parent-1.7.25.pom Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino-parent/3.0.9/janino-parent-3.0.9.pom Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.25/slf4j-parent-1.7.25.pom Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc-parent/0.7.2/hppc-parent-0.7.2.pom Downloaded https://repo1.maven.org/maven2/io/airlift/airbase/78/airbase-78.pom Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.7/antlr4-master-4.7.pom Downloading https://repo1.maven.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom Downloaded https://repo1.maven.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom Downloading https://repo1.maven.org/maven2/org/apache/apache/15/apache-15.pom Downloaded https://repo1.maven.org/maven2/org/apache/apache/15/apache-15.pom Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.jar Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.jar Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.jar Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.jar Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.jar Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.jar Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.jar Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.jar Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.jar Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.jar Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-nohive.jar Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-nohive.jar Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.jar Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.jar Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.jar Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-nohive.jar Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-nohive.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.jar Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.jar Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.jar Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2-sources.jar Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2-sources.jar Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25-sources.jar Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10-sources.jar Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3-sources.jar Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10-sources.jar Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9-sources.jar Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25-sources.jar Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7-sources.jar Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2-sources.jar Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9-sources.jar Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5-sources.jar Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9-sources.jar Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-sources.jar Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7-sources.jar Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10-sources.jar Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1-sources.jar Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0-sources.jar Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1-sources.jar Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1-sources.jar Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2-sources.jar Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055-sources.jar Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9-sources.jar Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0-sources.jar Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0-sources.jar Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0-sources.jar Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2-sources.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.jar Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.jar
In [3]:
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder()
.master("local[2]")
.appName("Spark Example")
.getOrCreate()
import spark.implicits._
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 20/03/23 17:30:21 INFO SparkContext: Running Spark version 2.4.2 20/03/23 17:30:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/03/23 17:30:22 INFO SparkContext: Submitted application: Spark Example 20/03/23 17:30:22 INFO SecurityManager: Changing view acls to: dclong 20/03/23 17:30:22 INFO SecurityManager: Changing modify acls to: dclong 20/03/23 17:30:22 INFO SecurityManager: Changing view acls groups to: 20/03/23 17:30:22 INFO SecurityManager: Changing modify acls groups to: 20/03/23 17:30:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dclong); groups with view permissions: Set(); users with modify permissions: Set(dclong); groups with modify permissions: Set() 20/03/23 17:30:23 INFO Utils: Successfully started service 'sparkDriver' on port 46185. 20/03/23 17:30:23 INFO SparkEnv: Registering MapOutputTracker 20/03/23 17:30:23 INFO SparkEnv: Registering BlockManagerMaster 20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/03/23 17:30:23 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8657440f-2a62-4cb0-8134-acf22f99a429 20/03/23 17:30:23 INFO MemoryStore: MemoryStore started with capacity 1315.2 MB 20/03/23 17:30:23 INFO SparkEnv: Registering OutputCommitCoordinator 20/03/23 17:30:23 INFO Utils: Successfully started service 'SparkUI' on port 4040. 20/03/23 17:30:23 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://jupyterhub-ds:4040 20/03/23 17:30:23 INFO Executor: Starting executor ID driver on host localhost 20/03/23 17:30:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33483. 20/03/23 17:30:23 INFO NettyBlockTransferService: Server created on jupyterhub-ds:33483 20/03/23 17:30:23 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/03/23 17:30:23 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, jupyterhub-ds, 33483, None) 20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: Registering block manager jupyterhub-ds:33483 with 1315.2 MB RAM, BlockManagerId(driver, jupyterhub-ds, 33483, None) 20/03/23 17:30:23 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, jupyterhub-ds, 33483, None) 20/03/23 17:30:23 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, jupyterhub-ds, 33483, None)
Out[3]:
import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ spark: SparkSession = org.apache.spark.sql.SparkSession@6721d392 import spark.implicits._
In [5]:
val df = Seq(
(0, "hello"),
(1, "world")
).toDF("id", "text")
df.show
20/03/23 17:31:19 INFO ContextCleaner: Cleaned accumulator 0 20/03/23 17:31:19 INFO CodeGenerator: Code generated in 33.2682 ms 20/03/23 17:31:20 INFO CodeGenerator: Code generated in 17.3675 ms 20/03/23 17:31:20 INFO CodeGenerator: Code generated in 15.8421 ms
+---+-----+ | id| text| +---+-----+ | 0|hello| | 1|world| +---+-----+
Out[5]:
df: org.apache.spark.sql.package.DataFrame = [id: int, text: string]