Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The Docker image dclong/jupyterhub-toree has Spark and Apache Toree installed and configured. Since Spark is already installed in it, you don't need to download and install Spark by yourself. By …
This notebook present a minimum example on how to use Spark with the Almond Scala kernel in Jupyterlab. Notice that Spark 2.4.2 is used since it is the only stable Spark version that supports Scala 2.12 as of now. Please refer to almond-sh/examples for more examples.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
https://github.com/Azure/mmlspark/blob/master/docs/lightgbm.md
MMLSpark seems to be the best option to use train models using LightGBM on a Spark cluster. Note that MMLSpark requires …
fold takes an initial value,
and the first invocation of the lambda you pass to it
will receive that initial value
and the first element of the collection as parameters.
reduce doesn't take an initial value,
but instead starts with the first element of the collection as the accumulator (called sum in the following example).
You can aggregate all values in Columns of a DataFrame.
Just use aggregation functions in select without groupBy,
which is very similar to SQL syntax.
The aggregation functions all and any are available since Spark 3.0.
However,
they can be achieved using other aggregation functions such as sum
HashTable is a trait without a factory method and thus cannot be instantiated. HashMap and HashSet are subclasses that you should use.
https://stackoverflow.com/questions/3648870/scala-using-hashmap-with-a-default-value