Ben Chuanlong Du's Blog

It is never too late to learn.

Spark vs Redshift

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement! Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://www.quora.com/Spark-vs-Redshift-Should-I-be-using-both-for-big-data-Which-is-better

Performance

https://dbseer.com/benchmark-comparison-spark-sql-redshift-cluster/

Redshift vs …

Docker Images for Zeppelin

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Official Zeppelin Docker Image

  1. Pull the official Zeppelin Docker image.

    docker pull apache/zeppelin
    
  2. Launch the image in a container.

    docker run -d -p 8080:8080 \
        -v $PWD/logs:/logs \
        -v …

Tips on Spark MLlib

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

  1. Spark MLlib RDD-based API supports stratified sampling but the DataFrame-based API hasn't implemented it yet as of Spark 2.4.3.

sample keys (not rows) with equal probability

References

https://spark …