Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The ORM library SQLAlchemy
can be leverage
Koalas is pandas API on PySpark
References¶
https://github.com/databricks/koalas
https://databricks.com/blog/2020/08/11/interoperability-between-koalas-and-apache-spark.html
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Column
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#module-pyspark.sql.functions
Fault Tolerance and Resilience Tools
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
https://github.com/jhalterman/failsafe
Run Commands on Remote Machines
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
On a Sinsgle Machine
SSH
-
The pipeline command is run locally. If you want the pipeline command to run remotely, place the whole command to be run remotely in double/single …
Tips on Optuna
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Optuna is a good framework for hyper parameter tuning in machine learning.
Tips on Apache Arrow
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
[Feather vs Parquet]https://github.com/wesm/feather/issues/188
References
https://github.com/wesm/feather