Ben Chuanlong Du's Blog

It is never too late to learn.

Rust and Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

The simplest and best way is to leverage pandas_udf in PySpark. In the pandas UDF, you can call subprocess.run to run any shell command and capture its output.

from pathlib …

JavaScript Alternatives to Shell

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://github.com/shelljs/shelljs

https://tutorialedge.net/javascript/nodejs/executing-shell-scripts-with-nodejs/