Ben Chuanlong Du's Blog

It is never too late to learn.

Tips on Git

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Git Tools

Access Token for Git

There are a few advantages of using (GitHub, etc.) access tokens for Git, especially in an enterprise environemnt.

  1. Access tokens can provided more fined access …

Tips on Distributed Development

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Distributed Log Tracing

jaeger

quickwit

Cloud Tools

terraform

Terraform Tenables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative …

Rust and Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

The simplest and best way is to leverage pandas_udf in PySpark. In the pandas UDF, you can call subprocess.run to run any shell command and capture its output.

from pathlib …

Distributed Programming in Python

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://github.com/rq/rq

https://python-rq.org/