Ben Chuanlong Du's Blog

It is never too late to learn.

Job Scheduling and Management Using Apache Airflow

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Installation (MySQL)

  1. Install Apache AirFlow.

    wajig install \
        python3-dev python3-pip \
        mysql-server libmysqlclient-dev
    sudo AIRFLOW_GPL_UNIDECODE=yes pip3 install apache-airflow[mysql]
    
  2. Add the following content into your my.cnf …

Spark Issue: Max Number of Executor Failures Reached

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

21/06/01 15:03:28 INFO ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (6) reached)

Possible Causes

The option spark.yarn.max.executor …

Write Documentation for Python Packages Using Sphinx

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Installation

You can install Sphinx and necessary extensions using the following command.

pip3 install sphinx sphinx-autodoc-typehints

Or simply

xinstall sphinx -ic

Since the above commands installs Sphinx to the user's local …

Messsage Queue Implementations

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Kafka seems to be the most popular message queue currently. Redis and memcached are more suitaable for situations where you need cache in memory.

Protocols

AMQP is the Advanced Message Queuing …

Show Image in Terminal

** Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

viu is a simple terminal image viewer written in Rust. It works perfectly in kitty . In terminals which does not support inline image display protocols, viu will show the image in …

Tips on yarn

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://yarnpkg.com/

https://yarnpkg.com/getting-started/install

yarn tauri:serve

References

https://github.com/yarnpkg/berry