Ben Chuanlong Du's Blog

It is never too late to learn.

Job Scheduling and Management Using Apache Airflow

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Installation (MySQL)

  1. Install Apache AirFlow.

    wajig install \
        python3-dev python3-pip \
        mysql-server libmysqlclient-dev
    sudo AIRFLOW_GPL_UNIDECODE=yes pip3 install apache-airflow[mysql]
    
  2. Add the following content into your my.cnf …

Spark Issue: Max Number of Executor Failures Reached

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

21/06/01 15:03:28 INFO ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (6) reached)

Possible Causes

The option spark.yarn.max.executor …

The Best Way to Find Files and Manipulate Them

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

There are many cool (command-line) tools which can help you quickly find/locate files. Notice that they can all be combined with fzf to make it interactive.

  1. fdfind
  2. find
  3. locate
  4. osquery
  5. fselect
  6. ripgrep

Those tools can be combined with the pipe operator |

Write Documentation for Python Packages Using Sphinx

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Installation

You can install Sphinx and necessary extensions using the following command.

pip3 install sphinx sphinx-autodoc-typehints

Or simply

xinstall sphinx -ic

Since the above commands installs Sphinx to the user's local …