Ben Chuanlong Du's Blog

It is never too late to learn.

Job Scheduling and Management Using Apache Airflow

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Installation (MySQL)

  1. Install Apache AirFlow.

    wajig install \
        python3-dev python3-pip \
        mysql-server libmysqlclient-dev
    sudo AIRFLOW_GPL_UNIDECODE=yes pip3 install apache-airflow[mysql]
    
  2. Add the following content into your my.cnf …

Spark Issue: Max Number of Executor Failures Reached

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

21/06/01 15:03:28 INFO ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (6) reached)

Possible Causes

The option spark.yarn.max.executor …

Messsage Queue Implementations

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Kafka seems to be the most popular message queue currently. Redis and memcached are more suitaable for situations where you need cache in memory.

Protocols

AMQP is the Advanced Message Queuing …

Show Image in Terminal

** Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

viu is a simple terminal image viewer written in Rust. It works perfectly in kitty . In terminals which does not support inline image display protocols, viu will show the image in …

Tips on yarn

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://yarnpkg.com/

https://yarnpkg.com/getting-started/install

yarn tauri:serve

References

https://github.com/yarnpkg/berry

Tips on Search Engines

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Search Engine Applications

Apache Solr

Apache Solr is an enterprise search platform written in Java and using Apache Lucene . Major features include full-text search, index replication and sharding, and result faceting …