Ben Chuanlong Du's Blog

It is never too late to learn.

Tips on Selenium

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Tips and Traps

  1. Selenium IDE is very useful. You can use it to record (test) actions and then export it into (testing) code in different programming languages (e.g., Python).

    selenium-ide-menu

    selenium-ide-menu-export

Examples …

Spark Issue: a Master URL Must Be Set in Your Configuration

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

Error initializzing SparkContext: A master URL must be set in your configuration.

Possible Causes

The master of Spark cluster is not specified.

Solutions

Add .master("yarn") into the following …

Spark Issue: Spark Application Submission Is Not Finished

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

Application submission is not finished, submitted application application__1524215324275_0081 is still …

Spark Issue: Cannot Create a Path from An Empty String

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Issue

java.lang.IllegalArgumentException: Can not create a Path from an empty string

Possible Causes

The error you are seeing could be from number of things:

  1. parameters , check for ${param} in …

Spark Issue: Duplicated Partitions

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

There seems to be an issue in Spark that it might fail to overwrite files even if mode of spark.write is set to be "overwrite".

Spark Issue: Too Many Containers Asked

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Error Message

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Too many containers asked, 16731530.

image

Possible Causes

"Too many containers asked" is a protection mechanism of the Resource Manager. It might be triggered …