Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

The limit clause (or the method DataFrame.limit if you are using Spark) is a better alternative if randomness is not critical.

PostgreSQL

SELECT id from table TABLESAMPLE BERNOULLI(10) WHERE …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Plan Your Work

1. Have a clear idea about what you want to do is very important, especially when you are working on an explorative project. It often saves you time to …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

java.lang.IllegalArgumentException: Wrong FS: hdfs://..., expected: viewfs://...

Possible Causes

The Spark cluster has migrated to Router-based Federation (RBF) namenodes, and viewfs:// (instead of hdfs://) is required to access HDFS …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://cluster-name-ns02/

Possible Causes

As the error message says, viewfs://cluster-name-ns02 is not configured.

1. It is possible that …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64

Possible Causes

A pandas_udf tag specifies a return type of String but the corresponding pandas udf returns a different …

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptoms

RuntimeError: Arrow legacy IPC format is not supported in PySpark, please unset ARROW_PRE_0_15_IPC_FORMAT

Possible Causes

You are using PySpark 3.0+ with one (or both) of the following options.

--conf …