AQE (Adaptive Query Execution)¶

To enable AQE, you have to set spark.sql.adaptive.enabled to true (using --conf spark.sql.adaptive.enabled=true in spark-submit or using `spark.config("spark.sql.adaptive,enabled", "true") in Spark/PySpark code.)

Pandas UDFs¶

Pandas UDFs are user defined functions that are executed by Spark using Arrow to transfer data to Pandas to work with the data, which allows vectorized operations. A Pandas UDF is defined using pandas_udf

framequery¶

export¶

A new child process forked from a parent process does not inherit parent's variables by default. The export command marks an environment variable to be exported with any newly forked child processes and thus it allows a child process to inherit all marked variables.

unset¶

References¶

https://stackoverflow.com/questions/6877727/how-do-i-delete-an-exported-environment-variable

Parse Arguments in Bash¶

Environment Variables¶

export

unset

Tips and Traps¶

1. explainshell.com is a great place for learning shell.

2. Bash-it/bash-it is a great community driven Bash framework.

3. It is suggested that you avoid writing complicated Bash scripts. IPython is a much better alternative.

4. Do NOT use ; to delimit paths passed to a shell command because ;

Comments¶

1. As long as the class name is not need at definition time of the class, it is OK to use it.

You cannot use a class in default values of the __init__ function of the class.