Activation Functions in Neural Network

Dec 17, 2019

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

GELU

GELU is the best activation function currently (at least in NLP).

$$ GELU(x) == x \Phi(x) $$

where $\Phi(x)$ is the cumulative distribution function of the standard normal distribution.

Dec 03, 2012

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Java 7 allows use of strings in switch instead of just integers, which make things much more convenient (see the following example).

public void foo(Foo t) {
    String …