FastActivations.jl

A collection of activation function approximations for Flux.

In some models the accuracy of the sigmoid and tanh functions can be reduced without a loss of accuracy in the training process. Switching to an approximation can reduce training times significantly for some models.

Sigmoid Approximations

For sigmoid we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation which uses a fast exp imeplemention based on an approximation of the formula: exp(x) = limn->inf(1 + x/n)

Fitted Functions	Fast Expr

Theano Sigmoid

There is also an implementation of TheanoFastSigmoid which is currently accepted in the Theano project. It is here mostly for comparison because it is both slower and less accurate than other Sigmoid approximations.

Tanh Approximations

For tanh we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation based on the continuous fraction approximation of tanh.

Additionally we also provide the serpentine function.

Fitted Functions	Continuous Fraction	Serpentine