A collection of activation function approximations for Flux.
In some models the accuracy of the
tanh functions can be reduced without a loss of accuracy in the training process. Switching to an approximation can reduce training times significantly for some models.
sigmoid we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation which uses a fast
exp imeplemention based on an approximation of the formula:
exp(x) = lim
(1 + x/n)
There is also an implementation of
TheanoFastSigmoid which is currently accepted in the Theano project. It is here mostly for comparison because it is both slower and less accurate than other
tanh we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation based on the continuous fraction approximation of tanh.
Additionally we also provide the