NeuralArithmetic.NAC
— TypeNAC(in::Int, out::Int; initW=glorot_uniform, initM=glorot_uniform)
Neural Accumulator. Special case of affine layer in which the parameters are encouraged to be close to {-1, 0, 1}.
Paper: https://arxiv.org/abs/1808.00508
NeuralArithmetic.NALU
— TypeNALU(in::Int, out::Int; initNAC=glorot_uniform, initG=glorot_uniform, initb=glorot_uniform)
Neural Arithmetic Logic Unit. Layer that is capable of learing multiplication, division, power functions, addition, and subtraction.
Paper: https://arxiv.org/abs/1808.00508
NeuralArithmetic.NALUX
— TypeNALUX(in::Int, out::Int, G; initA=glorot_uniform, initM=glorot_uniform)
Extends the NALU to work with negative and small numbers by using a complex multiplication matrix.
NeuralArithmetic.NAU
— TypeNAU(in::Int, out::Int; init=glorot_uniform)
Neural addition unit.
Lacks the regularization suggested in https://openreview.net/pdf?id=H1gNOeHKPS as it is intended to be used with ARD (automatic relevance determination)
NeuralArithmetic.NMU
— TypeNMU(in::Int, out::Int; init=rand)
Neural multiplication unit. Can represent multiplications between inputs. Weights are clipped to [0,1].
Lacks the regularization suggested in https://openreview.net/pdf?id=H1gNOeHKPS as it is intended to be used with ARD (automatic relevance determination)