Neurons (Activation Functions)¶
Neurons can be attached to any layer. The neuron of each layer will affect the output in the forward pass and the gradient in the backward pass automatically unless it is an identity neuron. Layers have an identity neuron by default [1].

class
Neurons.
Identity
¶ An activation function that does not change its input.

class
Neurons.
ReLU
¶ Rectified Linear Unit. During the forward pass, it inhibits all inhibitions below some threshold \(\epsilon\), typically 0. In other words, it computes pointwise \(y=\max(\epsilon, x)\). The pointwise derivative for ReLU is
\[\begin{split}\frac{dy}{dx} = \begin{cases}1 & x > \epsilon \\ 0 & x \leq \epsilon\end{cases}\end{split}\]
epsilon
¶ Specifies the minimum threshold at which the neuron will truncate. Default
0
.
Note
ReLU is actually not differentiable at \(\epsilon\). But it has subdifferential \([0,1]\). Any value in that interval can be taken as a subderivative, and can be used in SGD if we generalize from gradient descent to subgradient descent. In the implementation, we choose the subgradient at \(x==0\) to be 0.


class
Neurons.
LReLU
¶ Leaky Rectified Linear Unit. A Leaky ReLU can help fix the “dying ReLU” problem. ReLU’s can “die” if a large enough gradient changes the weights such that the neuron never activates on new data.
\[\begin{split}\frac{dy}{dx} = \begin{cases}1 & x > 0 \\ 0.01 & x \leq 0\end{cases}\end{split}\]

class
Neurons.
Sigmoid
¶ Sigmoid is a smoothed step function that produces approximate 0 for negative input with large absolute values and approximate 1 for large positive inputs. The pointwise formula is \(y = 1/(1+e^{x})\). The pointwise derivative is
\[\frac{dy}{dx} = \frac{e^{x}}{\left(1+e^{x}\right)^2} = (1y)y\]

class
Neurons.
Tanh
¶ Tanh is a transformed version of Sigmoid, that takes values in \(\pm 1\) instead of the unit interval. input with large absolute values and approximate 1 for large positive inputs. The pointwise formula is \(y = (1e^{2x})/(1+e^{2x})\). The pointwise derivative is
\[\frac{dy}{dx} = 4e^{2x}/(e^{2x} + 1)^2 = (1y^2)\]

class
Neurons.
Exponential
¶ The exponential function.
\[y = exp(x)\]
[1]  This is actually not true: not all layers in Mocha support neurons. For example, data layers currently does not have neurons, but this feature could be added by simply adding a neuron property to the data layer type. However, for some layer types like loss layers or accuracy layers, it does not make much sense to have neurons. 