Notes

Recap: The Backpropagation of the Error Method (BP) An Artificial Neural Network (ANN) is a computational model that is vaguely inspired by the biological network of neurons constituting the brain of vertebrates. It can be used as a trainable classifier of data points. Similarly to the biological analogue, it consists of a set of computing units, or neurons and of directed links connecting them. The strength and sign of a link is given through its weight. The neurons take in a set of inputs and produce an output based on a given input function and a given non-linearity, the activation function. If we bundle many neurons to a layer, and then connect multiple layers by linking the neuronal output of each layer to the neuronal input of the next layer, we get a deep neural network structure, where we differentiate between the input layer, the output layer and the in-between-laying hidden layers. Based on data on the input layer, the network will perform a forward-pass of the information and make a prediction. During training, the prediction is then compared with the ground-truth, or label, of the data. A loss is calculated based on a difference-norm between the prediction and the true label of the data point. Subsequently, weights of the network are updated. If back-propagation is used, which is the most common algorithm for supervised learning of ANNs, the derivative of the loss with respect to each weight is obtained, the information passed backwards and the weights adjusted accordingly.

The above described concept can be formulated mathematically. A network with L layers can be defined as:

$$\mathbf{h}^{j}\mathbf{=}\sigma^{j}(W^{j} \bullet \mathbf{h}^{j - 1}\mathbf{) =}\sigma^{j}(z^{j}),\ \ j = 1,\ ...,\ L,$$

Where $\mathbf{h}^{j}$ is the state of the j-th hidden layer and $z_{i}^{j} = \sum_{k = 0}^{N^{j}}w_{ik}^{j}h_{i}^{j - 1}$ the input for the i-th neuron in the j-th hidden layer $\mathbf{h}^{L}$ is the output layer and $\mathbf{h}^{0}\mathbf{= x}$ the input layer. The forward-mapping is defined by the non-linear activation function $\sigma^{j}$ and the weight vector $W^{j}$. Most often, the bias term of the j-th layer $w_{0}^{j}$ is included in the weight vector, thus $W^{j} = (w_{0}^{j},\ w_{1}^{j},\ ...,\ w_{N}^{j})$ defines the weight vector of a layer containing $N_{\ }^{j}$ neurons. Now, let's fix the network architecture, meaning the number of neurons, the wiring scheme and the activation $\sigma^{j}$, and define the network parameters as all the weights $w_{i}^{j}$. If we define all parameters between layer j and l $(0\ \leq j < l \leq L)$ as $\theta_{W}^{j,l} = W^{k},\ k = j + 1,\ ...,\ l$, we can write the l-th layer $\mathbf{h}^{l}$ as a function of the j-th layer $\mathbf{h}^{j}$, given the parameters $\theta_{W}^{j,l}$:

$$\mathbf{h}^{l} = \mathbf{h}^{l}\mathbf{(}\mathbf{h}^{j}\mathbf{;}\theta_{W}^{j,l})$$

For a given data set $\left( \mathbf{x},\mathbf{y} \right) = \left( \left( \mathbf{x}_{\mathbf{1}}\mathbf{,}\mathbf{y}_{\mathbf{1}} \right)\mathbf{,\ldots,}\left( \mathbf{x}_{\mathbf{D}}\mathbf{,}\mathbf{y}_{\mathbf{D}} \right) \right)$, a global loss function $\mathcal{L(}\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{),\ y)}$ gives a measure of the difference between the network output $\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{)}$ and the label y. During training, the goal is to minimize the expectation value of the global loss $\mathbb{E}_{p}\{\mathcal{L(}\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{),\ y)}\mathbf{\}}$ based on a data distribution $\mathbf{p}\left( \mathbf{x},\mathbf{y} \right)$. Common loss functions are the Mean-Squared-Error (MSE) for regression:

$$\mathcal{L}_{MSE}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)\mathbf{=}\frac{\mathbf{1}}{\mathbf{2}\mathbf{D}}\sum_{\mathbf{i = 1}}^{\mathbf{D}}\left( \mathbf{h}^{L}\left( \mathbf{x}_{\mathbf{i}}\mathbf{;}\theta_{W}^{0,L} \right)\mathbf{-}\mathbf{y}_{\mathbf{i}} \right)^{\mathbf{2}}$$

And the Cross-Entropy Loss (CE) for classification with C classes:

$$\mathcal{L}_{CE}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)\mathbf{= -}\sum_{\mathbf{c = 1}}^{\mathbf{C}}\mathbf{y}_{\mathbf{c}}\mathbf{\log}{\mathbf{h}^{L}}_{\mathbf{c}}\left( \mathbf{x}_{\mathbf{c}}\mathbf{;}\theta_{W}^{0,L} \right)$$

$$\mathbf{\ }$$

If the backpropagation algorithm is used, updating the weights is done by taking the derivatives of the loss with respect to all weights:

$$w_{j}^{l}\lbrack t + 1\rbrack = w_{j}^{l}\lbrack t\rbrack - \eta\frac{\partial\mathcal{L}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)}{\partial w_{j}^{l}}$$

The introduced learning-rate factor $\eta$ does in general not need to be constant, thus it can be adaptive over time.

Backpropagation Backpropagation (BP) is a widely used algorithm in training feedforward neural networks for supervised learning. It computes the gradient of the loss function with respect to the weights of the network for a single input/output example, and does so efficiently, unlike a naïve direct computation of the gradient with respect to each weight individually. This efficiency makes it feasible to use gradient methods for training multilayer networks, updating weights to minimize loss; gradient descent, or variants such as stochastic gradient descent, are commonly used. The backpropagation algorithm works by computing the gradient of the loss function with respect to each weight by the chain rule, computing the gradient one layer at a time, iterating backwards from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this is an example of dynamic programming.

We want to calculate the weight error, which is the gradient of the loss with respect to the input of the neuron j of the l-th layer $z_{j}^{l}$:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}}$$

First, we calculate the gradient with respect to the ultimate layer L:

$$\delta_{j}^{L} = \frac{\partial\mathcal{L}}{\partial z_{j}^{L}} = \frac{\partial\mathcal{L}}{\partial h_{j}^{L}}\frac{\partial h_{j}^{L}}{\partial z_{j}^{L}} = \frac{\partial\mathcal{L}}{\partial h_{j}^{L}}\sigma^{'}(z_{j}^{L})$$

Then, we calculate the gradient with respect to an intermediate layer l:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l + 1}}\frac{\partial z_{j}^{l + 1}}{\partial z_{j}^{l}} = \delta_{j}^{l + 1}\frac{\partial z_{j}^{l + 1}}{\partial z_{j}^{l}} = \sum_{k}^{}{\delta_{j}^{l + 1}w_{k}^{l + 1}}\sigma^{'}(z_{j}^{l})$$

We note, that the gradient can also be written with a dependency to the weight:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}} = \frac{\partial\mathcal{L}}{\partial w_{k}^{l}}\frac{\partial w_{k}^{l}}{\partial z_{j}^{l}}\ = \frac{\partial\mathcal{L}}{\partial w_{k}^{l}}\frac{1}{h_{k}^{l - 1}}$$

Thus, we arrive at the recursive form:

$$\frac{\partial\mathcal{L}}{\partial w_{k}^{l}} = h_{k}^{l - 1}\delta_{j}^{l}$$

Which gives us the weight update equation as follows:

$${w^{'}}_{k}^{l} = w_{k}^{l} - \eta\mathrm{\Delta}w_{k}^{l} = w_{k}^{l} - \eta\frac{\partial\mathcal{L}}{\partial w_{k}^{l}} = w_{k}^{l} - \eta h_{k}^{l - 1}\delta_{j}^{l}$$

Where $\eta$ is the learning rate.

Biological Plausibility Issues

• The backpropagation computation is purely linear, whereas biological neurons interleave linear and non-linear operations.
• If the feedback paths were used to propagate credit assignment by backpropagation, they would need precise knowledge of the derivates of the non-linearities at the operating point used in the corresponding feedforward computation.
• Similarly, these feedback paths would have to use exact symmetric weights (with the same connectivity, transposed) of the feedforward connections.
• Real neurons communicate by (possibly stochastic) binary values (spikes).
• The computation would have to be precisely clocked to alternate between feedforward and backpropagation phases.
• It is not clear where the output targets would come from.