Notes

Why is this Topic important? Papers: "Cognitiva 85" & "Learning Representations by Back-Propagating Errors".

• BP might not be optimal, in particular to train generative models.
• BP is not fully local and computation is not easy to parallelize.
• BP needs to store all network activations for the BP steps.
• Understanding DNN training and optimization methods might provide a different angle to understand learning in biological networks in the Brain.
• Given that the functionality of DNNs is reflected by biological networks, the different variants of DNN training even provide testable hypothesis that neuroscience research can test.

Content of the Lecture:

• Recap: The Backpropagation of the Error Method (BP).
• Feedback Alignment (FA).
• Variants of FA (Direct, Indirect, Bi-Directional).
• Target Propagation and its Variants.
• Local (layer wise) Training for Deep Neuronal Networks.
• The Deep Feedback Control Method.

The Backpropagation of the Error Method (BP)

Recap: The Backpropagation of the Error Method (BP) An Artificial Neural Network (ANN) is a computational model that is vaguely inspired by the biological network of neurons constituting the brain of vertebrates. It can be used as a trainable classifier of data points. Similarly to the biological analogue, it consists of a set of computing units, or neurons and of directed links connecting them. The strength and sign of a link is given through its weight. The neurons take in a set of inputs and produce an output based on a given input function and a given non-linearity, the activation function. If we bundle many neurons to a layer, and then connect multiple layers by linking the neuronal output of each layer to the neuronal input of the next layer, we get a deep neural network structure, where we differentiate between the input layer, the output layer and the in-between-laying hidden layers. Based on data on the input layer, the network will perform a forward-pass of the information and make a prediction. During training, the prediction is then compared with the ground-truth, or label, of the data. A loss is calculated based on a difference-norm between the prediction and the true label of the data point. Subsequently, weights of the network are updated. If back-propagation is used, which is the most common algorithm for supervised learning of ANNs, the derivative of the loss with respect to each weight is obtained, the information passed backwards and the weights adjusted accordingly.

The above described concept can be formulated mathematically. A network with L layers can be defined as:

$$\mathbf{h}^{j}\mathbf{=}\sigma^{j}(W^{j} \bullet \mathbf{h}^{j - 1}\mathbf{) =}\sigma^{j}(z^{j}),\ \ j = 1,\ ...,\ L,$$

Where $\mathbf{h}^{j}$ is the state of the j-th hidden layer and $z_{i}^{j} = \sum_{k = 0}^{N^{j}}w_{ik}^{j}h_{i}^{j - 1}$ the input for the i-th neuron in the j-th hidden layer $\mathbf{h}^{L}$ is the output layer and $\mathbf{h}^{0}\mathbf{= x}$ the input layer. The forward-mapping is defined by the non-linear activation function $\sigma^{j}$ and the weight vector $W^{j}$. Most often, the bias term of the j-th layer $w_{0}^{j}$ is included in the weight vector, thus $W^{j} = (w_{0}^{j},\ w_{1}^{j},\ ...,\ w_{N}^{j})$ defines the weight vector of a layer containing $N_{\ }^{j}$ neurons. Now, let's fix the network architecture, meaning the number of neurons, the wiring scheme and the activation $\sigma^{j}$, and define the network parameters as all the weights $w_{i}^{j}$. If we define all parameters between layer j and l $(0\ \leq j < l \leq L)$ as $\theta_{W}^{j,l} = W^{k},\ k = j + 1,\ ...,\ l$, we can write the l-th layer $\mathbf{h}^{l}$ as a function of the j-th layer $\mathbf{h}^{j}$, given the parameters $\theta_{W}^{j,l}$:

$$\mathbf{h}^{l} = \mathbf{h}^{l}\mathbf{(}\mathbf{h}^{j}\mathbf{;}\theta_{W}^{j,l})$$

For a given data set $\left( \mathbf{x},\mathbf{y} \right) = \left( \left( \mathbf{x}_{\mathbf{1}}\mathbf{,}\mathbf{y}_{\mathbf{1}} \right)\mathbf{,\ldots,}\left( \mathbf{x}_{\mathbf{D}}\mathbf{,}\mathbf{y}_{\mathbf{D}} \right) \right)$, a global loss function $\mathcal{L(}\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{),\ y)}$ gives a measure of the difference between the network output $\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{)}$ and the label y. During training, the goal is to minimize the expectation value of the global loss $\mathbb{E}_{p}\{\mathcal{L(}\mathbf{h}^{L}\mathbf{(x;}\theta_{W}^{0,L}\mathbf{),\ y)}\mathbf{\}}$ based on a data distribution $\mathbf{p}\left( \mathbf{x},\mathbf{y} \right)$. Common loss functions are the Mean-Squared-Error (MSE) for regression:

$$\mathcal{L}_{MSE}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)\mathbf{=}\frac{\mathbf{1}}{\mathbf{2}\mathbf{D}}\sum_{\mathbf{i = 1}}^{\mathbf{D}}\left( \mathbf{h}^{L}\left( \mathbf{x}_{\mathbf{i}}\mathbf{;}\theta_{W}^{0,L} \right)\mathbf{-}\mathbf{y}_{\mathbf{i}} \right)^{\mathbf{2}}$$

And the Cross-Entropy Loss (CE) for classification with C classes:

$$\mathcal{L}_{CE}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)\mathbf{= -}\sum_{\mathbf{c = 1}}^{\mathbf{C}}\mathbf{y}_{\mathbf{c}}\mathbf{\log}{\mathbf{h}^{L}}_{\mathbf{c}}\left( \mathbf{x}_{\mathbf{c}}\mathbf{;}\theta_{W}^{0,L} \right)$$

$$\mathbf{\ }$$

If the backpropagation algorithm is used, updating the weights is done by taking the derivatives of the loss with respect to all weights:

$$w_{j}^{l}\lbrack t + 1\rbrack = w_{j}^{l}\lbrack t\rbrack - \eta\frac{\partial\mathcal{L}\left( \mathbf{h}^{L}\left( \mathbf{x;}\theta_{W}^{0,L} \right)\mathbf{,y} \right)}{\partial w_{j}^{l}}$$

The introduced learning-rate factor $\eta$ does in general not need to be constant, thus it can be adaptive over time.

Backpropagation Backpropagation (BP) is a widely used algorithm in training feedforward neural networks for supervised learning. It computes the gradient of the loss function with respect to the weights of the network for a single input/output example, and does so efficiently, unlike a naïve direct computation of the gradient with respect to each weight individually. This efficiency makes it feasible to use gradient methods for training multilayer networks, updating weights to minimize loss; gradient descent, or variants such as stochastic gradient descent, are commonly used. The backpropagation algorithm works by computing the gradient of the loss function with respect to each weight by the chain rule, computing the gradient one layer at a time, iterating backwards from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this is an example of dynamic programming.

We want to calculate the weight error, which is the gradient of the loss with respect to the input of the neuron j of the l-th layer $z_{j}^{l}$:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}}$$

First, we calculate the gradient with respect to the ultimate layer L:

$$\delta_{j}^{L} = \frac{\partial\mathcal{L}}{\partial z_{j}^{L}} = \frac{\partial\mathcal{L}}{\partial h_{j}^{L}}\frac{\partial h_{j}^{L}}{\partial z_{j}^{L}} = \frac{\partial\mathcal{L}}{\partial h_{j}^{L}}\sigma^{'}(z_{j}^{L})$$

Then, we calculate the gradient with respect to an intermediate layer l:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l + 1}}\frac{\partial z_{j}^{l + 1}}{\partial z_{j}^{l}} = \delta_{j}^{l + 1}\frac{\partial z_{j}^{l + 1}}{\partial z_{j}^{l}} = \sum_{k}^{}{\delta_{j}^{l + 1}w_{k}^{l + 1}}\sigma^{'}(z_{j}^{l})$$

We note, that the gradient can also be written with a dependency to the weight:

$$\delta_{j}^{l} = \frac{\partial\mathcal{L}}{\partial z_{j}^{l}} = \frac{\partial\mathcal{L}}{\partial w_{k}^{l}}\frac{\partial w_{k}^{l}}{\partial z_{j}^{l}}\ = \frac{\partial\mathcal{L}}{\partial w_{k}^{l}}\frac{1}{h_{k}^{l - 1}}$$

Thus, we arrive at the recursive form:

$$\frac{\partial\mathcal{L}}{\partial w_{k}^{l}} = h_{k}^{l - 1}\delta_{j}^{l}$$

Which gives us the weight update equation as follows:

$${w^{'}}_{k}^{l} = w_{k}^{l} - \eta\mathrm{\Delta}w_{k}^{l} = w_{k}^{l} - \eta\frac{\partial\mathcal{L}}{\partial w_{k}^{l}} = w_{k}^{l} - \eta h_{k}^{l - 1}\delta_{j}^{l}$$

Where $\eta$ is the learning rate.

Biological Plausibility Issues

• The backpropagation computation is purely linear, whereas biological neurons interleave linear and non-linear operations.
• If the feedback paths were used to propagate credit assignment by backpropagation, they would need precise knowledge of the derivates of the non-linearities at the operating point used in the corresponding feedforward computation.
• Similarly, these feedback paths would have to use exact symmetric weights (with the same connectivity, transposed) of the feedforward connections.
• Real neurons communicate by (possibly stochastic) binary values (spikes).
• The computation would have to be precisely clocked to alternate between feedforward and backpropagation phases.
• It is not clear where the output targets would come from.

Feedback Alignment (FA)

Feedback Alignment Papers: "Random Synaptic Feedback Weights Support Error Backpropagation for Deep Learning" & "Bio-Inspired Computer Vision: Towards a Synergistic Approach of Artificial and Biological Vision".

From a biological perspective, one of the biggest issues with backpropagation is that it uses the same weights for the forward and the backward pass. This is tackled with a learning method called feedback alignment. Instead of using the transposed weight matrix W^T^ for the backward pass, it uses a matrix of randomly initialized and fixed weights B. On simple example (such as MNIST), the method has been shown to work almost as well as regular back-propagation. We note the change in weight update:

Backpropagation

Feedback Alignment

Where $\beta^{l}$ is a random matrix with fixed weights (does not get updated) belonging to the l-th layer.

(A) The Backprop learning algorithm requires that neurons know each others' synaptic weights, for example, the three coloured synapses on the feedback cell at the bottom must have weights equal to those of the corresponding coloured synapses in the forward path. (B) Backprop computes teaching, or modulator, vectors by multiplying the error vector e by the transpose of the forward weight matrix W, that is, $\delta_{BP} = W^{T}e$. (C) Our feedback alignment method replaces $W^{T}$ with a matrix of fixed random weights, B, so that $\delta_{FA} = Be$. Thus, each neuron in the hidden layer receives a random projection of the error vector. (D) Potential synaptic circuitry underlying feedback alignment, shown for a single unit (matrix superscripts denote single synapses). There are many possible configurations that could support learning with feedback alignment, or algorithms like it, and it is this structural flexibility that we believe is important.

FA is on par with BP for Linear Classification Problems & Works in Multilayer Networks

Why Does Feedback Alignment Work? Some mathematical reasoning and the convergence proof is shown in the original paper and stated again in a follow-up paper. Intuitively: for FA the feedback weights are fixed, but if the forward weights are adapted, they will approximately align with the pseudo-inverse of the feedback weights in order to make the feedback useful. In some sense, the network learns how to learn, which is pretty dope.

Variations of Feedback Alignment Papers: "Direct Feedback Alignment Provides Learning in Deep Neural Networks" & "Adaptive Bidirectional Backpropagation: Towards Biologically Plausible Error Signal Transmission in Neural Networks".

There are variations of feedback alignment, which show to be useful especially for deeper network architectures. The variations shown below are Feedback Alignment (FA), Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IFA), Bi-directional Feedback Alignment (BFA) and Bi-directional Direct Feedback Alignment (BDFA).

The figure to the left gives an overview of different error transportation configurations. Grey arrows indicate activation paths and black arrows indicate error paths. Weights that are adapted during learning are denoted as Wi, and weights that are fixed and random are denoted as $B_i$. The figure to the right describes the Bi-directional Feedback Alignment. Black arrows represent forward activation paths. Red arrows indicate error (gradient) propagation paths.

The figure below introduces the last discussed variation, i.e., the Bi-directional Direct Feedback Alignment.

Deep Learning without Weight Transport Papers: "Deep Learning without Weight Transport".

Target Propagation

Target Propagation as a BP Alternative The concept of Target Propagation (targetprop) goes back to Lecun (1986). The intuition is simple: instead of focusing solely on the "forward - direction" model ($y = f(x)$), we also try to fit the "backward - direction" model ($x = g(y)$). f and g form an auto-encoding relationship: f is the encoder, creating a latent representation and predicted outputs given inputs x, and g is the decoder, generating input representations/samples from latent/output variables. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. By its nature, target propagation can in principle handle stronger (and even discrete) non-linearities, and it deals with the biological plausibility issues described before.

Local (layer-wise) Training for Deep Neural Networks

Local Learning Papers: "Training Neural Networks with Local Error Signals".

There are approaches that train neuronal networks using Local Layer error signals. This has the advantages that activations don't occupy space in memory, and that parallelization becomes easy (each layer in own GPU, train all simultaneously). One can use a combination of Similarity Matching Loss (sim) and Cross Entropy Loss (pred).

Optimization vs. Generalization

• Back-prop has fastest & lowest drop in training error.
• Local learning is competitive with back-prop in terms of test error.
• Local learning generalizes well.
• But: both pred- and sim- losses help optimization in a complementary way.
• Indeed, similarity matchings helps standard BP.

The Deep Feedback Control Method

Bio-Plausible Deep Learning Through Control A novel, Bio-plausible Network Learning Algorithm: "Deep Feedback Control". It is based on a recurrent loop that stops when the output error is 0.

• u(t) is the error.
• Qi and QL are set to be the inverse of the feedforward weights.
• The error is multiplied by Qi and QL in the different layers to update the weights.
• The gradient is implicit.
• Hebbian type learning rule for Q that it updates to the transpose of the weight.

Advantages of the Deep Feedback Control (DFC) Algorithm:

• Supports continuous/asynchronous updates.
• Hidden layer activity does not need to be stored (no extra memory required).
• Highly parallelizable, but requires custom hardware.
• Very simple learning rule that is local in space and time solely based on the neuron's activity and effectively implements a delta rule.
• Absence of phases or back-propagation of errors (e.g., as in standard BP).
• The optimization approach "Gauss-Newton" which is fundamentally different from BP and standard gradient descent learning.