Notes

Feedback Alignment Papers: "Random Synaptic Feedback Weights Support Error Backpropagation for Deep Learning" & "Bio-Inspired Computer Vision: Towards a Synergistic Approach of Artificial and Biological Vision".

From a biological perspective, one of the biggest issues with backpropagation is that it uses the same weights for the forward and the backward pass. This is tackled with a learning method called feedback alignment. Instead of using the transposed weight matrix W^T^ for the backward pass, it uses a matrix of randomly initialized and fixed weights B. On simple example (such as MNIST), the method has been shown to work almost as well as regular back-propagation. We note the change in weight update:

Backpropagation

Feedback Alignment

Where $\beta^{l}$ is a random matrix with fixed weights (does not get updated) belonging to the l-th layer.

(A) The Backprop learning algorithm requires that neurons know each others' synaptic weights, for example, the three coloured synapses on the feedback cell at the bottom must have weights equal to those of the corresponding coloured synapses in the forward path. (B) Backprop computes teaching, or modulator, vectors by multiplying the error vector e by the transpose of the forward weight matrix W, that is, $\delta_{BP} = W^{T}e$. (C) Our feedback alignment method replaces $W^{T}$ with a matrix of fixed random weights, B, so that $\delta_{FA} = Be$. Thus, each neuron in the hidden layer receives a random projection of the error vector. (D) Potential synaptic circuitry underlying feedback alignment, shown for a single unit (matrix superscripts denote single synapses). There are many possible configurations that could support learning with feedback alignment, or algorithms like it, and it is this structural flexibility that we believe is important.

FA is on par with BP for Linear Classification Problems & Works in Multilayer Networks

Why Does Feedback Alignment Work? Some mathematical reasoning and the convergence proof is shown in the original paper and stated again in a follow-up paper. Intuitively: for FA the feedback weights are fixed, but if the forward weights are adapted, they will approximately align with the pseudo-inverse of the feedback weights in order to make the feedback useful. In some sense, the network learns how to learn, which is pretty dope.

Variations of Feedback Alignment Papers: "Direct Feedback Alignment Provides Learning in Deep Neural Networks" & "Adaptive Bidirectional Backpropagation: Towards Biologically Plausible Error Signal Transmission in Neural Networks".

There are variations of feedback alignment, which show to be useful especially for deeper network architectures. The variations shown below are Feedback Alignment (FA), Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IFA), Bi-directional Feedback Alignment (BFA) and Bi-directional Direct Feedback Alignment (BDFA).

The figure to the left gives an overview of different error transportation configurations. Grey arrows indicate activation paths and black arrows indicate error paths. Weights that are adapted during learning are denoted as Wi, and weights that are fixed and random are denoted as $B_i$. The figure to the right describes the Bi-directional Feedback Alignment. Black arrows represent forward activation paths. Red arrows indicate error (gradient) propagation paths.

The figure below introduces the last discussed variation, i.e., the Bi-directional Direct Feedback Alignment.

Deep Learning without Weight Transport Papers: "Deep Learning without Weight Transport".