Notes

Target Propagation as a BP Alternative The concept of Target Propagation (targetprop) goes back to Lecun (1986). The intuition is simple: instead of focusing solely on the "forward - direction" model ($y = f(x)$), we also try to fit the "backward - direction" model ($x = g(y)$). f and g form an auto-encoding relationship: f is the encoder, creating a latent representation and predicted outputs given inputs x, and g is the decoder, generating input representations/samples from latent/output variables. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. By its nature, target propagation can in principle handle stronger (and even discrete) non-linearities, and it deals with the biological plausibility issues described before.