Notes

Local Learning Papers: "Training Neural Networks with Local Error Signals".

There are approaches that train neuronal networks using Local Layer error signals. This has the advantages that activations don't occupy space in memory, and that parallelization becomes easy (each layer in own GPU, train all simultaneously). One can use a combination of Similarity Matching Loss (sim) and Cross Entropy Loss (pred).

Optimization vs. Generalization

• Back-prop has fastest & lowest drop in training error.
• Local learning is competitive with back-prop in terms of test error.
• Local learning generalizes well.
• But: both pred- and sim- losses help optimization in a complementary way.
• Indeed, similarity matchings helps standard BP.