Notes

Oja's Rule Paper: "A Simplified Neuron Model as a Principal Component Analyzer".

A modification of the Hebbian rule above in which a weight decay term is added. As this weight decay term is proportional to $y^{2}$, a quadratic result, it eventually limits the magnitude of the weights w to unit length while maintaining the tendency of the weights to point in the direction of maximum variance.

$$\nabla w_{oja} = \nabla w - y^{2} \times w = \alpha \times x \times y - y^{2} \times w = \alpha \times y(x - y \times w)$$