Notes

Sanger's (PCA) Rule PCA Recap: It is a tool from statistics for data analysis. It can reveal structure in high-N-dimensional data that is not otherwise obvious. Like Hebbian learning, it discovers the direction of maximum variance in the data. But then in the (N - 1)-dimensional subspace perpendicular to that direction, it discovers the direction of maximum remaining variance, and so on for all N. The result is an ordered sequence of principal components. These are equivalently the eigenvectors of the correlation matrix C for zero-mean data, ordered by magnitude of eigenvalue in descending order. They are mutually orthogonal.

The idea is that we use a single Hebbian neuron that points in the direction of maximum variance, as described previously, and we view this as a principal component of the data. We subtract the contribution of this first principal component from the data, feeding the remaining data into a different neuron which subsequently identifies the direction of maximum variance in this data. This process can be repeated and resembles the addition of principal components in Principal Component Analysis (PCA).

We have seen that Hebbian learning, with appropriate provisions for preventing blow up, extracts the largest principal component. Let's take a look at two different neural network architectures capable of extracting more of them:

• Cascading Multiple Hebbian Neurons.
• Autoencoder Networks (we have already seen them before).

Algorithm:

1. Subtract the contribution of the first principal component.
2. Drive the difference into another Hebbian neuron.
3. This extracts the next principal component.
4. Subtract its contribution. Go to step 2. With N Hebbian neurons, we will get all N principal components.