Notes

Sparse Coding The sparse code is found when each sample of a given data set is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons. Each image can be represented by a set of basis functions: $I(x,y) \approx \sum_{i}^{}a_{i}\phi_{i}(x,y) = \widehat{I}$.

We define an energy function which is essentially the true image I minus approximation $\overline{I}$ + regularizer S:

Presented in the lectures was a set of known regularizers $\mathbb{S}$:

For each image presentation E is minimized with respect to $a_{i}$. Thus, for a given image, the $a_{i}$ are determined from the equilibrium solution to the differential equation:

The $\phi_{i}$ then evolve by gradient descent on E averaged over many image presentations. The learning rule for updating $\phi$ is then:

Comments on Parameters and Operations:

• $a_{i}$ factors should be sparse, different for each sample.
• $\phi$ components, basis functions that are the same over all samples.
• $\tau$ scaling constant.
• $\lambda$ scaling constant: sparseness vs information perseverance.
• $\eta$ learning rate.
• $\left\langle x \right\rangle$ mean.

In the figure we have representative training images are shown at the left and the resulting basis functions that were learned from these examples are shown at the right. In a, images were composed of sparse pixels: each pixel was activated independently according to an exponential distribution. In b, images were composed similarly to a, except with gratings instead of pixels (i.e., sparse pixels in the Fourier domain). In c, images were composed of spare, non-orthogonal Gabor functions with the methods described by Field. In all cases, the basis functions were initialized to random initial conditions. The learned basis functions successfully recover the sparse components from which the images were composed.

Relation to Neuroscience Paper: "Spatial Structure of Neuronal Receptive Field in Awake Monkey Secondary Visual Cortex (V2)".

This paper shows that cells of sub-units in V1 have receptive fields that apply signal filtering that is very similar to sparse coding. In V2 they identified sub-units with spatial feature selectivity.