Notes

Infomax (ICA) Paper: "An Information-Maximisation Approach to Blind Separation and Blind Deconvolution".

Infomax is an optimization principle for artificial neural networks and other information processing systems. It prescribes that a function that maps a set of input values I to a set of output values O should be chosen or learned so as to maximize the average Shannon mutual information between I and O. One of the applications of Infomax has been to an independent component analysis (ICA) that finds independent signals by maximizing entropy. ICA via mutual information is one way to find independent components. The independence criterion is stronger than uncorrelatedness which is defined as:

$$\left\langle X_{1},X_{2} \right\rangle = \left\langle X_{2} \right\rangle\left\langle X_{1} \right\rangle = 0$$

Or

$$Cov\left( X_{1},X_{2} \right) = E\left\lbrack X_{1},X_{2} \right\rbrack - E\left\lbrack X_{1} \right\rbrack E\left\lbrack X_{2} \right\rbrack = 0$$

Remember: If two variables are uncorrelated, there is no linear relationship between them. However, this does not mean that they are independent. The other way works: if $X_{1}$ and$\ X_{2}$ are independent (with finite second moments), then they are uncorrelated.

For ICA we want statistical independence:

$$p\left( \alpha_{1:N} \right) = \prod_{i = 1}^{N}{p\left( \alpha_{i} \right)}$$

To measure the degree of dependence we look at the pairwise mutual information of two random variables X, Y. Mutual information is non-negative and symmetric:

We use the entropy:

Idea: X, Y independent if: $p(X,Y) = p(X)p(Y)$

For the discrete case we can then see best that last part of eq. is log(1) = 0:

In the picture below we have a Venn diagram of the Infomax objective: We want to maximize the entropy and minimize the mutual information. Entropy maximization forces the network to generalize.

• H(x) = Entropy of Distribution of the First Neuron's Output.
• H(x|y) = Conditional/Marginal Entropy.
• H(x,y) = Joint Entropy.
• I(x,y) = Mutual Information.

We want to ensure that the outputs $y_{i}$ are maximally independent. This is identical to requiring that the mutual information be small or alternatively that the joint entropy is large. Gradient ascent in this objective function is called INFOMAX (maximize the enclosed area representing both information quantities).

How do we actually implement Infomax? We can think of Infomax as a one layer linear neural network that produces: $\mathbf{y = Wx}$ such that outputs $y_{i}$ that are maximally independent:

but keep in mind we also maximize I(X;Y).

In the picture below we can see that the PCA features are very different from what we know that the brain uses for feature representation. However the ICA representation looks like the (Gabor-like) filters from the previous chapter.

• ICA is similar to Principal Component Analysis, except that we are looking for a transformation subject to the stronger requirement of independence, rather than uncorrelatedness.
• In general, no analytic solution (like eigenvalue decomposition for PCA) exists. Thus, ICA is typically implemented using neural network models and GD.
• For the ICA NN implementation, we need an architecture and an objective function to descend/climb.
• Results in N independent (or as independent as possible) components in an N-dimensional space; these don't need to be orthogonal.