Notes

Motivation Geoffrey Hinton ("Learning Representations by Back-Propagating Errors") suggest networks should be able to become intelligent on their own, unsupervised, without backprop.

• Does not need labelled data. In practice we often assume a fixed number of labels and call them cluster centers.
• Is able to learn/reveal the intricate structure (features) and can be used for dimensionality reduction. This means we try to find projections in this lower dimension space.
• Is able to generate new data examples that are consistent with the statistics of the training data.
• Can predict future data examples (e.g., video frame prediction). If we know the data representation we can try to find an inverse function out of our lower dimensional space. Which means we can sometimes reconstruct high dimensional data.
• Can be used to detect familiar or out-of-distribution examples. If we freeze our clusters, and then switch to another data set, then we can calculate the new data distance to the original data cluster centers.
• Can facilitate future supervised learning.
• Can be used to cluster the data.
• Can be used to de-noise data. We lose some information when projecting to a lower dimensional space. We optimize out method to retain the most useful information, this process might also de-noise our data.
• Can be used to encode data in a particular way (e.g., for data compression).
• We assume that humans have learned the world in an almost unsupervised way.

The difference between a supervised regression task where the data points are separated by some function that represents a bound in between different classes and an unsupervised clustering where the algorithm highlights the intrinsic structure of the data.

Unsupervised Learning

• Non-Probabilistic Models:
  • Sparse Coding
  • Autoencoders
  • KNN, K-Means
• Probabilistic (Generative) Models:
  • Tractable Models
    • Fully Observed Belief Nets
    • Pixel RNNs
  • Non-Tractable Models
    • Boltzmann Machine
    • Variational Autoencoders
    • Helmholtz Machines
    • Many others
  • Generative Adversarial Networks (GAN)
  • Moment-to-Moment Networks (not covered)