Notes

Training the Restricted Boltzmann Machine (Contrastive Divergence) Paper: "Training Products of Experts by Minimizing Contrastive Divergence", "Reducing the Dimensionality of Data with Neural Networks".

Is not used anymore because back-propagation works so well. Attempts were made to use RBMs as dimensionality reduction algorithm. Results were okay, compared to PCA, the clusters seem more dense and separated.

Training algorithm:

1. Take a sample $V_{0}$ and compute the hidden activation vector $h_{0}$. Call $V_{0}h_{0}^{T}$ (outer product) the positive gradient.
2. Sample from $h_{0}$, compute $V_{1}$, resample from $V_{1}$, compute $h_{1}$. Call $V_{1}h_{1}^{T}$ the negative gradient.
3. Update weights: $\nabla w = \eta\left( v_{0}h_{0}^{T} - v_{1}h_{1}^{T} \right)$.