Notes

De-Noising Autoencoder (DEA) Paper: "Extracting and Composing Robust Features with Denoising Autoencoders"

Since the autoencoder learns the identity function, we are facing the risk of "overfitting" when there are more network parameters than the number of data points. To avoid overfitting and improve the robustness, Denoising Autoencoder (Vincent et al. 2008) proposed a modification to the basic autoencoder. The input is partially corrupted by adding noises to or masking some values of the input vector in a stochastic manner. To "repair" the partially destroyed input, the denoising autoencoder has to discover and capture relationship between dimensions of input in order to infer missing pieces. Similar to dropout. Note: In the experiment of the original DAE paper, the noise is applied in that a fixed portion of input dimensions are selected at random and their values are forced to 0. This is very similar to Dropout but the denoising autoencoder was proposed in 2008, 4 years before the dropout paper (Hinton, et al. 2012).

Sparse Autoencoder The Sparse Autoencoder applies a sparsity constraint on the hidden unit activation to avoid overfitting and improve robustness. It forces the model to only have a small number of hidden units being activated at the same time.

Let's say there are $s_{l}$ neurons in the l-th hidden layer and the activation function for the j-th neuron in this layer is labelled as $a_{j}^{(l)}( \bullet ),\ \ j = 1,\ldots,s_{l}$. The fraction of activation of this neuron $\rho^{j}$ is expected to be a small number $\rho$, kowns as sparsity parameter; a common config is $\rho = 0.05$.

Keep in mind that we specify our desired target distribution that is $\rho$. Common activation functions include sigmoid, tanh, relu, leaky relu, etc. A neuron is activated when the value is close to 1 and inactive with a value close to 0.