A collection of fragments of understanding in the pursuit of deeper questions.
Contracting Autoencoder Similar to sparse autoencoder, Contractive Autoencoder (Rifai, et al., 2011) encourages the learned representation to stay in a contractive space for better robustness. It adds a term in the loss function to penalize the representation being too sensitive to the input, and thus improve the robustness to small perturbations around the training data points. The sensitivity is measured by the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input:
Where is one unit output in the compressed code .
This penalty term is the sum of squares of all partial derivatives of the learned encoding with respect to input dimensions. The authors claimed that empirically this penalty was found to carve a representation that corresponds to a lower-dimensional non-linear manifold, while staying more invariant to majority directions orthogonal to the manifold.