Notes

Autoencoders

• $g\phi$: encoding function.
• $f\phi$: decoding function.
• $\phi$: parameters of encoding function (trainable).
• $\theta$: parameters of decoding function (trainable).
• MSE Loss:

$$L_{AE} = \frac{1}{2}\left\| \mathbf{x -}\mathbf{x}^{\mathbf{'}} \right\|^{2},\mathbf{x}^{\mathbf{'}} = f_{\theta}\left( g_{\phi}(x) \right)$$

The autoencoder is trained by gradient descent.

Semi-Supervised Autoencoder Paper: "Supervised Autoencoders: Improving Generalization Performance with Unsupervised Regularizers".

The supervised AE uses the latent space to define a second decoding pathway. This path is added as another term to the loss and one calculates the gradients from two different ends. In the shared part, these gradients then merge. This is called multi-task learning (having a shared pathway for different objectives).

$$L_{SAE} = \frac{1}{t}\sum_{i = 1}^{t}\left\lbrack L_{P}\left( \mathbf{x}_{\mathbf{i}},\mathbf{W}_{\mathbf{1:2}},\mathbf{y}_{\mathbf{i}} \right) + L_{v}\left( \mathbf{x}_{\mathbf{i}},\mathbf{W}_{\mathbf{1:4}},{\widehat{\mathbf{x}}}_{\mathbf{i}} \right) \right\rbrack$$

where:

• $L_{P}$ is the loss of label $y_{i}$
• $L_{v}$ is the loss of reconstruction ${\widehat{x}}_{i}$
• $\mathbf{W}_{\mathbf{1:2}}$ are the weights of encoder (that produce label y)
• $\mathbf{W}_{\mathbf{1:4}}$ are the weights of encoder + decoder.