Notes

Model-Based ML Model-based meta-learning models make no assumption on the form of $P_{\theta}\left( y \middle| x \right)$. Rather it depends on a model designed specifically for fast learning - a model that updates its parameters rapidly with a few training steps. This rapid parameter update can be achieved by its internal architecture or controlled by another meta-learner model.

Hypernetworks Paper: "Continual Learning in Recurrent Neural Networks", "Continual Learning with Hypernetworks", "Meta-Learning with Latent Embedding Optimization".

They are networks that generate the weights of a target model based on task identity. Continual Learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks., additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories.

Commonly, the parameters of a neural network are directly adjusted from data to solve a task. Here, a weight generator termed hypernetwork is learned instead. Hypernetworks map embedding vectors to weights, which parametrize a target neural network. In a continual learning scenario, a set of task-specific embeddings is learned via backpropagation. Embedding vectors provide task-dependent context and bias the hypernetwork to particular solutions.

Few-Shot Meta-Learning with Hypernetworks In a few-shot meta-learning setting, a base network is trained on a set of tasks, and then the parameters of this base network are used as inputs to a hypernetwork, which generates the task-specific weights for the base network. When presented with a new task, the base network's parameters are passed through the hypernetwork again, generating the weights for the base network to use on the new task. The key idea behind this approach is that the base network's parameters contain information about how to solve a wide variety of tasks, and the hypernetwork learns to generate task-specific weights based on this information. This allows the base network to quickly adapt to new tasks with very little data, because it can leverage its previous experience to quickly learn the new task.

In the figure we have two experimental results: (A) Experiments on the permuted MNIST benchmark. Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNIST-100). Task-conditioned hypernetworks (hnet, in red) achieve very large memory lifetimes on the permuted MNIST benchmark. Synaptic Intelligence (SI, in blue), online EWC (in orange) and Deep Generative Replay (DGR+distill, in green) methods are shown for comparison. (B) Split CIFAR-10/100 continual learning benchmark. Test set accuracies (mean +- STD, n=5) on the entire CIFAR-10 dataset and subsequent CIFAR-100 splits. The hypernetwork-protected ResNet-32 displays virtually no forgetting; final averaged performance (hnet, in red) matches the immediate one (hnet-during, in blue). Furthermore, information is transferred across tasks, as performance is higher than when training each task from scratch (purple).