Notes

Content of the Lecture

• Meta-Learning with ANNs
  • What is Meta-Learning?
  • Different Types of Meta-Learning Approaches.
  • Optimization Based (MAML).
  • Model Based (Meta and Hyper Networks).
• Meta-Learning in the Brain
  • Evidence for Meta-Learning in the Brain.
  • The Prefrontal-Cortex as Meta-Reinforcement Learner.
  • The Role of Neuromodulation in Meta Learning.

Meta-Learning with ANNs - What is Meta-Learning?

Paper: "Human-Level Concept Learning Through Probabilistic Program Induction".

In the first lecture we listed the standing challenges in deep learning research, from which we now want to discuss continual learning and meta-learning, which can allow to learn fast and from few-data only. So why do we need continual and meta-learning?

• For many applications we don't have large training datasets (medical imaging, robotics, recommendations, real-world agent training).
• Life-long learning systems should quickly adapt to new tasks, but not forget previous ones (can't learn every task/classifier from scratch).
• Sometimes our training data has a long-tail, meaning that there are only a few data points for a large range of categories.
• Concept learning enables humans to extrapolate from learned tasks to a similar task.

The Principle of Learning the Learn

Meta-Learning with ANNs Meta-learning, also known as "learning to learn", intends to design models that can learn new skills or adapt to new environments rapidly with a few training examples. There are three common approaches:

• Learn an efficient distance metric (metric-based).
• Use (recurrent) network with external or internal memory (model-based).
• Optimize the model parameters explicitly for fast learning (optimization-based).

A good meta-learning model should be trained over a variety of learning tasks and optimized for the best performance on a distribution of tasks, including potentially unseen tasks. Each task is associated with a dataset $\mathcal{D}$, containing both feature vectors and true labels. The optimal model parameters are:

It looks very similar to a normal learning task, but one dataset is considered as one data sample. The concept of Few-shot classification is an instantiation of meta-learning in the field of supervised learning. The dataset $\mathcal{D}$ is often split into two parts, a support set $\mathcal{S}$ for learning and a prediction set $\mathcal{B}$ for training or testing, $\mathcal{D =}\left\langle S,B \right\rangle$. Another popular view of meta-learning decomposes the model update into two stages:

1. A classifier $f_{\theta}$ is the "learner" model, trained for operating a given task.
2. In the meantime, an optimizer $g_{\phi}$ learns how to update the learner model's parameters via the support set $S,\theta^{'} = g_{\phi}(\theta,S).$

In the final optimization step, one needs to update both $\theta$ and $\phi$ to maximize:

Meta-Learning with ANNs - Metric-Based (Prototypical, Siamese, Matching and Relation Networks)

Metric-Based ML The core idea in metric-based meta-learning is similar to nearest neighbors algorithms and kernel density estimation. The predicted probability over a set of known labels y is a weights sum of label of support set samples. The weight is generated by a kernel function $k_{\theta}$, measuring the similarity between two data samples.

To learn a good kernel is crucial to the success of a metric-based meta-learning model. Metric learning is well aligned with this intention, as it aims to learn a metric or distance function over objects. The notion of a good metric is problem-dependent. It should represent the relationship between inputs in the task space and facilitate problem solving. A few models are introduced, that learn embedding vectors of input data explicitly and use them to design proper kernel functions:

• Prototypical Networks ("Prototypical Networks for Few-shotLearning")
• Siamese Networks
• Matching Networks ("Matching Networks for One Shot Learning")
• Relation Networks ("Learning to Compare: Relation Network for Few-Shot Learning")

Prototypical Networks They use an embedding function $f_{\theta}$ to encode each input into a M-dimensional feature vector. A prototype feature vector is defined for every class $c \in C$, as the mean vector of the embedded support data samples in this class.

The distribution over classes for a given test input x is a softmax over the inverse of distances between the test data embedding and prototype vectors.

where $d_{\varphi}$ can be any distance function as long as $\varphi$ is differentiable. In the paper, they used the squared Euclidean distance. The loss function is the negative log-likelihood:

Siamese Networks They are composed of two twin networks and their outputs are jointly trained on top with a function to learn the relationship between pairs of input data samples. The twin networks are identical, sharing the same weights and network parameters. In other words, both refer to the same embedding network that learns an efficient embedding to reveal relationship between pairs of data points. Convolutional Siamese Neural Networks have been applied to one-shot image classification.

Training: The Siamese network is trained for a verification task for telling whether two input images are in the same class. It outputs the probability of two images belonging to the same class.

1. First, convolutional Siamese network learns to encode two images into feature vectors via an embedding function $f_{\theta}$ which contains a couple of convolutional layers.
2. The L1-distance between two embeddings is $\left| f_{\theta}\left( \mathbf{x}_{\mathbf{i}} \right) - f_{\theta}\left( \mathbf{x}_{\mathbf{j}} \right) \right|$
3. The distance is converted to a probability p by a linear feedforward layer and sigmoid. It is the probability of whether two images are drawn from the same class.
4. Intuitively the loss is cross-entropy because the label is binary.

$$p\left( \mathbf{x}_{\mathbf{i}}\mathbf{,}\mathbf{x}_{\mathbf{j}} \right) = \sigma\left( \mathbf{W}\left| f_{\theta}\left( \mathbf{x}_{\mathbf{i}} \right) - f_{\theta}\left( \mathbf{x}_{\mathbf{j}} \right) \right| \right)$$

Testing: The Siamese network processes all the image pairs between a test image and every image in the support set. The final prediction is the class of the support image with the highest probability. Given a support set S and a test image $\mathbf{x}$, the final predicted class is:

where c(x) is the class label of an image x and $\widehat{c}( \bullet )$ is the predicted label.

Matching Networks

They aim at learning a classifier $c_{S}$ for any given (small) support set $S = {{\text{\{}x}_{i},y_{i}\}}_{i = 1}^{k}$ (k-shot classification). This classifier defines a probability distribution over output labels y given a test example $\mathbf{x}$. Similar to other metric-based models, the classifier output is defined as a sum of labels of support samples weighted by attention kernel $a\left( \mathbf{x,}\mathbf{x}_{\mathbf{i}} \right)$ - which should be proportional to the similarity between $\mathbf{x}$ and $\mathbf{x}_{\mathbf{i}}$.

The attention kernel depends on two embedding functions, f and g, for decoding the test sample and the support set samples respectively. The attention weight between two data points is the cosine similarity, cosine(.), between their embedding vectors, normalized by softmax:

The embedding has to be chosen carefully. In a simple version, an embedding function is a neural network with a single data sample as input. Taking a single data point as input might not be enough to efficiently gauge the entire feature space. Therefore, the Matching Network model further proposed to enhance the embedding functions by taking as input the whole support set S in addition to the original input, so that the learned embedding can be adjusted based on the relationship with other support samples.

Relation Networks They are similar to Siamese Networks but with a few differences:

• The relationship is not captured by a simple L1 distance in the feature space, but predicted by a CNN classifier $g_{\phi}$. The relation score between a pair of inputs, $\mathbf{x}_{\mathbf{i}}$ and $\mathbf{x}_{\mathbf{j}}$, is $r_{ij} = g_{\phi}\left( \left\lbrack \mathbf{x}_{\mathbf{i}}\mathbf{,}\mathbf{x}_{\mathbf{j}} \right\rbrack \right)$ where [.,.] is a concatenation.
• The objective function is MSE loss instead of cross-entropy, because conceptually RN focuses more on predicting relation scores which is more like regression, rather than binary classification: $\mathcal{L}(B) = \sum_{\left( \mathbf{x}_{\mathbf{i}},\mathbf{x}_{\mathbf{j}},y_{i},y_{j} \right) \in B}^{}\left( r_{ij} - \mathbf{1}_{y_{i} = y_{j}} \right)^{2}$.

In the figure we have a Relation Network architecture for a 5-way 1-shot problem with one query example.

Meta-Learning with ANNs - Model-Based (Meta & Hyper Networks)

Model-Based ML Model-based meta-learning models make no assumption on the form of $P_{\theta}\left( y \middle| x \right)$. Rather it depends on a model designed specifically for fast learning - a model that updates its parameters rapidly with a few training steps. This rapid parameter update can be achieved by its internal architecture or controlled by another meta-learner model.

Hypernetworks Paper: "Continual Learning in Recurrent Neural Networks", "Continual Learning with Hypernetworks", "Meta-Learning with Latent Embedding Optimization".

They are networks that generate the weights of a target model based on task identity. Continual Learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks., additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories.

Commonly, the parameters of a neural network are directly adjusted from data to solve a task. Here, a weight generator termed hypernetwork is learned instead. Hypernetworks map embedding vectors to weights, which parametrize a target neural network. In a continual learning scenario, a set of task-specific embeddings is learned via backpropagation. Embedding vectors provide task-dependent context and bias the hypernetwork to particular solutions.

Few-Shot Meta-Learning with Hypernetworks In a few-shot meta-learning setting, a base network is trained on a set of tasks, and then the parameters of this base network are used as inputs to a hypernetwork, which generates the task-specific weights for the base network. When presented with a new task, the base network's parameters are passed through the hypernetwork again, generating the weights for the base network to use on the new task. The key idea behind this approach is that the base network's parameters contain information about how to solve a wide variety of tasks, and the hypernetwork learns to generate task-specific weights based on this information. This allows the base network to quickly adapt to new tasks with very little data, because it can leverage its previous experience to quickly learn the new task.

In the figure we have two experimental results: (A) Experiments on the permuted MNIST benchmark. Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNIST-100). Task-conditioned hypernetworks (hnet, in red) achieve very large memory lifetimes on the permuted MNIST benchmark. Synaptic Intelligence (SI, in blue), online EWC (in orange) and Deep Generative Replay (DGR+distill, in green) methods are shown for comparison. (B) Split CIFAR-10/100 continual learning benchmark. Test set accuracies (mean +- STD, n=5) on the entire CIFAR-10 dataset and subsequent CIFAR-100 splits. The hypernetwork-protected ResNet-32 displays virtually no forgetting; final averaged performance (hnet, in red) matches the immediate one (hnet-during, in blue). Furthermore, information is transferred across tasks, as performance is higher than when training each task from scratch (purple).

Meta-Learning with ANNs - Optimization-Based (Model-Agnostic Meta-Learning)

Optimization-Based ML Deep Learning models learn through backpropagation of gradients. However, the gradient-based optimization is neither designed to cope with a small number of training sample, nor to converge within a small number of optimization steps. Is there a way to adjust the optimization algorithm so that the model can be good at learning with a few examples? This is what optimization-based approach meta-learning algorithms intend for. Look at Model-Agnostic Meta-Learning (MAML) and LSTM Meta-Learner, Reptile for further information (Not covered in this class).

Model-Agnostic Meta-Learning (MAML) This is a fairly general optimization algorithm, compatible with any model that learns through gradient descent. Let's say our model is $f_{\theta}$ with parameters $\theta$. Given a task $\tau_{i}$ and its associated dataset ($\mathcal{D}^{(i)}$ train, $\mathcal{D}^{(i)}$ test), we can update the model parameters by one or more gradient descent steps (the following example only contains one step):

where $\mathcal{L}^{(0)}$ is the loss computed using the mini data batch with id (0). The above formula only optimizes for one task. To achieve a good generalization across a variety of tasks, we would like to find the optimal $\theta^{*}$ so that the task-specific fine-tuning is more efficient. Now, we sample a new data batch with id (1) for updating the meta-objective. The loss, denoted as $\mathcal{L}^{(1)}$, depends on the mini batch (1). The superscripts in $\mathcal{L}^{(0)}$ and $\mathcal{L}^{(1)}$ only indicate different data batches, and they refer to the same loss objective for the same task.

Meta-Learning in the Brain

Meta-Learning in the Brain Over the past 20 years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine "stamps in" associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. A recent study introduces a new theory, where the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deal with a wider range of observations.

In the picture we have a Meta-RL architecture across episodes to learn efficiently within an episode. (a) Agent architecture. The prefrontal network (PFN), including sectors of the basal ganglia and the thalamus that connects directly with PFC, is modeled as a recurrent neural network, with synaptic weights adjusted through an RL algorithm driven by dopamine (DA); o is perceptual input, a is action, r is reward, v is state value, t is time-step and $\sigma$ is RPE. The central box denotes a single, fully connected set of LSTM units. (b) A more detailed schematic of the neural network implementation used in the stimulations.

Meta-Learning via Neuromodulation Neuromodulators play an important role in meta-learning in the brain. Some of the key modulators are listed below. Neuromodulatory systems can be seen to mediate the global signals that regulate the distributed learning mechanisms in the brain. Based on the review of experimental data and theoretical models, some key modulators are described below:

• Dopamine is proposed to act as a "global learning" signal, critical to prediction of rewards and action reinforcement.
• Serotorin is proposed to control the balance between short and long term reward prediction, essentially by variably "discounting" expected future reward sums that may require too much expenditure to achieve.
• Norepinephrine is proposed to facilitate "wide exploration" by stochastic action selection (control exploration vs. exploitation).
• Acetylcholine is proposed to facilitate the balance between memory storage and memory renewal, finding an optimal balance between stability and effectiveness of learning algorithms for the specific environmental task.

The paper "Reinforcement Learning, Fast and Slow" presents a framework for understanding the difference between two types of reinforcement learning algorithms: "fast" RL and "slow" RL. Fast RL algorithms, such as Q-learning, learn quickly but are prone to overfitting and instability. Slow RL algorithms, such as Policy Gradient methods, learn more slowly but are more stable and less prone to overfitting. The paper argues that a combination of fast and slow RL can lead to better performance in a variety of tasks. Additionally, the paper also suggest that human learning can be understood in terms of these two types of RL, with fast RL corresponding to trial-and-error learning and slow RL corresponding to more deliberate, goal-directed learning.

The Harlow experiment is a study conducted by psychologist Harry Harlow in the 1950s, which aimed to investigate the role of learning in the development of complex problem-solving abilities. The experiment used rhesus monkeys as subjects, and it consisted of two parts. In the first part, the monkeys were presented with a series of problems, such as reaching through a hole in a barrier to retrieve food. The monkeys were allowed to explore the problems and find solutions through trial and error. In the second part of the experiment, the monkeys were presented with a new set of problems that were more difficult than the ones they had encountered before. The monkeys were able to use the knowledge and skills they had acquired during the first part of the experiment to solve the new problems more quickly and effectively. This experiment demonstrated that the experience of solving problems through trial and error could lead to the development of problem-solving skills and strategies, which can be applied to new and more complex problems. This experiment was important in supporting the idea that learning to learn is possible, and that this type of learning can be achieved through experience and exposure to different challenges.

The concept of "Bio-plausible Modulatory Networks" is a method that attempts to mimic the way the brain continually learns. This approach is based on the idea that the brain uses a combination of different neural networks, each with a specific function, to process and learn from information. These networks work together and communicate with each other through modulatory signals, which can adjust the activity of different networks depending on the task or context. In this approach, the continual learning model is composed of several networks, each specialized in a specific task, and these networks are modulated by a central controller network. The central controller network is responsible for adapting the activity of the specialized networks depending on the task to be solved, and this allows the model to continue to learn new tasks without forgetting the previous ones.