Notes

Metric-Based ML The core idea in metric-based meta-learning is similar to nearest neighbors algorithms and kernel density estimation. The predicted probability over a set of known labels y is a weights sum of label of support set samples. The weight is generated by a kernel function $k_{\theta}$, measuring the similarity between two data samples.

To learn a good kernel is crucial to the success of a metric-based meta-learning model. Metric learning is well aligned with this intention, as it aims to learn a metric or distance function over objects. The notion of a good metric is problem-dependent. It should represent the relationship between inputs in the task space and facilitate problem solving. A few models are introduced, that learn embedding vectors of input data explicitly and use them to design proper kernel functions:

• Prototypical Networks ("Prototypical Networks for Few-shotLearning")
• Siamese Networks
• Matching Networks ("Matching Networks for One Shot Learning")
• Relation Networks ("Learning to Compare: Relation Network for Few-Shot Learning")

Prototypical Networks They use an embedding function $f_{\theta}$ to encode each input into a M-dimensional feature vector. A prototype feature vector is defined for every class $c \in C$, as the mean vector of the embedded support data samples in this class.

The distribution over classes for a given test input x is a softmax over the inverse of distances between the test data embedding and prototype vectors.

where $d_{\varphi}$ can be any distance function as long as $\varphi$ is differentiable. In the paper, they used the squared Euclidean distance. The loss function is the negative log-likelihood:

Siamese Networks They are composed of two twin networks and their outputs are jointly trained on top with a function to learn the relationship between pairs of input data samples. The twin networks are identical, sharing the same weights and network parameters. In other words, both refer to the same embedding network that learns an efficient embedding to reveal relationship between pairs of data points. Convolutional Siamese Neural Networks have been applied to one-shot image classification.

Training: The Siamese network is trained for a verification task for telling whether two input images are in the same class. It outputs the probability of two images belonging to the same class.

1. First, convolutional Siamese network learns to encode two images into feature vectors via an embedding function $f_{\theta}$ which contains a couple of convolutional layers.
2. The L1-distance between two embeddings is $\left| f_{\theta}\left( \mathbf{x}_{\mathbf{i}} \right) - f_{\theta}\left( \mathbf{x}_{\mathbf{j}} \right) \right|$
3. The distance is converted to a probability p by a linear feedforward layer and sigmoid. It is the probability of whether two images are drawn from the same class.
4. Intuitively the loss is cross-entropy because the label is binary.

$$p\left( \mathbf{x}_{\mathbf{i}}\mathbf{,}\mathbf{x}_{\mathbf{j}} \right) = \sigma\left( \mathbf{W}\left| f_{\theta}\left( \mathbf{x}_{\mathbf{i}} \right) - f_{\theta}\left( \mathbf{x}_{\mathbf{j}} \right) \right| \right)$$

Testing: The Siamese network processes all the image pairs between a test image and every image in the support set. The final prediction is the class of the support image with the highest probability. Given a support set S and a test image $\mathbf{x}$, the final predicted class is:

where c(x) is the class label of an image x and $\widehat{c}( \bullet )$ is the predicted label.

Matching Networks

They aim at learning a classifier $c_{S}$ for any given (small) support set $S = {{\text{\{}x}_{i},y_{i}\}}_{i = 1}^{k}$ (k-shot classification). This classifier defines a probability distribution over output labels y given a test example $\mathbf{x}$. Similar to other metric-based models, the classifier output is defined as a sum of labels of support samples weighted by attention kernel $a\left( \mathbf{x,}\mathbf{x}_{\mathbf{i}} \right)$ - which should be proportional to the similarity between $\mathbf{x}$ and $\mathbf{x}_{\mathbf{i}}$.

The attention kernel depends on two embedding functions, f and g, for decoding the test sample and the support set samples respectively. The attention weight between two data points is the cosine similarity, cosine(.), between their embedding vectors, normalized by softmax:

The embedding has to be chosen carefully. In a simple version, an embedding function is a neural network with a single data sample as input. Taking a single data point as input might not be enough to efficiently gauge the entire feature space. Therefore, the Matching Network model further proposed to enhance the embedding functions by taking as input the whole support set S in addition to the original input, so that the learned embedding can be adjusted based on the relationship with other support samples.

Relation Networks They are similar to Siamese Networks but with a few differences:

• The relationship is not captured by a simple L1 distance in the feature space, but predicted by a CNN classifier $g_{\phi}$. The relation score between a pair of inputs, $\mathbf{x}_{\mathbf{i}}$ and $\mathbf{x}_{\mathbf{j}}$, is $r_{ij} = g_{\phi}\left( \left\lbrack \mathbf{x}_{\mathbf{i}}\mathbf{,}\mathbf{x}_{\mathbf{j}} \right\rbrack \right)$ where [.,.] is a concatenation.
• The objective function is MSE loss instead of cross-entropy, because conceptually RN focuses more on predicting relation scores which is more like regression, rather than binary classification: $\mathcal{L}(B) = \sum_{\left( \mathbf{x}_{\mathbf{i}},\mathbf{x}_{\mathbf{j}},y_{i},y_{j} \right) \in B}^{}\left( r_{ij} - \mathbf{1}_{y_{i} = y_{j}} \right)^{2}$.

In the figure we have a Relation Network architecture for a 5-way 1-shot problem with one query example.