Notes

Content of the Lecture

• What do we mean by catastrophic forgetting and continual learning?
• How do we build models that are more resilient to forgetting?
• How does this relate to what happens in a biological brain?
• More in-depth study of a few algorithms.

Continual Learning - Introduction

Learning Arithmetics & Word Associations "Catastrophic interference in connectionist networks: the sequential learning problem". In front of a 2-task incremental learning scenario, Humans can still perform decently on the first task after having learned the second one, while neural nets immediately forget the first when they start learning the second.

In the picture we have the view of the parameter space.

Continual Learning "Scenarios" and Benchmarks There are three scenarios:

• Task-IL, solve tasks so far, task-ID provided.
• Domain-IL, solve tasks so far, task-ID not provided.
• Class-IL, solve tasks so far and infer task-ID.

Continual Learning - Strategies

Strategies for Continual Machine Learning

• Architectural
  • Use only part of a network.
  • Select subnetwork based on task.
  • Or, add new parts to a network when new tasks are learnt.
  • Progressive Networks.
• Regularization-Based
  • Add loss terms that try to keep the network constant:
    • In behavior or
    • In weights.
    • Elastic Weight Consolidation & Synaptic Intelligence.
• Data Replay-Based
  • Store, or learn to generate some of the previous data (Experience Replay).
  • Keep training on them while you train on new data (Generative Replay).
  • Deep Generative Replay.

An Example of Architectural methods (Progressive Networks) "Progressive Neural Networks" is a paper published by Google Brain team in 2017, that describes a method for incremental learning, which allows neural networks to learn new tasks without forgetting the previous ones. The paper propose a technique called Progressive Networks (PN), which is based on the idea of growing the neural network incrementally as new tasks are encountered. The PN approach consists of a multi-task neural network, where each task is associated with a specific sub-network called a "column". Each column is trained to perform a specific task, and new columns can be added as new tasks are encountered. The new columns are connected to the previously learned columns, allowing the network to transfer knowledge from previous tasks to new ones. The paper shows that the PN approach can learn a wide range of tasks, with different levels of difficulty, and it can also achieve better performance compared to other methods for incremental learning, such as fine-tuning or freezing the previous layers.

The Progressive Network approach is useful in scenarios where the number of tasks or the amount of data is not known in advance, and it can be useful in applications such as lifelong learning, online learning and online adaptation.

Continual Learning - Regularization Methods (Elastic Weight Consolidation & Synaptic Intelligence)

Regularization Methods Regularization methods are used in continual learning to prevent catastrophic forgetting, which occurs when a model forgets previously learned tasks when learning new tasks.

Elastic Weight Consolidation (EWC) Elastic Weight Consolidation (EWC) aims to keep the parameters of a neural network that are important for previous tasks fixed while allowing the parameters that are important for new tasks to change. This is done by adding a penalty term to the loss function based on the difference between the current parameters and the parameters that were optimal for the previous tasks.The main idea of EWC is to keep the parameters of a neural network that are important for previous tasks fixed while allowing the parameters that are important for new tasks to change. This is done by adding a penalty term to the loss function based on the difference between the current parameters and the parameters that were optimal for the previous tasks. The importance of each parameter is measured by the Fisher information matrix, which quantifies the amount of information that the parameter contains about the task.

Synaptic Intelligence (SI) Synaptic Intelligence (SI) aims to keep the parameters that were important for previous tasks fixed by adjusting the learning rate of each parameter based on how much it has changed during previous tasks. The main idea of SI is to adjust the learning rate of each parameter in a neural network based on how much the parameter has changed during previous tasks. The authors propose a measure of the "importance" of each parameter, which is based on the magnitude of the gradient of the parameter with respect to the loss function during previous tasks. Parameters that have had a large gradient in the past are considered more important and have a lower learning rate, while parameters that have had a small gradient in the past have a higher learning rate. The authors test SI on a variety of image classification tasks and show that it outperforms EWC and other baselines. The paper also introduces a novel method for evaluating the performance of continual learning algorithms called "learning progress".

Continual Learning - Data Replay Methods

Data Replay Methods

Data Replay methods in continual learning involve storing previously seen data and reusing it to help the model retain information from previous tasks when learning new tasks. This can be done in several ways, however the most common are:

• Experience Replay: this method involves storing a dataset of previous experiences, such as input-output pairs, and randomly sampling from this dataset during training on new tasks to provide additional examples for the model to learn from.
• Generative Replay: this method involves training a generative model, such as a Generative Adversarial Network (GAN) or Variational Autoencoder (VAE), on the previous task data, and then using the generative model to generate new examples from the previous task to be used during training on new tasks.

Deep Generative Replay Data Generative Replay is a method in continual learning that uses a generative model to generate new examples from previous tasks to be used during training on new tasks. The general process of Data Generative Replay is as follows:

1. A generative model, such as a Generative Adversarial Network (GAN) or Variational Autoencoder (VAE), is trained on data from a previous task.
2. The generative model is then used to generate new examples from the previous task.
3. These generated examples are added to the training set for the new task, along with the examples from the new task.
4. The model is then trained on the combined dataset (new task examples + generated examples from previous task)
5. Steps 1-4 are repeated for each new task.

Continual Learning & the Brain

How Does the Brain Do Continual Learning? May these ML methods be relevant for the brain?

The Stability - Plasticity Dilemma In ML, we can either:

• Speed up the learning rate, so we learn task 2 faster - but we shall also forget faster.
• Slow down the learning rate, so we forget task 1 slower - but we shall also learn task 2 less.

Similarly the brain needs to strike a balance between:

• More stable synapses, less sensitive to learning.
• More plastic synapses, whiche are more unstable.

For example, children have more plastic brains, and both learn and forget faster than adults. How can we solve this dilemma?

• Two complementary Learning Systems
• Neurogenesis
• Metaplasticity

Two Complementary Learning Systems As evidenced by experiments in rats, memories stored in the Hippocampus are replayed in the same order during sleep for consolidation.

Neurogenesis The process of growth of new neurons. It is known to happen during development in small children at a high rate. Adult neurogenesis:

• Slow rates.
• Limited to certain areas of the brain.

The Hippocampus is one of the areas with adult neurogenesis. We could guess new neurons to be plastic, and older neurons to be more stable. Research on the role of neurogenesis in the formation of new memories is inconclusive so far.

Metaplasticity

We usually talk about synaptic strength, or "weight", which is changed by plasticity. But previous activity could also change how easily a synapse undergoes plasticity. There are mechanisms to regulate the stability or plasticity of the single synapse, acting on a longer timescale. This is called metaplasticity.