Notes

Meta-Learning in the Brain Over the past 20 years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine "stamps in" associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. A recent study introduces a new theory, where the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deal with a wider range of observations.

In the picture we have a Meta-RL architecture across episodes to learn efficiently within an episode. (a) Agent architecture. The prefrontal network (PFN), including sectors of the basal ganglia and the thalamus that connects directly with PFC, is modeled as a recurrent neural network, with synaptic weights adjusted through an RL algorithm driven by dopamine (DA); o is perceptual input, a is action, r is reward, v is state value, t is time-step and $\sigma$ is RPE. The central box denotes a single, fully connected set of LSTM units. (b) A more detailed schematic of the neural network implementation used in the stimulations.

Meta-Learning via Neuromodulation Neuromodulators play an important role in meta-learning in the brain. Some of the key modulators are listed below. Neuromodulatory systems can be seen to mediate the global signals that regulate the distributed learning mechanisms in the brain. Based on the review of experimental data and theoretical models, some key modulators are described below:

• Dopamine is proposed to act as a "global learning" signal, critical to prediction of rewards and action reinforcement.
• Serotorin is proposed to control the balance between short and long term reward prediction, essentially by variably "discounting" expected future reward sums that may require too much expenditure to achieve.
• Norepinephrine is proposed to facilitate "wide exploration" by stochastic action selection (control exploration vs. exploitation).
• Acetylcholine is proposed to facilitate the balance between memory storage and memory renewal, finding an optimal balance between stability and effectiveness of learning algorithms for the specific environmental task.

The paper "Reinforcement Learning, Fast and Slow" presents a framework for understanding the difference between two types of reinforcement learning algorithms: "fast" RL and "slow" RL. Fast RL algorithms, such as Q-learning, learn quickly but are prone to overfitting and instability. Slow RL algorithms, such as Policy Gradient methods, learn more slowly but are more stable and less prone to overfitting. The paper argues that a combination of fast and slow RL can lead to better performance in a variety of tasks. Additionally, the paper also suggest that human learning can be understood in terms of these two types of RL, with fast RL corresponding to trial-and-error learning and slow RL corresponding to more deliberate, goal-directed learning.

The Harlow experiment is a study conducted by psychologist Harry Harlow in the 1950s, which aimed to investigate the role of learning in the development of complex problem-solving abilities. The experiment used rhesus monkeys as subjects, and it consisted of two parts. In the first part, the monkeys were presented with a series of problems, such as reaching through a hole in a barrier to retrieve food. The monkeys were allowed to explore the problems and find solutions through trial and error. In the second part of the experiment, the monkeys were presented with a new set of problems that were more difficult than the ones they had encountered before. The monkeys were able to use the knowledge and skills they had acquired during the first part of the experiment to solve the new problems more quickly and effectively. This experiment demonstrated that the experience of solving problems through trial and error could lead to the development of problem-solving skills and strategies, which can be applied to new and more complex problems. This experiment was important in supporting the idea that learning to learn is possible, and that this type of learning can be achieved through experience and exposure to different challenges.

The concept of "Bio-plausible Modulatory Networks" is a method that attempts to mimic the way the brain continually learns. This approach is based on the idea that the brain uses a combination of different neural networks, each with a specific function, to process and learn from information. These networks work together and communicate with each other through modulatory signals, which can adjust the activity of different networks depending on the task or context. In this approach, the continual learning model is composed of several networks, each specialized in a specific task, and these networks are modulated by a central controller network. The central controller network is responsible for adapting the activity of the specialized networks depending on the task to be solved, and this allows the model to continue to learn new tasks without forgetting the previous ones.