Notes

What is Reinforcement Learning? Reinforcement Learning fuses ideas from neuroscience and AI. The model describes how an agent can interact with an environment and in that environment learn to improve its actions when it comes to gathering a targeted reward.

What makes reinforcement learning different from other machine learning paradigms?

• There is only a reward/supervision signal after each action.
• Feedback is often delayed and not instantaneous.
• Time needs to be taken into account.
• The agent's actions affect the subsequent data it receives.

Dopamine: Reward Prediction Error Papers: "Predictive Reward Signal of Dopamine Neurons"

From Schultz (89): "Dopamine neurons are activated by rewarding events that are better than predicted, remain uninfluenced by events that are as good as predicted, and are depressed by events that are worse than predicted. Most dopamine neurons show phasic activations [...] reward-predicting [...] However, only few phasic activations follow aversive (causing avoidance of a thing) stimuli. By signalling rewards according to a prediction error, dopamine responses have the formal characteristics of a teaching signal postulated by reinforcement learning theories."

If the neocortex mostly performs unsupervised learning why does the VTA strongly project to almost all cortical areas and what is the effect of DA on a cortical neuron?

• Ventral structure does not project to dorsal (where we assume RL is happening).
• Gated reinforcement learning: we only want to learn relevant information.
• Dopamine is connected to plasticity because it is very sensitive to new/unseen data.

The figure above pictures an animal experiment: Dopamine neurons report rewards according to an error in reward prediction. Top: drop of liquid (reward) occurs although no reward is predicted at this time. Middle: conditioned stimulus predicts a reward, and the reward occurs according to the prediction, hence no error in the prediction of reward. Bottom: conditioned stimulus predicts a reward, but the reward fails to occur because of lack of reaction by the animal. (CS = Conditioned Stimulus; R = Primary Reward).

The predicted reward is further modified by other factors:

• Timing of Reward: Across species, it is clear that signals related to prediction errors are modulated by cues that predict delayed reward. Animals prefer an immediate reward over a delayed reward even when the delayed reward is more economically valuable in the long run. Paper: "Impact of Size and Delay on Neural Activity in the Rat Limbic Corticostriatal System".
• Adaptation to a New Situation: We found that midbrain dopamine neurons rapidly adapted to the information provided by reward-predicting stimuli. Responses shifted to the expected reward value, and the gain adjusted to the variance of reward value. Paper: "Adaptive Coding of Reward Value by Dopamine Neurons".

Models of Learning Reward Prediction Even though these models here are called predicting models, we are looking at update rules which means the system changes over time -- it learns. We might connect one of these learning rules to a MDP or RL to find an optimal behavior function for our agent.