Learning about the distribution of rewards through dopamine and the basal ganglia pathways

Thesis

The basal ganglia pathways and their dopaminergic innervation are essential for reinforcement learning and are thought to play an important role in value-based decision making. Yet, it is not fully understood how these components interact and which computations they perform. In this thesis, we aim to improve this understanding, building on the Actor learning Uncertainty (AU) hypothesis of Mikhael and Bogacz (2016). According to this hypothesis, the basal ganglia might learn statistics (averages and uncertainties) of reward distributions and use them to inform choices, with dopamine acting as a teaching signal as well as a modulator.

First, we reapply the AU model, shifting the focus from reward uncertainty to tasks in which subjects need to work for rewards. We test whether the model can learn the payoffs and costs of actions and whether it can reproduce the effects of dopaminergic medication on the willingness to work for food. Second, we develop a new variant of the AU model—the scaled prediction error model—to understand how tracking uncertainty can improve learning. We investigate the new model’s performance, biological plausibility and consistency with behavioural data. Third, we test the AU model empirically, using behaviour in a learning task. The model predicts that there should be an association be- tween unexpected rewards and subsequent risk-seeking. We test that prediction, controlling for alternative explanations using trial-by-trial modelling.

We find that the AU model can learn the payoffs and costs of actions and repro- duce related behavioural effects. We further find that tracking uncertainty can improve learning if the estimate of uncertainty is used to scale reward prediction errors. This also makes the model consistent with the empirical phenomenon of dopaminergic range adaptation. Finally, we confirm the AU model’s prediction: unexpected rewards are associated with risk-seeking. Our results consolidate and extend our theoretical understanding of the role that dopamine and the basal ganglia play in learning about the distribution of rewards.

Authors: Möller, M

• DOI: 10.5287/ora-debognkee
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: reinforcement learning; dopamine; basal ganglia
• Subjects: Computational neuroscience