Reinforcement Learning Approaches to Understanding Human Decision-Making in Depression and Anxiety

Abstract

Decision-making has long been a topic studied by neuroscientists and psychologists for its centrality to human survival, and has begun to be studied by computer scientists using various normative frameworks that allow for the analysis of latent parameters involved in learning and choice processes. Reinforcement learning (RL) is one of such frameworks. Recently, computational psychiatry has begun to use RL meth- ods, allowing researchers to better understand the causes of specific psychopathologic symptoms. Using behavioral data from a norming study where subjects were asked to complete a modified risk-sensitivity task (n = 200), we evaluated models that captured subject behavior, retrieved the best-fit parameters, and regressed the pa- rameters against subject responses to the QIDS and GAD-7. We found that models with decaying learning rates modulated by static coefficients, and those that separated reward and punishment learning rates, best fit the data. We also found that subjects who score high overall on the GAD-7 have lower learning rates on average, and that subjects scoring high on the QIDS generally have a higher negative than positive learning rate (corresponding to risk-aversion), as well as a lower positive learning rate overall. There exist persistent correlations between psychomotor slowing and a higher negative than positive learning rate. Correlations become more significant when solely run on participants with scores corresponding to moderate anxiety and depression on the GAD-7 and QIDS severely, rather than on the overall population (which included subjects with low to zero overall score). We end by examining the limitations of using solely RL based, model-free approaches to understanding decision-making in people with symptoms of depression and anxiety, the validity of the methodology, and the ethics of using computational models of decision-making to understand the self.