Emotion and decision-making: affect-driven belief systems in anxiety and depression

Emotion

and

decision-making:

affect-driven

belief

systems

in

anxiety

and

depression

Martin

P.

Paulus

and

Angela

J.

Yu

1_{DepartmentofPsychiatry,UniversityofCaliforniaSanDiego,LaJolla,CA92037,USA}

2_{PsychiatryService,VASanDiegoHealthcareSystem,LaJolla,CA92161,USA}

3

DepartmentofCognitiveScience,MC0515,UniversityofCalifornia,SanDiego,LaJolla,CA92093,USA

Emotion processing anddecision-making are integral aspectsofdailylife.However,ourunderstandingofthe interactionbetweentheseconstructsislimited.Inthis review,wesummarizetheoreticalapproachesthatlink emotion and decision-making, and focus on research with anxious or depressed individuals to show how emotionscaninterferewithdecision-making.We inte-grate theemotional frameworkbasedonvalenceand arousalwithaBayesianapproachtodecision-makingin termsofprobabilityandvalueprocessing.Wediscuss howstudiesofindividualswithemotionaldysfunctions provide evidence that alterations of decision-making canbeviewedintermsofalteredprobabilityandvalue computation.Wearguethattheprobabilistic represen-tationofbeliefstatesinthecontextofpartially observ-able Markov decision processes provides a useful approachtoexaminealterationsinprobabilityand val-ue representation in individuals with anxiety and de-pression, andoutline the broader implicationsofthis approach.

Overview

Emotionsareanintegralpartofaperson’sinternalstate and, thus, have profound influences on the choices one makes; yet, our understandingof howemotions interact withdecision-making(seeGlossary)issurprisingly incom-plete[1].Decision-makingintegrallydependsonthe com-putation of the value of available options [2], which, in turn,are afunctionofthe environmentandtheinternal state oftheindividual[3].Recentstudieshaveexamined how choices are computed in dynamic environments in whichthevalue oftheoptionschanges asaconsequence of the actions selected (see [4], fora review). Thus,how emotioninteractswiththeselectionofanoptionbasedon value isfurthercomplicatedby thefactthatvaluationis highly dynamic and can be a function of the number of availableoptions[5],theindividual’scognitivestate[6,7], effort[8],time[9],informationabouttheoptions[10],and thepresenceofother individuals[11].

Recently, there have been a number of attempts to determine and integrate the influence of emotions on decision-making(see[12],forareview).Moreover, signifi-cant advances have been made in understanding the

behavioral[2],neural[13–15],andcomputational[16–18] basisofreward-relateddecision-making.Inthelightofthis recentwork,theintegrationofemotionanddecision-making canbeconceptualizedasadynamic,iterativeprocessaimed to help the individual adapt to his or her environment, taking into account the internal state of the individual, thecharacteristicsanddeterminantsofthevaluation pro-cess,andthecharacteristicsoftheenvironment.

Inthisreview,wenarrowourfocusonexistingstudiesof decision-makinginpopulationswith emotionalproblems, butbroadenthescopetoincorporaterecentinsightsfrom Bayesian approaches to decision-making. The focus on clinicalpopulations provides insight intohow emotional

Glossary

Bayes’rule:essentially,Bayes’ruleexplainshowexistingbeliefschangeinthe lightofnewevidence.Theupdatedpredictionofaneventgivenanobservation istheproductoftheprobabilityoftheeventandtheprobabilitythatonemakes theobservationiftheeventoccurs.Forexample,thechancethatitwillrain tomorrowgivencloudstodayisthechanceofraintimesthechancethatclouds occurthedaybeforeitrains.

Bayesian statisticalmodel:atheoreticalapproachwherebypriorprobability distributions,whichquantifybeliefs,aremodifiedbylikelihoodfunctions,which resultfromobservations,toformposteriorprobabilitiesviatheBayes’rule[96]. Decision-making:acomplexprocessoftransformingoptionsintoactionsby makingchoicesbasedonsomemetricthatrepresentstheimportanceofthese optionstoanindividual[2].

Dual-systemtheory:atheoryofhumanreasoning,whichholdsthatindividuals engagetwodifferentsystemswhenprocessingoptionsinadecision-making situation[97]:System1,whichisconceptualizedasautomatic,unconscious, implicitandassociative,andSystem2,whichisviewedasexplicit,rule-based, rationalandanalytic.

Neuroeconomics:afieldthatstudiestheprocessesthatconnecttheassessment ofoptionstotheselectionofactions,basedontheconceptualapproaches developedineconomicsandthemechanisticframeworksusedinneuroscience

[77].

IowaGamblingTask(IGT):asimpledecision-makingtaskthatpitsoptionswith highshort-termgainsassociatedwithlong-termlossesagainstoptionswithlow short-termgainsbuthigherlonger-termgains.Thetaskisthoughttorelyon valuecomputationintheventromedialprefrontalcortex[98].

PartiallyObservableMarkovDecisionProcess(POMDP):anagent-based deci-sionprocessinwhichtheprecisecurrentstateisunknownand,therefore,a probability distributionover theset ofpossible states ismaintained.The probabilitydistributionismodifiedaccordingtoBayes’rulebasedonaset ofobservations[90].

Prospecttheory:aformaldescriptionofhowindividualstransformprobabilities intoweightsandlossesorgainsintovaluestoestimatetheutilityofanoptionin adecision-makingsituation[99].

Reinforcementlearning:theprocessesofbehavioralchangethatunderliehow anindividualdecidesinanenvironmentsoastomaximizerewards[79]. Utility:thedegreeofpreferencethatanindividualassignstoanoptionina decision-makingsituation[100].

dysfunctioncaninterferewithdecision-making.The adop-tionofaBayesianperspectiveontheinteractionbetween emotionanddecision-makingextendsapreviouslyproposed frameworkofunderstandingdecision-makingasa homeo-staticprocess[3].Morespecifically,weproposethat affect-drivenbeliefsystemsprofoundlyaffectthetransformation ofactionsintochoices,aswellasthemodificationoffuture expectations. Thismodulationby affect-driven belief sys-temscanbecastintoaBayesianlearningframework,that is,aniterativeprobabilisticbelief-updatingsystem,andcan beusedtoprovideaquantitativeaswellasheuristicbasis for measuring and explaining dysfunctions of decision-makinginindividualswithaffectivedisorders.

Decision-makingandemotion

Decision-makingisaprocessthatunfoldsovertime.This temporalstructurecanbeusedtoidentifythreecomponent processes. Specifically, choosing among options initially involves the process of assessing the available options. This is followed by the selection of an option based on thevaluethathasbeenassociatedwiththeoption.Lastly, theoutcomeassociatedwiththeselectedactionis evaluat-ed and incorporated into existing knowledge for subse-quent decisions. The influences of emotions on these specificcomponentprocesseshavebeguntobeconsidered onlyrecently.Inpart,thismaybebecauseofthedifficulty toincorporateemotionsintocomputationalmodels,onthe onehand,whileretainingaconnectiontotherich litera-tureofthephenomenology[19],physiology[20],and psy-chologyofemotion[21],ontheother.

Traditionally,emotionshavebeenconceptualizedalong valence and arousal dimensions (for a review, see [21]). However,othershavecategorizedemotionsalongdifferent facialexpressions[22]orappraisaldimensions[23].These approachesacknowledgethatemotionshaveboth quantita-tive dimensions(e.g., degrees offeeling goodorbad) and qualitativedimensions(e.g.,feelingsadorangry).Withina dual-system framework of reasoning [24], emotions are thoughttoaffecttherelativeextenttowhichanintuitive, heuristic System1 and an analytic, deliberate System 2 contributetodecision-making.Moreover,basedona tempo-ralperspectiveofdelineatingdecision-makinginto compo-nentprocesses,emotionsalsocontributetothemodulation ofassessment,selection,andoutcomeevaluationofoptions. For example, a sad mood may result from an undesired outcome,butitcanalsoleadtoincreasedsalienceofnegative attributesofoptionsinfutureassessments.

TheclassicalworkofTverskyandKahneman[25] estab-lishedthatbothvalueandtheprobabilityofanoutcomeare representedasnonlinearfunctions,suchthathighervalues havedecreasingmarginalgains,lossesarevaluedgreater than gains, low probabilities areoverweighted, and high probabilitiesareunderweighted.Severalinvestigatorshave developedmodelsthatconceptualizetheeffectofemotions asmodulatingthesenonlinearvalueandprobability func-tions[26–28]withinadual-systemframework.Forexample, Mukherjee[29]developedamodeltoaccountforindividual differences and emotional influences on decision-making based on experimental findings by Hsee [6] and others [30].Intheseexperiments,individualswhowereinstructed toengage affectiveprocesseswhenprocessingvaluewere

lesssensitivetomagnitudechangesoftheavailableoptions andshowedgreaterdistortionsoftheS-shapedprobability weightfunction[7].Inthecomputationalmodeldeveloped byMukherjee[29],thevalueofanoptionisobtainedfroma mixture of System 1 and System 2 processes, which is parameterizedbyamixturecoefficientquantifyingthe de-greetowhichaffectinfluencesdecision-making.This mix-ture parameter can capture individual differences in decision-making dispositions, context-dependent outcome processing,andaffectiveconstructionofthe decision-mak-ingsituation.

Otherinvestigatorshavedepartedmoreradicallyfrom the traditional utility model to explain differences in choicesas theyrelatetoemotionprocessing.Specifically, Kusev[31]hasproposedthatindividualsdonotcalculate utilities explicitly.Instead, people construct preferences basedontheirexperiences[32].Forexample,people over-weightsmall,medium-sized,andmoderatelylarge proba-bilitiesandtheyalsoexaggeraterisks.However,neitherof thesefindingsisanticipatedbyprospecttheoryor experi-ence-baseddecision research.Thissuggeststhat people’s experiencesofeventsleakintodecisions,evenwhenrisk informationisexplicitlyprovided[33].Asaconsequence, choices depend strongly on context, the type of options, implicitly the degree of affect associated with these options,andthenatureofthepresentationoftheavailable optionsinthedecision-makingsituation.Similarly,Vlaev andcolleagues[34,35]havesuggestedthatindividualsdo nothaveacommonrepresentationofvalueacrossdifferent domains.Forexample,peoplewilloffertopaymoremoney when stakesare highinapairingoflow versus medium painthanwhenstakesarelow,whichisthoughttobedue tothefactthatindividualsarenotabletocompare quali-tatively different options or outcomes on a single value dimension. Althoughtheseinvestigatorsdo notdeny the evidenceprovidedbydescriptivetheoriesofutilities asso-ciated with options in decision-making situations, they challenge the universality of the shape of the observed valueandprobabilityweightfunctionsandproposeamuch more dynamic formalism to explain behavioral observa-tionsindecision-makingsituations.

Theexperimentalandtheoreticalaccountsofthe influ-ence of emotion on decision-making we have discussed reveal that: (i) emotions appear to influence the value and weight computation of available options; (ii) these computations are dynamically adjusted based on the environmentandtheindividual’sinternalstate; (iii)itis notyetclearwhethertherearevalence-orarousal-specific influencesonthisdynamiciterativeprocess.

Evidencefordisruptionofdecision-makinginanxiety anddepression

Studies of decision-making in clinicalpopulations are of immensevalue,becausetheycanhelptoestablish brain-behaviorrelationships,clarifythenatureofdysfunctional process(es) in a disorder group, and point toward the developmentofpotentialtreatmentsfordisorders. Howev-er, whenstudying decision-makingin individualswith a particular disorder, it is important to note that the observeddifferencesbetweenthepsychiatrictarget popu-lationandthecomparisongrouparetypicallytheresultof

acomplexsetoffactorsthatincludepre-disease character-istics, disease-related (e.g., treatments) and disease-unrelated(e.g.,lifeevents)developments,andthe particu-larstatetheindividualfindshim-/herselfinatthetimeof testing. Moreover, most studies have utilized a limited numberofdecision-makingparadigmstoexamineanxiety andmoodrelatedeffectsondecision-making.Giventhese complexities andlimitations, it should notbe surprising thatitcanbedifficulttolinkaparticularmoodoranxiety statetodecision-makingdysfunctions.

Alteredbeliefsystems[36–38](forareview,see[39])play acriticalroleintheconceptualizationofbothanxietyand depression.Inparticular,individualswithanxietydisorders showanincreasedbiastowardsthreat-relatedcontent[40] andanintoleranceofuncertainty[41].Incomparison, de-pressedindividualsshowreducedresponsivenesstoreward [42]andanincreasednegativeevaluationofself[43].There is some consensus that anxiety makes individuals more sensitive, and thus more aversive, tooptions with large negativeconsequences[44].Thisheightenedsensitivityto options with large negative consequences can hurt them whenanoptionoccasionallyassociatedwithahighly nega-tiveoutcomeisactuallyonaveragethebestoption,butmay help themontasks with intermittent largepunishments thatsignaltheneedforrepresentationaloverhaul,suchas theIowa GamblingTask(IGT).In particular,individuals withgeneralizedanxietydisorderlearntoavoiddecisions with highimmediategainbuthighlong-termloss signifi-cantlyfaster thancomparisonsubjects[45].Onthis deci-sion-making task, better performance is associated with poorer ability to regulate emotions [46], which may be duetoamoreprolongedvisceralresponse[47].However, othergroupshavefoundimpairedperformanceontheIGT asafunctionofbothhighandlowtraitanxiety[48,49].In particular, high trait anxious individuals generate an increasedanticipatoryphysiological responsepriortolow reward, lowpunishment, or advantageous choices. These investigatorshavearguedthatanxietymayresultin dis-traction from task-relevant processing, inefficient proces-singofrelevantvs irrelevantcues,and interferencefrom increased verbal processing (rumination).Post-traumatic stress disorder subjects do not show an impairment of learning the contingencies ofthe IGT, butshow reduced activationofreward-relatedareas[50],whichisconsistent withthefindingthatstressinterfereswiththeacquisitionof advantageousresponseselectiononthistask[51].Patients with obsessivecompulsivedisordershowdecision-making dysfunctionsthathavebeenattributedtotheirinabilityto appropriately process emotions [52]. On the whole, the effectsofanxietydisorderondecision-makingarecomplex andmayberelatedtoother,moreanxiety-specific dysfunc-tions,suchasavoidanceofthreat-relevantinformation[53], interferencebynegativedistracters[54],orageneraldeficit ofinhibitoryprocessinginthepresenceoflimited availabil-ityofcontrolledprocessingresources[55].

Anhedonia [56,57],that is, the inabilitytoexperience pleasure, and deficits in reward-related processing [58] have been considered to bethe critical components that contributetodysfunctionsindecision-makingindepressed individuals. Consequently, itis not surprising that indi-viduals with depression showreduced responsivenessto

reward[42].OntheIGT,individualswithmajordepressive disorder show poorer performance: fewer advantageous card selections [59], fewer selections of risky card decks [60], and less shifting [61]. Moreover, these individuals select more cards from decks with high frequency, low-magnitude punishment contingencies [61]. Surprisingly, otherresearchhasshowedthatacutelydepressed individ-uals make better choices relative to controls or those recoveringfromdepression[62],whichisconsistentwith thefindingthatdepressiveindividualslearntoavoidrisky responsesfasterthancontrolparticipants[60].Ingeneral, depressedindividualsappeartoexperienceanincreasein decisionalconflictinanumberofdifferentdecision-making situations[63,64],attenuatedprocessingofcounterfactual outcomes [65], anda prolonged attenuation of temporal discountingofrewards[66].Inaddition,individualswith depressive symptoms fail todevelop a response bias to-wardsrewardedstimuli[67–70],intasksinwhichwhich subjects must categorize a briefly presented stimulus as belonging to category A or B. In comparison, manic, depressed, and euthymic bipolar subjects select signifi-cantlymorecardsfromriskydecksandpreferdecksthat yield infrequent penalties over those yielding frequent penalties[71].Thepercentageoftrialsonwhichsubjects choose the more likely of two possible outcomes is also significantlyimpairedin depressedbipolarpatients [72]. Othershavereportedthatbipolarmanicindividualsshow impaired[73]orerratic[74]decision-makingability,which has been termed suboptimal [75] decision-making. Depressedandmanicindividualsshowslowerdeliberation times,afailuretoaccumulateasmanypointsascontrols, and suboptimal betting strategies [75]. These findings point toward both trait-dependent (i.e., disorderrelated and presumably long-term) and state-dependent (i.e., internalstateandmood-related)decision-making dysfunc-tion. This dysfunction may be due to decision-specific alterationinvalue-related,probability-related,or tempo-ralprocessingofavailableoptions.

In sum, anxiety, depression, and mood swings exert complex effectsondecision-making as measuredby per-formanceontheIGT.Specifically,increasedsensitivityto losses, attenuatedprocessing of reward, and differential selectionbasedonthehistoryofrewardsandpunishment point towards emotion-related modulation of both value andprobabilityinanxietyanddepression.Inthefollowing section,weintegratethesefindingswithacomputational approachaimedatquantifying beliefsystems.

Understandingtheinfluenceofaffect-drivenbelief systemson decision-makinginanxietyanddepression

Bayesianmodelsofdecision-makingandcontrol

In recent years, the understanding of the behavioral and neural processes underlying decision-making has benefited significantly from neuroeconomic models [2,76,77] and reinforcement learning models [13,78,79]. Theseapproacheshaveprovidedinsightsintohow individ-ualsquantifythevalueofoptions,whatbrainsystemsplay akeyroleinthisprocess,as wellashowtheunderlying neural substrates give rise to behavioral phenomena. However,onecriticalaspectofgoal-directedaction selec-tionthat does not typicallyreceive explicittreatmentin

neuroeconomic or reinforcement learning models is the subjective uncertainty individuals have about both the stateoftheworldandtheeventualconsequences associat-edwith the differentactionchoices inthat state. Uncer-tainty arises from a multitude of sources, for example, noisysensing,imperfectmotorcontrol,neuronal commu-nicationerrors,andintrinsicstochasticityand non-statio-narityintheenvironment.Moreover,selectionandaction mayleadtocomplexorindirectlyobservablechangesinthe stateoftheworldorthatofanindividualintheworld,in addition toany directreward/punishmentconsequences. There are two ways in which uncertainty complicates decision-making: oneis therepresentation and propaga-tionofimpreciseinformation,theotherthetranslationof thatinformationintoagoal-directedcourseofactionviaa decisionpolicy.

Inthepastdecade,significantprogresshasbeenmade inunderstandinghowthe brainrepresentsuncertain in-formation with the help of Bayesian statistical models, whichofferamathematicallypreciselanguagefor describ-ingprobabilistic knowledge, as wellas normative proce-dures for computation based on such knowledge. In particular,Bayesianmodelsprovideawayofformalizing howbeliefsarelinkedtoobservationstoarriveat estima-tionsofthestatusoftheworld,whichguidethe decision-makingprocess.Inthecontext ofBayesianmodels, emo-tionscanbeconceptualizedasmodulatingthe mathemati-cal representation of these beliefs andthe processing of observations.Asageneralizationoftheclassicalnotionof anidealobserverinpsychology[80],Bayesianmodelshave helped to demonstrate that humans perform close to Bayesianideal (optimum) ina numberofsimple experi-mental tasks (e.g., multimodal cue integration [81–83], rewardlearning[84],attentionalselection[85],andmotor adaptation[86])andshedlightontheneural representa-tionofvalue anduncertainty[84,87].

However, even more revealing than when the brain performsoptimallyiswhenitdoesnot.Idiosyncratic fail-ures ofthebraintorepresent/process probabilistic infor-mation,bymakingcertainimplicitstatisticalassumptions (biases),canrevealimportantneuraldesignprinciplesthat underliebothnormalbehaviorandcognitivedysfunctions. Forexample,ithasbeenshownthattheapparently irra-tional tendency for normal subjects to extract transient stimulus patterns in a truly random stimulus sequence maybeduetotheinappropriate,reflexiveengagementof neural mechanisms necessary for adapting to changing environmentalcontingencies[88].

Oneapproachtolinkconceptuallytheemotionalstateof theindividualtohisorherdecision-makingistoassume thatfeelingsaffectthewayprobabilitiesofgainsorlosses aretransformedtoweightsandvalues(seeFigure1andits description for an example). In the context of Bayesian statisticalmodels,thismayimplyamodification ofprior beliefs about world states (prior probabilities), the rela-tionship between hidden world states and sensory/rein-forcementobservations(likelihood),ortheevolutionofthe external environment due to one’s actions (transition matrix). Onepromising framework forformalizing these conceptsispartiallyobservableMarkovdecisionprocesses (POMDPs, see Figure 2 for a schematic description)

[89,90].Inthisframework,individualsdonothaveprecise knowledge of the state in which they are or the conse-quences of their action choices; instead they maintain (possiblyimplicitly)probabilisticbeliefsofbeingina par-ticular state or incurring certain outcomes with their choices,whichdependonpreviousexperiences.The prob-abilitydistributionscanbeusedtoestimatetheexpected rewardorcostassociatedwithanactionandcan,thus,be usedtoselectthebestaction.Observations(oroutcomes) resulting from the selected action, in turn, update the probability distributionover statesviaaso-called belief-state estimator. Belief updating occurs via Bayes’ rule. Actionselectionoccursaccordingtosomevariantof Bell-man’sEquations[91].Theoptimaldecisionisdetermined by thedistribution ofthecurrent belief states,which,in turn,isdeterminedbytheprobabilityoftransitioningfrom onestatetoanotherviaaparticularactionandthe proba-bilityofobservinganoutcomewhenselectingaparticular action.Inaddition,theoptimaldecisionisdeterminedby thenatureofthecostsandrewards,thedegreetowhichthe decision-maker looks ahead and estimates costs and rewards of subsequent actions, and the probability of experiencing a cost or reward when transitioning from onestatetoanother.Arichcomputational[89,92],applied [93], and even neural-systems level [94] literature is emergingbasedonthisnormativeframework.

Thistheoreticalapproachtothesubstratesof informa-tionrepresentationandgoal-directeddecision-makingcan be fruitfully applied to understanding the influence of emotions on decision-making. The experience of an out-come and, in particular, the differences between the expected and observed outcome provide an opportunity to improve one’s beliefs about the consequences (value) oftheavailableoptionsandadoptabetterdecisionpolicy inthefuture.HuysandDayan[95]havedevelopedlayered notions of control in order to formalize dysfunctions of theseprocessesinthecontextofdecision-makingfor indi-viduals with anxiety or depression. In this framework, three layers of control are constructed to explain the relationship between emotional dysfunction and deci-sion-making:(i)beliefsaboutthecontrolofthereliability ofoutcomes(i.e.,howlikely aspecificoutcome is),which canbequantifiedbytheaction-dependentoutcome entro-py; (ii)thedegreetowhichanactionisassociatedwitha specificoutcome,whichcanbequantifiedbyassessingthe precisionoftheaction-outcomerelationshipintransition probabilities or the entropy of the transition probability matrix;(iii)thedegreetowhichdesirableoutcomescanbe achievedviareinforcementlearningprocesses,whichisa functionoftheentropyandvalueofthemarginaloutcome distribution.HuysandDayansuggestthatcontrollability of reinforcement is a critical factor of decision-making dysfunctionindepression.

Aconceptualframework fortheinfluenceof

affect-drivenbeliefsystemsondecision-makinginanxietyand

depression

Within the context of Bayesian inference, POMDPs (Figure2)provideaparticularlyusefulformalframework toexaminealterationsindecisionprocessesduetoaltered affect-driven belief systems. Specifically, the belief-state

Choice: Op!on A :

10% chance of gaining $1800 or 90% chance of losing $200

Op!on B :

40% chance of gaining $450 or 60% chance of losing $300

Subjec

!ve

value

Objec!ve value

Weight

Probability Gamble A Gamble B

Subjec !ve u! lity Subjec !ve value Objec!ve value Weight

Probability Gamble A Gamble B

Subjec !ve u! lity Subjec !ve value Objec!ve value Weight

Probability Gamble A Gamble B

Subjec !ve u !lity Subjec !ve value Objec!ve value Weight

Probability Gamble A Gamble B

Subjec

!ve u

!lity

Preference reversal due to "affec!vely derived" change in probability weights.

Preference reversal due to "affec!vely derived" change in value func!on.

"Affec"vely derived" subjec"ve value Key: Subjec"ve value "Affec"vely derived" weight Weight No affect Affect Key: Key: "Affec"vely derived" subjec"ve value Key: Subjec"ve value "Affec"vely derived" weight Weight No affect Affect Key: Key: "Affec"vely derived" subjec"ve value Key: Subjec"ve value "Affec"vely derived" weight Weight No affect Affect Key: Key: "Affec"vely derived" subjec"ve value Key: Subjec"ve value "Affec"vely derived" weight Weight No affect Affect Key: Key:

TRENDS in Cognitive Sciences

Figure1.Thisfigureshowsasimplegambleconsistingoftwooptions(AandB)withprobableoutcomesandshowstheeffectofmodulatingthevaluefunction(left column),theprobabilityweightingfunction(middlecolumn)andtheresultingutilityofthetwo availableoptions(rightcolumn).Specifically,theobjectivevalue istransformedaccordingto[SubjectiveValue]=k[ObjectiveValue]b_{andtheprobabilityistransformedaccordingto[SubjectiveWeight]=[Probability]}1-a_{/([Probability]}1-a₊

(1-[Probability])1-a_{),inaccordancewithproposalsbyMukherjee[29]andHsee[6].Thedarkgreylinessignifyalargerdistortionduetopresumedaffectivelydriven}

modulationofobjectivevalueorprobability.ThebargraphsindicatetheoverallutilityofoptionAorB;arelativelylargersubjectiveutilityofAoverBisassumedtoresultin apreferenceforAoverB.Inthefirsttworows,examplesaregivensuchthatalterationsoftheparameterdeterminingthesubjectiveweight,a,canreverse(firstline)ornot reverse(secondline)thepreferenceofthegamble.Inthethirdandfourthrowasimilarexampleisprovidedforalterationsofthevalueparameter,b,thatreversesordoes notreversethepreferenceofthegamble.Takentogether,thesesimplecalculationsshowthatalterationsofprobabilityandvalueinaccordancewithempiricaland theoreticalapproachestounderstandingtheeffectofemotionondecision-makingcanhavesignificantpreferencereversaleffects.Weproposethattheinfluenceof emotion,ingeneral,andofanxietyordepression,inparticular,isthatofchangingthenonlinearityoftheweightingandthevaluefunction,suchthatpreferencereversals occur.Thesereversalshelptoexplainperformancedifferencesonrisk-relateddecision-makingtasks.

estimatorrepresentsprocessesthatdeterminean individ-ual’s expectations of the current state. The probabilistic formalism cansupport the generalhypothesis that indi-vidualswithanxietyanddepressionrepresentinformation andactionoutcomesinsuchawaythat probabilitiesare misrepresentedandoutcome valuesare biased.Thus,in both disorders, transformation of probabilitiesinto deci-sionweightsmayresultingreaterweightingoflow prob-abilityevents,assuggestedbyMukherjee[29].Moreover, the altered valuation process that leads to attenuated sensitivitytomagnitudechangesofthevalueofavailable options [6] can contribute to inappropriate valuation of expectedoutcomes.Asaconsequence,alteredreward per-ceptionaffects thebelief-state estimatorprocesses. More importantly, the process of updating belief states via Bayes’ rule may also be affected,due to suboptimalities inprobabilisticinferenceandvaluefunctioncomputation/ comparison, and/or the representation of priors or the likelihoodfunction.Consequently,ananxiousordepressed individual mayselectoptions based onanaltered repre-sentation of the current belief states, which can be the

result of biased transition probabilities due to altered processingofcostsandrewards.Forexample,low proba-bilitieswithhighthreatorcostvaluemaybe over-repre-sented in the decision-making processes of an anxious individual,whichmayaidperformanceinasituationwith infrequentbuthighlycostlyoutcomessuchastheIGT.

In additiontoupdatingthe currentbelief states, indi-vidualsalsohavetodeterminehowfaraheadtheyneedto look when computing the optimal policy in a decision-making situation. In anxious individuals, intolerance of uncertaintymaylead toavoidanceoflooking ahead[53], therefore rushing into action with suboptimal choices, which isalso consistent with an increased susceptibility tointerferencebynegativedistracters [54].In depressed individuals,theattenuatedabilitytorepresentthe differ-entialvalueofavailableoptionsmayaffect decision-mak-ing when the outcomes of options are very similar. Moreover,theseindividualsmaynotbeabletoadequately update the value structure and, therefore, show dimin-ished learning and updating of the belief systems that providethe basis formaking optimaldecisions, whichis consistent withfailure tolearna biastowardsrewarded stimuli [67–70] and with the use of suboptimal betting strategies[75].

In sum, altered probability and value processing can haveprofound effectsonanappropriatelyupdatedbelief systemoftheindividual’sstate.Beliefsystemsdrivenby miscalibratedaffectcanleadtoamisrepresentationofthe chances and potential outcomes that an individual per-ceives whenselecting an action. Forexample,the avoid-anceoftheinitiationofasocialinteractionbyananxious individualmaybeguidedbyanover-representationofthe state that leads to a rejection by the other individual. Similarly, the difficultyto selectactions associated with rewardsbyadepressedpersonmaybeduetothe attenu-atedeffectsoftheobservedrewards,whichleadsto under-valuation ofthe state associated with actionsleading to rewardingoutcomes.However,thepreciseeffectsof emo-tion on decision-making, in general, and in the case of anxietyordepression,inparticular,needfurtherempirical andtheoreticalstudy.

Concludingremarks

Theintegrationofemotionprocessinganddecision-making is challenging on a number of different levels (Box 1). Conceptually, emotionshave been definedalong arousal and affective valence dimensions, which are difficult to integratewithinvalue andprobabilityframeworksof de-cision-making. Phenomenologically, emotions are often viewed as highlyintrospective, whereas decision-making

Outside world Individual 1 2 Ac"ons Observa"ons rewards Belief state

es"mator Belief state

Decision-making: state-ac!on mapping 1: Bayes' rule: 2: Bellman's equa!on: P(s_t|x₁,...,x_t–1; a₁,..., a_t–1) ∝ ∑P(x_t–1|s_t–1; a_t–1)P(s_t–1|x₁,..., x_t–2; a₁,..., a_t–2)p(s_t|s_t–1; a_t–1) V(b_t) = max E[r_t|b_t; a_t = k)] + a∑V(b_t+1)P(b_t+1|bt, at = k st–1 b_t+1 k

TRENDS in Cognitive Sciences Figure2.AschematicrepresentationofapartiallyobservableMarkovdecision process(POMDP)[89,90].APOMDPconsistsofabeliefstate,whichsummarizes previousexperiences,isrepresentedinaprobabilisticframework,andisupdated bythebelief-stateestimator.Theupdatingoccursbetweenthelastaction,the currentobservation,andthepreviousbeliefstatebasedonBayes’rule.Bayes’rule underliesbeliefupdating,wherebythenewbeliefstate(distributionoverstatesthe agentisinattimet),bt,isinformed,fromlefttoright,by(i)thelikelihoodof

makingthelatestobservationxt!1,(ii)beliefsaboutthestateatthelasttimepoint,

and(iii)thetransitionprobabilitiesfromstatetostateasafunctionofthelast action,allintegratedoveruncertaintyaboutthelaststate.Decision-makingoccurs asaconsequence ofadecisionpolicythatmapsthecurrentbelief stateinto actions.ActionselectionisbasedonBellman’svalueiterationequation,whereby themaximalvalueobtainableattimet,V(bt),isachievedbychoosingtheaction

withthebestimmediateexpectedrewardrt(afunctionofxt,firstterm)plusthe

expected maximal value obtainable at the next time point summed over uncertaintyaboutthenextbeliefstate,possiblydiscountedbyamultiplicative factora"1(secondterm).Emotionscanaffectthisprocessintwoways.First,the observedrewardsarehypothesizedtobetransformedintovaluesbasedonthe subjective value function as shown in Figure 1. Second, probabilities are hypothesized to be transformed into weights and can, therefore, affect the updatingviathebelief-stateestimator.Inotherwords,faultyupdatingbythe belief-stateestimatorduetoattenuatedvaluationorexaggeratedrepresentationof lowprobabilitiescanresultinsuboptimalestimationof thecurrentstateand thereforepoorselectionofadecisionpolicy.Forexample,greaterweightingof threat-relatedstatesinanxietymayresultinavoidanceofaction.Alternatively, attenuatedrepresentationofsubjectivevaluemayresultindiminishedlearning andupdatingofthebelief systemsthatprovidethebasisformakingoptimal decisionsindepression.

Box1.Questionsforfutureresearch

#Whatarethevalence-andarousal-specificeffectsofemotionson probabilityandvaluecomputationduringdecision-making? #Canemotion-baseddistortionsofprobabilities explain

subopti-maldecision-makingwithinaBayesianframework?

#What is theinfluence ofprocessing emotional information on subsequentdecision-making?

#Do interventionsthat target emotion processing inanxious or depressedindividualsalsoaffectprobabilityandvalue computa-tionduringdecision-making?

hasbeenquantifiedalongafewexternalvariables. Com-putationally,emotionprocessinghasbeenconceptualized in terms of associative networks,but also as a result of conditioningprocesses,whereasdecision-makinghasbeen conceptualized as a normative process. Pathologically, emotionaldisorders havebeenapproached fromthe per-spective of altered view of self, belief, and conditioning, whereas decision-making dysfunctions have been de-scribed in terms of altered preference structures. The Bayesian approachwehaveproposed inthisarticlemay helptodevelopacommonframeworkforemotion proces-sing and decision-making, by providing a quantitative approachtomeasuringtheinfluenceofemotionsonchoices (andviceversa)intermsofalteredbeliefstatesandaction consequences.

References

1Mitchell, D.G. (2011) The nexus between decision-making and emotion regulation: a review of convergent neurocognitive substrates.Behav.BrainRes.217,215–231

2Rangel,A.etal.(2008)Aframeworkforstudyingtheneurobiologyof value-baseddecision-making.Nat.Rev.Neurosci.9,545–556 3Paulus, M.P.(2007)Decision-making dysfunctionsinpsychiatry –

alteredhomeostaticprocessing?Science318,602–606

4Walton, M.E. et al. (2011) Giving credit where credit is due: orbitofrontal cortexand valuationin anuncertainworld.Ann. N. Y.Acad.Sci.1239,14–24

5Mellers,B.A.andBiagini,K.(1994)Similarityandchoice.Psychol. Rev.101,505–518

6Hsee,C.K.andRottenstreich,Y.(2004)Music,pandas,andmuggers: ontheaffectivepsychologyofvalue.J.Exp.Psychol.Gen.133,23–30 7Rottenstreich,Y.andHsee,C.K.(2001)Money,kisses,andelectric shocks:ontheaffectivepsychologyofrisk.Psychol.Sci.12,185–190 8Rudebeck, P.H. et al. (2006) Separate neural pathways process

differentdecisioncosts.Nat.Neurosci.9,1161–1168

9Maule,A.J.etal.(2000)Effectsoftime-pressureondecision-making under uncertainty: changes in affective state and information processingstrategy.ActaPsychol.(Amst.)104,283–301

10 Tversky,A.andKahneman,D.(1973)Availability:aheuristic for judgingfrequencyandprobability.Cogn.Psychol.5,207–232 11 Rilling, J.K. and Sanfey, A.G. (2011) The neuroscience of social

decision-making.Annu.Rev.Psychol.62,23–48

12 Seymour,B.andDolan,R.(2008)Emotion,decision-making,andthe amygdala.Neuron58,662–671

13 Schultz,W.etal.(1997)Aneuralsubstrateofpredictionandreward.

Science275,1593–1599

14 O’Doherty, J. et al. (2001) Abstract reward and punishment representationsin the humanorbitofrontalcortex. Nat. Neurosci.

4,95–102

15 Glascher, J. et al. (2009) Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-relateddecision-making.Cereb.Cortex19,483–495 16 Daw,N.D.etal.(2006)Corticalsubstratesforexploratorydecisionsin

humans.Nature441,876–879

17 Montague, P. and Berns, G. (2002) Neural economics and the biologicalsubstratesofvaluation.Neuron36,265

18 Montague,P.R.etal. (2006)Imagingvaluationmodels in human choice.Annu.Rev.Neurosci.29,417–448

19 James,W.(1988)TheprinciplesofPsychology,H.HoltandCompany 20 Zajonc,R.B.etal.(1989)Feelingandfacialefference:implicationsof

thevasculartheoryofemotion.Psychol.Rev.96,395–416

21 Winkielman,P.etal.(2007)Affectiveinfluenceondecisions:moving towardscoremechanisms.Rev.Gen.Psychol.11,179–192

22 Ekman,P.(1992)Aretherebasicemotions?Psychol.Rev.99,550–553 23 Lerner,J.S.andKeltner,D.(2001)Fear,anger,andrisk.J.Pers.Soc.

Psychol.81,146–159

24 Evans,J.S.(2008)Dual-processingaccountsofreasoning,judgment, andsocialcognition.Annu.Rev.Psychol.59,255–278

25 Kahneman,D.andTversky,A.(1979)Prospecttheory:ananalysisof decisionunderrisk.Econometrica47,263–291

26Loewenstein,G.(1996)Outofcontrol:Visceralinfluencesonbehavior.

Organ.Behav.Hum.Decis.Process.65,272–292

27Damasio,A.R.(1996)Thesomaticmarkerhypothesisandthepossible functionsoftheprefrontalcortex.Philos.Trans.R.Soc.Lond.B:Biol. Sci.351,1413–1420

28Mellers,B.etal.(1999)Emotion-basedchoice.J.Exp.Psychol.Gen.

128,332–345

29Mukherjee,K.(2010)Adualsystemmodelofpreferencesunderrisk.

Psychol.Rev.117,243–255

30Loewenstein,G.F.etal.(2001)Riskasfeelings.Psychol.Bull.127, 267–286

31 Kusev, P. and van Schaik, P. (2011) Preferences under risk: content-dependent behaviorand psychologicalprocessing. Front. Psychol.2,269

32Gottlieb, D.A. et al. (2007) The format in which uncertainty information is presentedaffects decision biases.Psychol. Sci. 18, 240–246

33Kusev, P. et al. (2009) Exaggerated risk: prospect theory and probabilityweightinginriskychoice.J.Exp.Psychol.Learn.Mem. Cogn.35,1487–1505

34Vlaev,I.(2011)Inconsistencyinriskpreferences:apsychophysical anomaly.Front.Psychol.2,304

35Vlaev,I.etal.(2011)Doesthebraincalculatevalue?TrendsCogn.Sci.

15,546–554

36Ellis,A.(1976)Thebiologicalbasisofhumanirrationality.J.Individ. Psychol.32,145–168

37Nelson,R.E.(1977)Irrationalbeliefsindepression.J.Consult.Clin. Psychol.45,1190–1191

38Reiss,S.etal.(1986)Anxietysensitivity,anxietyfrequencyandthe predictionoffearfulness.Behav.Res.Ther.24,1–8

39Paulus, M.P.andStein, M.B.(2010)Interoceptionin anxietyand depression.BrainStruct.Funct.214,451–463

40MacLeod,C.andMathews,A.(1988)Anxietyandtheallocationof attentiontothreat.Q.J.Exp.Psychol.A40,653–670

41Dugas,M.J.etal.(1998)Generalizedanxietydisorder:apreliminary testofaconceptualmodel.Behav.Res.Ther.36,215–226

42Elliott,R.etal.(1996)Neuropsychologicalimpairmentsinunipolar depression: the influence of perceived failure on subsequent performance.Psychol.Med.26,975–989

43Beck,A.T.(1967)Depression:Clinical,Experimental,andTheoretical Aspects,HoeberMedicalDivision,Harper&Row

44Maner,J.K.andSchmidt,N.B.(2006)Theroleofriskavoidancein anxiety.Behav.Ther.37,181–189

45Mueller, E.M. et al. (2010) Future-oriented decision-making in GeneralizedAnxietyDisorderisevidentacrossdifferentversionsof theIowaGamblingTask.J.Behav.Ther.Exp.Psychiatry41,165–171 46Werner,N.S.etal.(2009)Relationshipsbetweenaffectivestatesand

decision-making.Int.J.Psychophysiol.74,259–265

47Verkuil,B.etal.(2009)Effectsofexplicitandimplicitperseverative cognition on cardiac recovery after cognitive stress. Int. J. Psychophysiol.74,220–228

48de Visser, L. et al. (2010) Trait anxiety affects decision-making differently in healthy men and women: towards gender-specific endophenotypesofanxiety.Neuropsychologia48,1598–1606 49Miu, A.C. et al. (2008) Anxiety impairs decision-making:

psychophysiological evidence from an Iowa Gambling Task. Biol. Psychol.77,353–358

50Sailer, U. etal. (2008) Altered reward processingin the nucleus accumbens and mesial prefrontal cortex of patients with posttraumaticstressdisorder.Neuropsychologia46,2836–2844 51Preston,S.D.etal.(2007)Effectsofanticipatorystresson

decision-makinginagamblingtask.Behav.Neurosci.121,257–263 52Sachdev, P.S. and Malhi, G.S. (2005) Obsessive-compulsive

behaviour:adisorderofdecision-making.Aust.N.Z.J.Psychiatry

39,757–763

53Amir,N.etal.(1998)Automaticactivationandstrategicavoidanceof threat-relevantinformationinsocialphobia.J.Abnorm.Psychol.107, 285–290

54Vythilingam, M.et al.(2007) Biasedemotional attention in post-traumaticstress disorder:ahelpaswell asahindrance?Psychol. Med.37,1445–1455

55Wood,J.etal.(2001)Anxietyandcognitiveinhibition.Emotion1, 166–181

56 Der-Avakian, A. and Markou, A. (2012) The neurobiology of anhedonia andother reward-relateddeficits.TrendsNeurosci.35, 68–77

57 Treadway,M.T.andZald,D.H.(2011)Reconsideringanhedoniain depression: lessons from translational neuroscience. Neurosci. Biobehav.Rev.35,537–555

58 Eshel,N.andRoiser,J.P.(2010)Rewardandpunishmentprocessing indepression.Biol.Psychiatry68,118–124

59 Han, G. et al. (2012) Selective neurocognitive impairments in adolescentswithmajordepressivedisorder.J.Adolesc.35,11–20 60 Smoski,M.J.etal.(2008)Decision-makingandriskaversionamong

depressiveadults.J.Behav.Ther.Exp.Psychiatry39,567–576 61 Cella,M.etal.(2010)Impaired flexibledecision-makingin Major

DepressiveDisorder.J.Affect.Disord.124,207–210

62 vonHelversen,B.etal.(2011)Performancebenefitsofdepression: sequential decision-making in a healthy sample and a clinically depressedsample.J.Abnorm.Psychol.120,962–968

63 van Randenborgh,A. etal. (2010)Decision-making indepression: differences in decisional conflict between healthy and depressed individuals.Clin.Psychol.Psychother.17,285–298

64 Harle,K.M.etal.(2010)Theimpactofdepressiononsocialeconomic decision-making.J.Abnorm.Psychol.119,440–446

65 Chase, H.W. et al. (2010) Regret and the negative evaluation of decision outcomes in major depression. Cogn. Affect. Behav. Neurosci.10,406–413

66 Lempert, K.M. and Pizzagalli,D.A.(2010)Delay discountingand future-directed thinkingin anhedonicindividuals.J.Behav.Ther. Exp.Psychiatry41,258–264

67 Pizzagalli, D.A. et al. (2005) Frontal brain asymmetry and reward responsiveness: a source-localization study. Psychol. Sci.

16,805–813

68 Pizzagalli,D.A.etal.(2007)Increasedperceivedstressisassociated withbluntedhedoniccapacity:potentialimplicationsfordepression research.Behav.Res.Ther.45,2742–2753

69 Pizzagalli, D.A. et al. (2008) Reduced hedonic capacity in major depressive disorder: evidence from a probabilistic reward task.

J.Psychiatr.Res.43,76–87

70 Pizzagalli, D.A. (2011) Frontocingulate dysfunction in depression: towardbiomarkersoftreatmentresponse.Neuropsychopharmacology

36,183–206

71 Adida,M.etal.(2011)Trait-relateddecision-makingimpairmentin thethreephasesofbipolardisorder.Biol.Psychiatry70,357–365 72 Rubinsztein,J.S.etal.(2006)Impairedcognitionanddecision-making

inbipolardepressionbutno‘affectivebias’evident.Psychol.Med.36, 629–639

73 Adida,M.etal.(2008)Lackofinsightmaypredictimpaired decision-makinginmanicpatients.BipolarDisord.10,829–837

74 Yechiam, E. et al. (2008) Decision-making in bipolar disorder: a cognitivemodelingapproach.PsychiatryRes.161,142–152 75 Murphy,F.C.etal.(2001)Decision-makingcognitioninmaniaand

depression.Psychol.Med.31,679–693

76 Platt, M.L. and Huettel, S.A. (2008) Risky business: the neuroeconomics of decision-making under uncertainty. Nat. Neurosci.11,398–403

77 Glimcher, P.W. and Rustichini, A. (2004) Neuroeconomics: the consilienceofbrainanddecision.Science306,447–452

78 O’Doherty,J.P.etal.(2003)Temporaldifferencemodelsand reward-relatedlearninginthehumanbrain.Neuron38,329–337

79 Sutton, R. and Barto, A. (1998) Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning), MITPress

80 Green,D.M. and Swets, J.A. (1974)Signal Detection Theory and Psychophysics,R.E.KriegerPub.Co

81 Battaglia, P.W. et al. (2003) Bayesian integration of visual and auditorysignals forspatial localization. J.Opt. Soc.Am.A: Opt. ImageSci.Vis.20,1391–1397

82 Ernst,M.O.andBanks, M.S.(2002)Humansintegratevisualand haptic information in a statisticallyoptimal fashion. Nature 415, 429–433

83 Kording,K.(2007)Decisiontheory:what‘should’thenervoussystem do?Science318,606–610

84 Behrens,T.E.etal.(2007)Learningthevalueofinformationinan uncertainworld.Nat.Neurosci.10,1214–1221

85 Yu,A.andDayan,P.(2005)Inference,attention,anddecisionina Bayesian neural architecture. Adv. Neural Inf. Process.Syst. 17, 1577–1584

86 Kording, K.P. and Wolpert, D.M. (2004) Bayesian integration in sensorimotorlearning.Nature427,244–247

87 Boorman,E.D.etal.(2011)Counterfactualchoiceandlearningina neuralnetworkcenteredonhumanlateralfrontopolarcortex.PLoS Biol.9,e1001093

88 Yu,A.J.andCohen,J.D.(2009)Sequentialeffects:Superstitionor rationalbehavior?Adv.NeuralInf.Process.Syst.21,1873–1880 89 Doshi, F. et al. (2008) Reinforcement learning with limited

reinforcement: using Bayes risk for active learning in POMDPs.

Proc.Int.Conf.Mach.Learn.301,256–263

90 Kaelbling, L.P. et al. (1998) Planning and acting in partially observablestochasticdomains.Artif.Intell.101,99–134

91 Bellman,R.(1952)Onthetheoryofdynamicprogramming.ProcNatl. Acad.Sci.U.S.A.38,716–719

92 Ross,S.etal.(2008)OnlinePlanningAlgorithmsforPOMDPs.J. Artif.Intell.Res.32,663–704

93 McDonald-Madden,E.etal.(2011)Allocatingconservationresources betweenareaswherepersistenceofaspeciesisuncertain.Ecol.Appl.

21,844–858

94 Rao,R.P.(2010)Decision-makingunderuncertainty:aneuralmodel based on partially observable markov decision processes. Front. Comput.Neurosci.4,146

95 Huys,Q.J.andDayan,P.(2009)ABayesianformulationofbehavioral control.Cognition113,314–328

96 Tenenbaum, J.B. et al. (2006) Theory-based Bayesian models of inductivelearningandreasoning.TrendsCogn.Sci.10,309–318 97 Epstein,S.andPacini,R.(1999)Somebasicissuesregarding

dual-processtheoriesfromtheperspectiveofcognitive-experiential self-theory.InDualProcessTheoriesinSocialPsychology(Chaiken,S. andTrope,Y.,eds),pp.462–482,GuilfordPublishers

98 Bechara,A.etal.(1994)Insensitivitytofutureconsequencesfollowing damagetohumanprefrontalcortex.Cognition50,7–15

99 Kahneman,D.andTversky,A.(2000)Choices,ValuesandFrames, CambridgeUniversityPress

100VonNeumann,J.andMorgenstern,O.(1947)TheoryofGamesand EconomicBehavior,PrincetonUniversityPress