Delayed-Reinforcement Learning
11.3 Temporal Discounting and Optimal Poli cies
In delayed reinforcement learning, one often assumes that rewards in the distant future are not as valuable as are more immediate rewards. This preference can be accomodated by a temporal discount factor, 0 <1.
The present value of a reward, ri, occuring i time units in the future, is
taken to beiri. Suppose we have a policy(
X
) that maps input vectorsinto actions, and letri(
X
) be the reward that will be received on thei-thtime step after one begins executing policystarting in state
X
. Then the total reward accumulated over all time steps by policybeginning in stateX
is: V(X
) = 1 X i=0 ir(X
) iR G 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8
Figure 11.3: An Optimal Policy in the Grid World
One reason for using a temporal discount factor is so that the above sum will be nite. An optimal policy is one that maximizesV(
X
) for all inputs,X
.In general, we want to consider the case in which the rewards, ri, are
random variables and in which the eects of actions on environmental states are random. In Markovian environments, for example, the probability that actionain state
X
iwill lead to stateX
jis given by a transition probabilityp
X
jjX
ia]. Then, we will want to maximize expected future reward andwould deneV(
X
) as:V(
X
) =E " 1 X i=0 iri(X
) #In either case, we callV(
X
) thevalueof policyfor inputX
.If the action prescribed by taken in state
X
leads to stateX
0 (ran-domly according to the transition probabilities), then we can writeV(
X
)in terms ofV(
X
0) as follows: V(X
) =rX
(X
)] +XX
0 pX
0 jX
(X
)]V(X
0)=the discount factor,
V(
X
) = the value of stateX
under policy,r
X
(X
)] =the expected immediate reward received when we execute the action prescribed byin stateX
, andp
X
0j
X
(X
)] =the probability that the environment transitions tostate
X
0when we execute the action prescribed byin stateX
.In other words, the value of state
X
under policyis the expected value of the immediate reward received when executing the action recommended byplus the average value (under) of all of the states accessible fromX
. For an optimal policy, (and no others!), we have the famous \opti-mality equation:" V (
X
) = maxa 2 4r(X
a) + XX
0 pX
0 jX
a]V (X
0) 3 5The theory of dynamic programming (DP) Bellman, 1957, Ross, 1983] assures us that there is at least one optimal policy, , that satises
this equation. DP also provides methods for calculating V
(
X
) and at least one , assuming that we know the average rewards and the tran-sition probabilities. If we knew the transition probabilities, the average rewards, and V
(
X
) for allX
and a, then it would be easy to imple- ment an optimal policy. We would simply select that a that maximizesr(
X
a) +PX
0pX
0 jX
a]V (X
0). That is, (X
) =argmax a 2 4r(X
a) + XX
0 pX
0 jX
a]V (X
0) 3 5But, of course, we are assuming that we do not know these average rewards nor the transition probabilities, so we have to nd a method that eectively learns them.
If we had a model of actions, that is, if we knew for every state,
X
, and actiona, which state,X
0resulted, then we could use a method calledvalueiteration to nd an optimal policy. Value iteration works as follows: We begin by assigning, randomly, anestimated valueV^(
X
) to every state,X
. On thei-th step of the process, suppose we are at stateX
i(that is, our inputon thei-th step is
X
i), and that the estimated value of stateX
i on thei-thstep is ^Vi(
X
i). We then select that actionathat maximizes the estimatedhaving the highest estimated value is
X
0i. Then we update the estimated
value, ^Vi(
X
i), of stateX
i as follows:^ Vi(
X
) = (1;ci)^Vi ;1(X
) +ci h ri+V^i;1(X
0 i) i ifX
=X
i, = ^Vi;1(X
) otherwise.We see that this adjustment moves the value of ^Vi(
X
i) an increment(depending onci) closer to h ri+V^i(
X
0 i) i . Assuming that ^Vi(X
0 i) is a good estimate forVi(X
0i), then this adjustment helps to make the two estimates
more consistent. Providing that 0 < ci < 1 and that we visit each state
innitely often, this process of value iteration will converge to the optimal values. Discuss synchronous dynamic programming, asynchronous dynamic programming, and policy iteration.
11.4
Q-Learning
Watkins Watkins, 1989] has proposed a technique that he callsincremental dynamic programming. Letastand for the policy that chooses actiona
once, and thereafter chooses actions according to policy. We dene:
Q(
X
a) =Va(X
)Then the optimal value from state
X
is given by:V
(
X
) = maxa Q(
X
a)This equation holds only for an optimal policy, . The optimal policy is
given by:
(
X
) =argmax a Q
(
X
a)Note that if an actiona makesQ(
X
a) larger thanV(X
), then we canimprove by changing it so that(
X
) =a. Making such a change is the basis for a powerful learning rule that we shall describe shortly.Suppose actionain state
X
leads to stateX
0. Then using the denitionsofQandV, it is easy to show that:
Q(
X
a) =r(X
a) +EV(X
0)]wherer(
X
a) is the average value of the immediate reward received when we execute actionain stateX
. For an optimal policy (and no others), we have another version of the optimality equation in terms ofQvalues:Q (
X
a) =maxa h r(X
a) +Eh Q (X
0a) iifor all actions,a, and states,
X
. Now, if we had the optimalQvalues (for allaand
X
), then we could implement an optimal policy simply by selecting that action that maximizedr(X
a) +EQ (
X
0a) . That is, (X
) =argmax a h r(X
a) +Eh Q (X
0a) i iWatkins' proposal amounts to a TD(0) method of learning theQvalues. We quote (with minor notational changes) from Watkins & Dayan, 1992, page 281]:
\InQ-Learning, the agent's experience consists of a sequence of distinct stages orepisodes. In thei-th episode, the agent:
observes its current state
X
i,selects using the method described below] and performs
an actionai,
observes the subsequent state
X
0 i, receives an immediate rewardri, and adjusts itsQi;1values using a learning factorci, according
to: Qi(
X
a) = (1;ci)Qi ;1(X
a) +ciri+Vi;1(X
0 i)] ifX
=X
i anda=ai, =Qi;1(X
a) otherwise,where
Vi;1(
X
0) = max
b Qi;1(
X
0b)]is the best the agent thinks it can do from state
X
0. :::The initial Qvalues, Q0(
X
a), for all states and actionsare assumed given."
Using the currentQvalues,Qi(
X
a), the agent always selects that ac-tion that maximizesQi(
X
a). Note that only the Q value correspondingto the state just exited and the action just taken is adjusted. And that
Q value is adjusted so that it is closer (by an amount determined by ci)
to the sum of the immediate reward plus the discounted maximum (over all actions) of the Q values of the state just entered. If we imagine the
Qvalues to be predictions of ultimate (innite horizon) total reward, then the learning procedure described above is exactly a TD(0) method of learn- ing how to predict theseQ values. Q learning strengthens the usual TD methods, however, because TD (applied to reinforcement problems using value iteration) requires a one-step lookahead, using a model of the eects of actions, whereasQlearning does not.
A convenient notation (proposed by Schwartz, 1993]) for representing the change in Qvalue is:
Q(
X
a) ;r+V(X
0)whereQ(
X
a) is the newQvalue for inputX
and actiona,ris the imme- diate reward when actionais taken in response to inputX
,V(X
0) is themaximum (over all actions) of theQvalue of the state next reached when action a is taken from state
X
, and is the fraction of the way toward which the newQvalue,Q(X
a), is adjusted to equalr+V(X
0).Watkins and Dayan Watkins & Dayan, 1992] prove that, under certain conditions, theQvalues computed by this learning procedure converge to optimal ones (that is, to ones on which an optimal policy can be based).
We deneni(
X
a) as the index (episode number) of thei-th time thatactionais tried in state