control actions
state info
Figure B1.5.3. A generic control problem.
Plant
disturbances
control actions
Inverse Plant Model
(+ "teacher")
state info.
error in
action
Figure B1.5.4. A supervised learning problem.
Now consider the reinforcement learning problem shown in figure B1.5.5 (Barto 1990). Here, no inverse plant model is available. However, a critic is available that indicates error in the state information from the plant. Because error is not directly provided in terms of control actions, the parameters of the controller cannot be directly adjusted by methods such as gradient descent.
Plant
disturbances
control actions
State Evaluator
(or "critic")
state info.
error in
state
Figure B1.5.5. A reinforcement learning problem.
The remaining discussion will consider the control problem to operate as a Markov decision problem. That is, the control problem operates in discrete time steps, the plant is always in one of a finite number of discrete states, and a finite, discrete number of control actions are available. At each time step, the control action alters the probability of moving the plant from the current state to any other state. Note that deterministic environments are a specific case. Although this discussion will limit itself to discrete problems, most of the points made can be related directly to continuous problems.
A characteristic of many reinforcement learning problems is that one may need to consider a sequence of control actions and their results to determine how to improve the controller. One can examine the
implications of this by associating a reward or cost with each control action. The error in state in figure B1.5.5 can be thought of as a cost. One can consider the long-term effects of an action formally as the expected, infinite-horizon discounted cost:
tX=∞ t=0
λtct
where 0≤λ≤1 is the discount parameter, andct is the cost of the action taken at timet.
To describe a strategy for picking actions, consider the following approach: for each action u
associated with a state i, assign a value Q(i, u). A ‘greedy’ strategy is to select the action associated with the best Q at every time step. Therefore, an optimum setting for the Q-values is one in which a ‘greedy’ strategy leads to the minimum expected, infinite-horizon discounted cost. Q-learning is a method that yields optimal Q-values in restricted situations. Consider beginning with random settings for each
Q-value, and updating eachQ-value on-line as follows:
Qt+1(i, ut)=Qt(i, u)(1−α)+α
ci(ut)+λminQ(j, ut+1)
where minQ(j, ut+1)is the minimumQavailable in statej, which is the state arrived in after actionut is taken in state i (Barto et al 1991, Watkins 1989). The parameter α is a learning rate parameter that is typically set to a small value between zero and one. Arguments based on dynamic programming and Bellman optimality show that if each state–action pair is tried an infinite number of times, this procedure results in optimalQ-values. Certainly, it is impractical to try every state–action pair an infinite number of times. With finite exploration,Q-values can often be arrived at that are approximately optimal. Regardless of the method employed to update a strategy in a reinforcement learning problem, this exploration– exploitation dilemma always exists.
Another difficulty in theQ-value approach is that it requires storage of a separateQ-value for each state–action pair. In a more practical approach, one could store aQ-value for a group of state–action pairs that share the same characteristics. However, it is not clear how state–action pairs should be grouped. In many ways, the LCS can be thought of as a GA-based technique for grouping state–action pairs.
B1.5.2.3 Learning classifier system introduction
Consider the following method for representing a state–action pair in a reinforcement learning problem: encode a state in binary, and couple it to an action, which is also encoded in binary. In other words, the string
0 1 1 0 / 0 1 0
represents one of 16 states and one of eight actions. This string can also be seen as a rule that says ‘IF in state0 1 1 0, THEN take action0 1 0’. In an LCS, such a rule is called a classifier. One can easily associate aQ-value, or other performance measures, with any given classifier.
Now consider generalizing over actions by introducing a ‘don’t care’ character (#) into the state portion of a classifier. In other words, the string
# 1 1 # / 0 1 0
is a rule that says ‘IF in state 0 1 1 0OR state0 1 1 1OR state 1 1 1 0OR state1 1 1 1, THEN take action0 1 0’. The introduction of this generality allows an LCS to represent clusters of states and associated actions. By using the genetic algorithm to search for such strings, one can search for ways of clustering states together, such that they can be assigned joint performance statistics, such as Qvalues.
Note, however, thatQ-learning is not the most common method of credit assignment in LCSs. The most common method is called the bucket brigade algorithm for updating a classifier performance statistic called strength. Details of the bucket brigade algorithm will be introduced later in this section.
The structure of a typical LCS is shown in figure B1.5.6. This is what is known as a stimulus–response LCS, since no internal messages are used as memory. Details of internal message posting in LCSs will be discussed later. In this system, detectors encode state information from an environment into binary messages, which are matched against a list of rules called classifiers. The classifiers used are of the form
Derivative methods