Model-Free Reinforcement Learning

An alternative to model-based learning is the model-free RL [100]. In this technique, a learning agent without any prior knowledge of the transitional probabilities or reward function learns a policy directly from the raw experience gained through recursively interacting with the environment. Monte Carlo (MC) methods and temporal difference (TD) learning are the two classes of methods under model-free RL. The Monte Carlo (MC) method updates the value functions episode-by-episode rather than step-by-step i.e., the value is updated only after an episode ends. The value function at a particular state is updated using the equation below:

V (s0) ← V (s) + α[Gt− V (s)] (2.4)

where, V (s0) is the new value estimate, V (s) is the old value estimate, α is a constant learning rate, Gt is the actual return received after the episode ends at time t. More-

over, an optimal policy is learnt by recursively performing policy evaluation and policy improvement to a fixed arbitrary policy π as:

π0 E−→ Qπ0 I − → π1 E_{−→ Q}π1 I − → π2 E_{−→ · · ·−→ π}I ∗ E_{−→ Q}∗ (2.5)

where−→ denotes policy evaluation andE −→ is policy improvement; making the policyI greedy with respect to the current value function. The initial policy and Q-value are presented by π0_{and Q}π0

whereas π∗ and Q∗relates to the optimal policy and Q-value. Another classification for solving an RL problem is TD learning. It is a combination of DP and MC method. The TD methods (a) does not require any prior knowledge of the model of environments dynamics or reward function to learn the solution to a problem, similar to MC methods, and (b) the learning agent utilizes the value update rule every time step to learn a policy in order to select its next action to reach the goal state, as in DP. The value at a particular state is evaluated by using the equation below:

where V (s0) is the new value estimate, V (s) is the old value estimate, α is a small positive value known as the learning rate which influences the learning process. Since the learning is based on the difference of values, i.e., [V (s0) − V (s)], therefore this technique is also referred to as temporal-difference learning method. The TD learning methods are further classified as on-policy and off-policy TD control methods. The most widely used on-policy TD method is SARSA [104] while Q-learning [105] is the most popular off-policy TD method.

SARSA: On-policy TD Control

The objective of an on-policy TD control method is to learn an action-value function rather than a state-value function. In particular, the learning agent estimates the value following a policy for the chosen state and action at every transition in time. It then recursively updates the current policy towards an optimal policy by performing policy evaluation and policy improvement. As seen in Figure 2.7, the algorithm follows a particular sequence of RL elements (state, action, reward, state, action), to learn a policy or value function, giving rise to its name SARSA. The value function, under the current policy π in SARSA is updated at every time step using the following update equation:

Q(s, a) ← Q(s, a) + α[r + γQ0(s0, a0) − Q(s, a)] (2.7) where, (Q(s, a) corresponds to the Q-value of the current state-action pair, α is the learning rate, γ is the discount factor, r is the reward received at each instance of time, and Q0(s0, a0) is the Q-value of the previous state-action pair. The general framework of SARSA algorithm is shown in Figure 2.7

ݏ௧ _ܽ ௧ ݎ௧ ݏ௧ାଵ _ܽ ௧ାଵ ݎ௧ାଵ ݏ௧ାଶ _ܽ ௧ାଶ ݎ௧ାଶ

Figure 2.7: Work flow diagram presenting different stages in SARSA learning ap- proach, redrawn from [140]

Q-learning: Off-policy TD Control

Q-learning (QL) proposed by Watkin is one of the most popular RL techniques in widespread use in wireless and artificial intelligence domain [105]. Moreover, it has been predominantly employed to develop the proposed intelligent user association techniques in the presented research work. The learning agent recursively interacts with the environment purely through trial-and-error iterations to gather the information related to the environment and learn the most appropriate solution. The key difference between Q-learning and SARSA is the action selection. In Q-learning, the learning agent in a particular state chooses an action based on greedy policy i.e., it selects the action with maximum Q-value. However, in the case of SARSA, the action selection depends on the current policy. The policy is improved gradually by performing pol- icy evaluation and policy improvement repeatedly, thus, leading to the selection of an optimal action.

The learning agent in Q-learning uses a learning policy, such the -greedy policy, to learn an action with maximum value. Subsequently, the Q-value associated with each action is recursively updated using the following update equation:

Q(s, a) ← Q(s, a) + α[r + γ max

a0 Q

0

(s0, a0) − Q(s, a)] (2.8) where, a is the action taken in the current state s, s0 is the next state, a0 is the action that can be taken in the next state s, Q(s, a) corresponds to the Q-value of the current state-action pair. The learning rate parameter, α ∈ [0, 1] controls the convergence rate. The discount factor γ ∈ [0, 1] controls the importance of future rewards with respect to immediate rewards. r represents the reward value that is awarded for the learnt action, and maxa0Q0(s0, a0) is the maximum Q-value among the available actions in the next

state s0.

In some of the learning problems the environment need not be represented by different states. The objective is to learn an appropriate action value by following a policy π. These learning problems are categorized as stateless learning or 1-dimensional learning problems [106] [107] . The Q-value associated with each action in stateless learning is updated using the following equation:

Q(a) ← (1 − α)Q(a) + α[r + γ max

a0 Q

0

(a0)] (2.9)

where, a is the action taken, Q(a) corresponds to the Q-value of the current action. The learning rate parameter, α ∈ [0, 1] controls the convergence rate. r is the reward awarded for the learnt action a. The discount factor γ ∈ [0, 1] controls the importance of future rewards with respect to immediate rewards. maxa0Q0(a0) is the maximum

Q-value among the available actions in the action space.