Optimal linear-quadratic control

(1)

Optimal linear-quadratic control

Martin Ellison

1 Motivation

The lectures so far have described a general method - value function iterations - for solving dynamic programming problems. However, one problem alluded to at the end of the last lecture was that the method suﬀers from the

“curse of dimensionality”. If the number of state variables is large then there are many arguments in the value function and it becomes computationally very intensive to iterate the value function. In practice, a state space with dimension about four or five already tests the limits of the power of the current generation of computers. In this lecture, we examine a particular class of dynamic programming problems that can be solved relatively easily. We focus on problems of linear-quadratic control, in which the payoﬀ function is quadratic and the transition equation is linear. Many standard problems in economics can be cast in such a linear-quadratic framework. We will show how a combination of analytical and numerical analysis can be used to derive the solution to the linear-quadratic problem.

1

(2)

2 Key reading

The formal analysis for this lecture is taken from “Dynamic Macroeconomic Theory” by Tom Sargent, Harvard University Press, 1987.

3 Other reading

Optimal linear-quadratic control is discussed in most graduate macroeco- nomics textbooks, e.g. chapter 4 of “Recursive Macroeconomic Theory”, 2nd ed by Lars Ljungqvist and Tom Sargent, MIT Press, 2000. The concepts are taken from the engineering theory of optimal control so more sophisticated treatments can be found in books such as “Analysis and Control of Dynamic Economic Systems” by Gregory Chow, 1975. Gauss codes for matrix Riccati equation iterations in a dynamic general equilibrium context are available from Morten Ravn’s homepage at http://faculty.london.edu/mravn/

4 Linear-quadratic control

The general framework we will analyse is one in which the agent chooses a vector of controlsu_tto influence a series of state variablesx_t. We do not limit the dimension of either of these vectors, although it is natural to consider cases where the number of state variables exceeds the number of controls, otherwise it may well be that there is a very simple trivial solution which controls the states perfectly. In the context of linear-quadratic control, we assume that the transition equation governing the evolution of state is linear linear in past values of the state variables and linear in current values of the control variables. We allow the problem to be stochastic by including random shocks "t (with variance-covariance matrix Σ) to the state variables. The payoﬀ function is assumed to be quadratic in the state and control variables,

(3)

giving quadratic forms in the objective. The fully-specified linear-quadratic control problem is specified below. The symmetric matricesR and Q are the weights of state and control variables in the payoﬀ function. MatricesA and B govern the linear evolution of state variables in the transition equation.

min{ut}EP^∞

t=0¯^t[x⁰_tRx_t+u⁰_tQu_t] s:t:

xt+1=Axt+But+"t

In dynamic programming form, the value function is defined over the state variables x_t.

V (xt) = min

ut [x⁰_tRxt+u⁰_tQut+¯EV (Axt+But+"t)] (1) It is certainly possible for us to proceed as before by discretising the state space for the value function and applying value function iteration to converge to the optimal policy. However, such a procedure is computationally very intensive and unnecessary in the special linear-quadratic case. Instead, we will use a diﬀerent approach which combines analytical and numerical methods. The key to the method is that we know the general form of the policy and value functions for linear-quadratic control problems. Armed with this knowledge, it is much easier to proceed. We begin by postulating a quadratic form for the value function, in which P is an idempotent matrix so P⁰ =P .

V (xt) =x⁰_tP xt+d

We proceed by substituting this form (with as yet undetermined matrices P and d) into the value function (1). For convenience of notation, we drop the time subscripts. In all cases, x and u refer to time t dated variables.

(4)

V (x) = min

u

£x⁰Rx + u⁰Qu + ¯E (Ax + Bu + ")⁰P (Ax + Bu + ") + ¯d¤ Expanding the quadratic terms in brackets, while remembering that (Ax)⁰ = x⁰A⁰ gives

V (x) = min

u







x⁰Rx + u⁰Qu +¯E







x⁰A⁰P Ax + x⁰A⁰P Bu + x⁰A⁰P "

+u⁰B⁰P Ax + u⁰B⁰P Bu + u⁰B⁰P "

+"⁰P Ax + "⁰P Bu + "⁰P "





 + ¯d







The expected values of the stochastic shocks is zero so terms of the form x⁰A⁰P "; u⁰B⁰P "; "⁰P Ax and "⁰P Bu drop out. We are left with

V (x) = min

u

"

x⁰Rx + u⁰Qu + ¯E

Ã x⁰A⁰P Ax + x⁰A⁰P Bu +u⁰B⁰P Ax + u⁰B⁰P Bu + "⁰P "

! +¯d

#

(2) The first order condition with respect tou can be used to derive optimal policy. Note that _du^du⁰Qu = 2Qu, _du^dZu = Z⁰and _du^du⁰Z = Z.

@V (x)

@u = 2Qu + 2¯B⁰P Ax + 2¯B⁰P Bu = 0 Solving in terms ofu implies

u = −(Q + ¯B⁰P B)⁻¹¯B⁰P Ax Or, more succinctly,

u = −Fx

F = (Q + ¯B⁰P B)⁻¹¯B⁰P A

(5)

Several things are worthy of note at this stage. Firstly, optimal control requires the control vector to react linearly to the state variables. We have yet to confirm that this implies a quadratic value function as first postulated, but it already suggests that the policy function has a very simple form. Secondly, the coeﬃcient matrix F in the policy function is a non-linear function of the fundamental matrices A; B; Q and the matrix P in the postulated value function. We therefore can approach the problem as one of determining F or P . Our choice is to calculate P , and then calculated the implied F , but other techniques take the opposite approach.

Economically, the policy reaction function is interesting because it is independent of the stochastic shocks "_t. This is because certainty equivalence holds in a linear-quadratic framework. There is no eﬀect on policy, unless shocks enter multiplicatively or payoﬀs are not quadratic.

We continue next to demonstrate that the linear policy function (derived from a postulated quadratic value function) does actually imply a quadratic value function. In the process, we will be able to determine the two matrices P and d. To do this, we substitute the policy function u = −F x back into the value function (2). Note that −x⁰F⁰B⁰P Ax is a scalar and so equal to

−x⁰A⁰P BF x.

x⁰P x + d =







x⁰Rx + x⁰F⁰QF x +¯E

Ã x⁰A⁰P Ax − 2x⁰A⁰P BF x +x⁰F⁰B⁰P BF x

!

+¯E"⁰P " + ¯d







Comparing coeﬃcients on constant terms, d = ¯E"⁰P " + ¯d

We simplify this equation by applying the result E"⁰P " = tr(E"⁰P ") = tr(P E"⁰") = tr(P Σ).

(6)

d = ¯

1− ¯tr(P Σ)

This equation shows how the additive uncertainty caused by the stochastic element " does have an effect on the value function, but that this effect is limited to the constant term, which is independent of policy. Hence, certainty equivalence holds in this respect. Comparing coefficients on the terms quadratic in x,

P = R + F⁰QF + ¯ (A⁰P A − 2A⁰P BF + F⁰B⁰P BF ) Rearranging,

P = R + ¯A⁰P A − 2¯A⁰P BF + F⁰(Q + ¯B⁰P B)F

We know that optimal policy definesF as (Q+¯B⁰P B)⁻¹¯B⁰P A. Hence, we have

P = R + ¯A⁰P A − 2¯A⁰P B(Q + ¯B⁰P B)⁻¹¯B⁰P A

+¯A⁰P B((Q + ¯B⁰P B)⁻¹)⁰(Q + ¯B⁰P B)(Q + ¯B⁰P B)⁻¹¯B⁰P A Using the fact that (Z⁻¹)⁰ = (Z⁰)⁻¹ and (Q + ¯B⁰P B)⁰ = (Q + ¯B⁰P B), this reduces to

P = R + ¯A⁰P A − ¯²A⁰P B(Q + ¯B⁰P B)⁻¹B⁰P A

This equation confirms that a linear policy function does imply a quadratic value function. It is often known as the algebraic matrix Riccati equation.

At present, it implicitly defines that matrix P in the value function in terms of the structural matrices A; B; R and Q. The matrix Riccati equation is as far as we can go analytically in linear-quadratic control. It does defineP as a function ofA; B; R and Q, but the relationship is not linear and potentially is

(7)

highly non-linear. Fortunately, a relatively simple iterative technique based on a matrix Riccati diﬀerence equation can be applied. Instead of trying to solve the Riccati equation directly, we start from an initial guess of theP matrix in the value function. The initial guess Pj is updated to Pj+1 according to

Pj+1 =R + ¯A⁰PjA − ¯²A⁰PjB(Q + ¯B⁰PjB)⁻¹B⁰PjA

This equation is iterated until convergence, which is guaranteed to unique- ness under very weak conditions. Specifically, having eigenvalues in A of modulus less than unity is a suﬃcient condition. In fact, even explosive systems with eigenvalues grater than one in absolute value can be handled if some other weak conditions hold.

Iteration of the matrix Riccati equation is directly analogous to the value function iterations we discussed in previous lectures. In fact, what we are doing is actually to iterate over the value function, with each successive P matrix equivalent to our earlier iterations overV . Once P has converged, it is a simple matter to calculate F in the optimal policy function.

5 Numerical application

To illustrate the practicalities of matrix Riccati diﬀerence equation iterations, we discuss Matlab code to solve a simple example of linear-quadratic control.

Our model is one in which a central bank is trying to simultaneously control inflation¼t and outputyt by choosing the interest ratert.The instantaneous payoﬀ function for the central bank is assumed to be quadratic in inflation, output and the interest rate.

Lt=¼²_t +y²_t + 0:1r_t²

(8)

We assume that the central bank places equal weight on inflation and output deviations from target (normalised to zero for convenience) and a smaller weight on deviations in the interest rate from target. The objective of the central bank is to minimise the present discounted value of expected losses, with discounting at the rate¯. The structure of the economy is given by two equations.

¼t+1 = 0:75¼t− 0:5rt+"^¼_t y_t+1 = 0:25y_t− 0:5rt+"^y_t

It is not intended that these equations are to be considered a serious representation of the structure of the economy. Rather, the purpose is to illustrate our technique. The first equation determines inflation, which is assumed to be highly persistent and negatively correlated with interest rates.

The timing is such that current interest rate decisions only aﬀect inflation with a lag - a timing convention favoured by Athanasios Orphanides amongst others. The second equation determines output in a similar fashion. High interest rates depress output but output itself is not as persistent as inflation.

The timing convention remains the same so interest rate decision only aﬀect output with a lag. Both inflation and output are subject to (potentially correlated) random disturbances in the form of shocks "^¼_t and "^y_t. The full minimisation problem is

min{rt}EP^∞

t=0¯^t£

¼²_t +y²_t + 0:1r_t²¤ s:t:

¼t+1= 0:75¼t− 0:5rt+"^¼_t yt+1 = 0:25yt− 0:5rt+"^y_t

(9)

The general form of optimal linear-quadratic control is

min{ut}EP^∞

t=0¯^t[x⁰_tRx_t+u⁰_tQu_t] s:t:

xt+1=Axt+But+"t

To cast our model in this general form, we define state variables as x_t≡ (¼t yt)⁰, the control variables as ut = rt, and the disturbances as "t ≡ ("^¼_t

"^y_t)⁰. The matrices R; Q; A and B are given by

R =

Ã 1 0 0 1

!

; Q = 0:1; A =

Ã 0:75 0 0 0:25

!

; B =

Ã −0:5

−0:5

!

The theory discussed in the previous section implies that all we need to do is iterate the matrix Riccati equation to find P , the calculate the policy reaction coeﬃcients F . The equations we will need are therefore

Pj+1 = R + ¯A⁰PjA − ¯²A⁰PjB(Q + ¯B⁰PjB)⁻¹B⁰PjA F = (Q + ¯B⁰P B)⁻¹¯B⁰P A

The Matlab code to solve the optimal linear-quadratic control problem is discussed below. Firstly, a new program is started by clearing the screen and the discount factor beta is defined.

CLEAR;

beta=0.99;

The matricesR; Q; A and B are first defined to be of the correct dimension and the non-zero elements are set.

(10)

Q=zeros(1,1);

R=zeros(2,2);

A=zeros(2,2);

B=zeros(2,1);

Q(1,1)=0.1;

R(1,1)=1;

R(1,2)=0;

R(2,1)=0;

R(2,2)=1;

A(1,1)=0.75;

A(1,2)=0;

A(2,1)=0;

A(2,2)=0.25;

B(1,1)=-0.5;

B(2,1)=-0.5;

The next section initialises the matrix Riccati equation iterations The variable d is used to measure the largest absolute in the elements of Pj between successive iterations. The variable i is simply a count of how many iterations have been carried out. The initial guess of the P matrix is contained in the matrix P 0. As initial values, we use

P0 =

Ã −0:000001 0

0 −0:000001

!

These starting values are used rather than zero because, ifQ = 0 and P 0 is zero then the matrixQ + ¯B⁰PjB in the Riccati equation is not invertible.

In our example, Q 6= 0 and we could just as easily used zeros as starting values. In practice, the algorithm is not sensitive to starting values in the vast majority of cases.

(11)

d=1;

i=0;

P0=-0.000001*eye(2);

Begin matrix Riccati equation iterations. We continue iterations until the maximum absolute diﬀerence in the elements of P_j between iterations is less than 0.0000000001. The new value Pj+1 is stored in the matrix P 1.

After each iteration, the new valueP 1 is compared to the old value P 0. The diﬀerence is contained in P d, from which the maximum absolute value is extracted into d. If d is not suﬃciently small then the initial guess P0 is updated and iterations continue. For each iteration, the iteration number i and maximum absolute deviation d are collected in I and D respectively in order to be printed at the end.

WHILE d>0.0000000001

P1=R+beta*A’P0*A-(beta*A’*P0*B)*(invpd(Q + beta*B’*P0*B))

*(beta*B’*P0*A);

Pd=P1-P0;

d=MAX(ABS(Pd));

d=MAX(d’);

P0=P1;

i=i+1;

END;

The matrix Riccati equation iterations are now complete. The policy function matrix F is calculated from the final iteration of the P matrix.

Both policy function matrix F and value function matrix P are printed in the command window.

P=P0;

F=-inv(Q+beta*B’*P*B)*(beta*B’*P*A);

(12)

ID=[I(2:length(I))’ D(2:length(I))’];

disp(’ i d’);

disp(ID);

disp(’ SOLUTIONS’);

disp(’F’);

disp(F);

disp(’P’);

disp(P);

The output of the computer code is as follows

i d

1.00000000000000 1.00000093812485 2.00000000000000 0.32523416362466 3.00000000000000 0.08055819178924 4.00000000000000 0.01887570616174 5.00000000000000 0.00433868656498 6.00000000000000 0.00099273060023 7.00000000000000 0.00022690780983 8.00000000000000 0.00005185174817 9.00000000000000 0.00001184823275 10.00000000000000 0.00000270731205 11.00000000000000 0.00000061861695 12.00000000000000 0.00000014135300 13.00000000000000 0.00000003229893 14.00000000000000 0.00000000738025 15.00000000000000 0.00000000168638 16.00000000000000 0.00000000038533 17.00000000000000 0.00000000008805

(13)

SOLUTIONS F

0.74495417123607 0.17590987848800 P

1.43029303877617 -0.10618330436474 -0.10618330436474 1.04418992871296

As can be seen from the low number of iterations, the matrix Riccati equation iterations converge quickly. Returning to the context of our numerical model, the results imply policy and value functions of the following form.

rt = 0:745¼t+ 0:176yt

V (¼t; yt) = 1:43¼²_t + 1:04y²_t − 2 × 0:11¼tyt

According to the policy function, the interest rate needs to rise whenever inflation or output is above target. The result is intuitively appealing, with the central bank deflating the economy when inflation and/or output is too high. The larger reaction to inflation than output is due to our assumption that inflation is more persistent than output. Inflation is intrinsically more problematic in the model since, if inflation deviates from target in the current period then the deviation is likely to persist to the next period.

The value function can similarly be interpreted. The coeﬃcient on the square of inflation exceeds that on the square of output precisely because the higher persistence of inflation makes it more problematic. The nega- tive coeﬃcient on the cross-product of inflation and output reflects the fact that it is easier to control inflation and output when they are deviating from target in the same direction. A rise in the interest rate depresses both inflation and output, so if inflation is above target and output below target (i.e.

stagflation) then it is very diﬃcult to stabilise the economy.