CHAPTER 2 THE MAXIMUM PRINCIPLE: CONTINUOUS TIME. Chapter2 p. 1/67

(1)

C

HAPTER

2

(2)

THE MAXIMUM PRINCIPLE: CONTINUOUS TIME

• _{Main Purpose: Introduce the maximum principle as a}

necessary condition to be satisfied by any optimal control.

• _{Necessary conditions for optimization of dynamic systems.} • _{General derivation by Pontryagin et al. in 1956-60.}

• _{A simple (but not completely rigorous) proof using dynamic}

programming.

• _Examples.

• _{Statement of sufficiency conditions.} • _{Computational method.}

(3)

2.1 STATEMENT OF THE PROBLEM

Optimal control theory deals with the problem of

optimization of dynamic systems.

(4)

2.1.1 THE MATHEMATICAL MODEL

State equation

˙

x(t) = f(x(t), u(t), t), x(0) = x₀, (1)

where the vector of state variables, x(t) ∈ En, the vector of control variables, u(t) ∈ Em, and the function

f : En × Em × E1 → En.

(5)

2.1.1 THE MATHEMATICAL MODEL CONT.

The path x(t), t ∈ [0, T], is called a state trajectory and u(t),

t ∈ [0, T], is called a control trajectory. Admissible control u(t), t ∈ [0, T], is piecewise continuous and satisfies, in addition,

(6)

2.1.3 THE OBJECTIVE FUNCTION

Objective function is defined as follows

J =

Z T

0

F(x(t), u(t), t)dt + S[x(T ), T], (3)

where the functions F : En × Em × E1 → E1 and S : En × E1 → E1 are assumed to be continuously differentiable.

(7)

2.1.4 THE OPTIMAL CONTROL PROBLEM

The problem is to find an admissible control u∗

, which

maximizes the objective function (3) subject to the state equation (1) and the control constraints (2). We now restate the optimal control problem as:

           max u(t)∈Ω(t) J = Z T 0 F(x, u, t)dt + S[x(T ), T] subject to ˙ x = f(x, u, t), x(0) = x₀. (4) The control u∗

is called an optimal control and x∗

is called the corresponding optimal trajectory. The optimal value of the objective function will be denoted as J(u∗

) or J∗

(8)

THREE CASES

• _{Case 1. The optimal control problem (4) is said to be in}

Bolza form.

• _{Case 2. When} _S _≡ ₀_{, it is said to be in Lagrange form.} • _{Case 3. When} _F _≡ ₀_{, it is said to be in Mayer form. If}

F ≡ 0 and S is linear, it is in linear Mayer form, i.e.,

         max u(t)∈Ω(t) {J = cx(T)} subject to ˙ x = f(x, u, t), x(0) = x₀, (5) where c = (c₁, c₂, · · · , c_n) ∈ En.

(9)

FORMS

The Bolza form can be reduced to the linear Mayer form:

Introduce a new state vector y = (y₁, y₂, . . . , yn+1), having n + 1

components defined as follows: yi = xi for i = 1, . . . , n, and

˙ yn+1 = F(x, u, t) + ∂S(x, t) ∂x f(x, u, t) + ∂S(x, t) ∂t , yn+1(0) = S(x0, 0), (6)

so that the objective function is J = ˆcy(T ) = y_n₊₁(T), where

ˆ

c = (0, 0, . . . , 0

| {z }

n

, 1). If we now integrate (6) from 0 to T, we have

J = ˆcy(T) = yn+1(T) =

Z T

0

(10)

PRINCIPLE OF OPTIMALITY

•

_{Statement of Bellman’s Optimality Principle}

"An optimal policy has the property that whatever the

initial state and initial decision are, the remaining

decisions must constitute an optimal policy with regard

to the state resulting from the first decision."

(11)

PRINCIPLE OF OPTIMALITY CONT. A B C E J_AB JBE J_BCE

ASSERTION: If ABE (shown in blue) is an optimal path from A to

E, then BE (in blue) is an optimal path from B to E

PROOF. Suppose it is not. Then there is another path (existence is

assumed here) BCE (in red), which is optimal from B to E, i.e., J_BCE > J_BE.

But then

JABE = JAB + JBE < JAB + JBCE = JABCE.

This contradicts the hypothesis that ABE is an optimal path from A to E.

(12)

A DYNAMIC PROGRAMMING EXAMPLE Stagecoach Problem

1

2

3

4

5

6

7

8

9

10

2 4 3 3 4 -R -3 3 -s -jw : j R q * s 3 Stages 1 2 3 4

Costs

(ci,j)

:

5 6 7 2 3 4 7 4 6 3 2 4 4 1 5 8 9 5 6 7 1 4 6 3 3 3

(13)

A DYNAMIC PROGRAMMING EXAMPLE CONT. Stagecoach Problem

1

2

3

4

5

6

7

8

9

10

2 4 3 4 -R 7 4 6 -s w 4 1 5 3 -6 3 : q 3 Stages 1 2 3 4 -3 j j R * s

Note that 1-2-6-9-10 is a greedy (or myopic) path that minimizes cost at each stage. However, this may not be the minimal cost

(14)

SOLUTION TO THE DP EXAMPLE

Let 1 − u₁ − u₂ − u₃ − 10 be the optimal path. Let f_n(s, u_n) be the minimal cost at stage n given that the current state is s and the decision taken is u_n. Let f∗

n(s) be the minimal cost at stage n

if the current state is s. Then f∗ n(s) = min un f(s, un) = min un cs,un + f ∗ n+1(un) . This is the Recursion Equation of DP. It can be solved by a

backward procedure, which starts at the terminal stage and stops at the initial stage.

(15)

SOLUTION TO THE DP EXAMPLE CONT. • _{Stage 4.} s f∗ 4(s) u ∗ 4 8 3 10 9 4 10 • _{Stage 3.} s f₃(s, u₃) = Cs,u3 + f ∗ 4(u3) f3∗(s) u∗3 8 9 5 1 + 3 = 4 4 + 4 = 8 4 8 6 6 + 3 = 9 3 + 4 = 7 7 9 7 3 + 3 = 6 3 + 4 = 7 6 8 8 9 5 6 7 1 4 6 3 3 3

(16)

SOLUTION TO THE DP EXAMPLE CONT. • _{Stage 2.} s f₂(s, u₂) = Cs,u2 + f ∗ 3(u2) f ∗ 2(s) u ∗ 2 5 6 7 2 7 + 4 = 11 4 + 7 = 11 6 + 6 = 12 11 5,6 3 3 + 4 = 7 2 + 7 = 9 4 + 6 = 10 7 5 4 4 + 4 = 8 1 + 7 = 8 5 + 6 = 11 8 5,6 5 6 7 2 3 4 7 4 6 3 2 4 4 1 5 • _{Stage 1.} s f₁(s, u₁) = Cs,u1 + f ∗ 2(u1) f ∗ 1 (s) u ∗ 1 2 3 4 1 2 + 11 = 13 4 + 7 = 11 3 + 8 = 11 11 3,4

The optimal paths:

1 − 3 − 5 − 8 − 10 1 − 4 − 5 − 8 − 10 1 − 4 − 6 − 9 − 10 9 > > = > > ; Total Cost = 11.

(17)

2.2 DYNAMIC PROGRAMMING AND THE

MAXIMUM PRINCIPLE

2.2.1 The Hamilton-Jacobi-Bellman Equation

V (x, t) = max u(s)∈Ω(s) Z T t F(x(s), u(s), s)ds + S(x(T ), T) , (8) where for s ≥ t, dx ds = f(x(s), u(s), s), x(t) = x.

(18)

FIGURE 2.1 AN OPTIMAL PATH IN THE

(19)

TWO CHANGES IN THE OBJECTIVE FUNCTION

(i) The incremental change in J from t to t + δt is given by the integral of F(x, u, t) from t to t + δt.

(ii) The value function V (x + δx, t + δt) at time t + δt. In equation form this is

V (x, t) = max u(τ)∈Ω(τ) τ∈[t,t+δt] Z t+δt t F[x(τ), u(τ), τ]dτ +V [x(t + δt), t + δt] , (9)

Since F is a continuous function, the integral in (9) is approximately F(x, u, t)δt so that we can rewrite (9) as V (x, t) = max

u∈Ω(t)

{F(x, u, t)δt + V [x(t + δt), t + δt]}+o(δt).

(20)

CONT.

Assume that the value function V is continuously differentiable. Use the Taylor series expansion of V with respect to δt and

obtain

V [x(t + δt), t + δt] = V (x, t) + [Vx(x, t) ˙x + Vt(x, t)]δt + o(δt),

(11)

Substituting for x˙ from (1), we obtain

V (x, t) = max u∈Ω(t) F(x, u, t)δt + V (x, t) +Vx(x, t)f(x, u, t)δt + Vt(x, t)δt + o(δt). (12)

Cancelling V (x, t) on both sides and then dividing by δt we get

0 = max u∈Ω(t) F(x, u, t) + Vx(x, t)f(x, u, t) + Vt(x, t) + o(δt) δt . (13)

(21)

CONT.

Letting δt → 0 in (10) gives the equation

0 = max

u∈Ω(t)

{F(x, u, t) + Vx(x, t)f(x, u, t) + Vt(x, t)} , (14)

for which the boundary condition is

V (x, T ) = S(x, T ). (15)

Vx(x, t) can be interpreted as the marginal contribution vector of

the state variable x to the maximized objective function. Denote it by the adjoint (row) vector λ(t) ∈ En, i.e.,

λ(t) = Vx(x

∗

(22)

HAMILTON-JACOBI-BELLMAN EQUATION

We introduce the so-called Hamiltonian

H[x, u, Vx, t] = F(x, u, t) + Vx(x, t)f(x, u, t) (17)

or, simply,

H(x, u, λ, t) = F(x, u, t) + λf(x, u, t). (18)

We can rewrite equation (14) as the following equation,

0 = max

u∈Ω(t)

[H(x, u, Vx, t) + Vt], (19)

called the Hamilton-Jacobi-Bellman equation or, simply, the HJB equation.

(23)

HAMILTONIAN MAXIMIZING CONDITION

From (19), we can get the Hamiltonian maximizing condition of the maximum principle,

H[x∗ (t), u∗ (t), λ(t), t] + Vt(x ∗ (t), t) ≥ H[x∗ (t), u, λ(t), t] +Vt(x ∗ (t), t). (20)

Cancelling the term Vt on both sides, we obtain

H[x∗

(t), u∗

(t), λ(t), t] ≥ H[x∗

(t), u, λ(t), t] (21)

for all u ∈ Ω(t).

Remark: H decouples the problem over time by means of λ(t), which is analogous to dual variables or shadow prices in Linear Programming.

(24)

2.2.2 DERIVATION OF THE ADJOINT EQUATION

Let

x(t) = x∗

(t) + δx(t), (22)

where k δx(t) k< ε for a small positive ε. Fix t and use the HJB equation in (19) as

H[x∗ (t), u∗ (t), Vx(x ∗ (t), t), t] + Vt(x ∗ (t), t) ≥ H[x(t), u∗ (t), Vx(x(t), t), t] + Vt(x(t), t). (23) LHS= 0 from (19) since u∗ (t) maximizes H + Vt. RHS will be zero if u∗

(t) also maximizes H + V_t with x(t) as the state. In general x(t) 6= x∗(t), and thus RHS ≤ 0. But then

(25)

CONT.

Since,

x(t)

is unconstrained, we have

∂RHS_∂x |_x₍_t₎₌_x∗₍_t₎ = 0

, or

Hx[x∗(t), u∗(t), Vx(x∗(t), t), t] + Vtx(x∗(t), t) = 0. (24)

By definition of H,

F_x+V_xf_x+fTV_xx+V_tx = F_x+V_xf_x+(V_xxf)T +V_tx = 0. (25)

Note: (25) assumes V to be twice continuously

(26)

CONT.

By definition (16) of

λ(t)

, the row vector

λ˙ (t)

is

dVx dt = _dV x₁ dt , dVx₂ dt , · · · , dV_xn dt = (V_x₁_xx˙ + V_x₁_t, V_x₂_xx˙ + V_x₂_t, · · · , V_x_n_xx˙ + V_x_n_t) = Pn_i₌₁ V_x₁_x_ix˙_i, Pn_i₌₁ V_x₂_x_ix˙_i, · · · , Pn_i₌₁ V_x_n_x_ix˙_i + (V_x)_t = (Vxxx˙)T + Vxt = (V_xxf)T + V_tx. (26)

Note

Vx1x = (Vx1x1, Vx1x2, ..., Vx1xn),

and

Vxxx˙ = 0 B B B B B B Vx1x1 Vx1x2 ... Vx1xn Vx2x1 Vx2x2 ... Vx2xn . . . . . . Vxnx1 Vxnx2 ... Vxnxn 1 C C C C C C A 0 B B B B B B ˙ x₁ ˙ x₂ . . . ˙ xn 1 C C C C C C A ,

a column vector.

(27)

CONT.

Using (25) and (26) we have dVx dt = −Fx − Vxfx. (27) Using (16) we have ˙ λ = −Fx − λfx. From (18) we have ˙ λ = −Hx. (28)

(28)

TERMINAL BOUNDARY OR TRANSVERSALITY

CONDITION:

λ(T) = ∂S(x, T)

∂x |x=x(T)= Sx[x(T ), T]. (29) (28) and (29) can determine the adjoint variables.

From (18), we can rewrite the state equation as

˙

x = f = Hλ. (30)

From (28), (29), (30) and (1), we get

   ˙ x = Hλ, x(0) = x0, ˙ λ = −Hx, λ(T ) = Sx[x(T), T], (31)

(29)

FREE AND FIXED END POINT PROBLEMS

• _{In our problem} _x₍_T ₎ _{is free. In this case,}

S ≡ 0 (no salvage value) ⇒ λ(T) = 0

• _{However, if} _x₍_T₎ _{is not free, that is,}

x(T) fixed (given) ⇒ λ(T) = a constant to be determined. In other words, λ(T) is free.

(30)

2.2.3 THE MAXIMUM PRINCIPLE

The necessary conditions for u∗

to be an optimal control are:

       ˙ x∗ = f(x∗ , u∗ , t), x∗ (0) = x₀, ˙ λ = −Hx[x∗, u∗, λ, t], λ(T ) = Sx[x∗(T ), T], H[x∗ (t), u∗ (t), λ(t), t] ≥ H[x∗ (t), u, λ(t), t], (32) for all u ∈ Ω(t), t ∈ [0, T].

(31)

2.2.4 ECONOMIC INTERPRETATION OF THE MAXIMUM PRINCIPLES J = Z T 0 F(x, u, t)dt + S[x(T), T],

where F is the instantaneous profit rate per unit of time, and

S[x, T] is the salvage value. Multiplying (18) formally by dt and using the state equation (1) gives

Hdt = F dt + λf dt = F dt + λxdt˙ = F dt + λdx.

F(x, u, t)dt: direct contribution to J from t to t + dt λdx: indirect contribution to J due to added capital dx

Hdt: total contribution to J from time t to t + dt, when x(t) = x and u(t) = u in the interval [t, t + dt].

(32)

2.2.4 ECONOMIC INTERPRETATION OF THE

MAXIMUM PRINCIPLES CONT.

By (28) and (29), we have ˙ λ = −∂H ∂x = − ∂F ∂x − λ ∂f ∂x, λ(T ) = Sx[x(T), T]. Rewrite the first equation as

−dλ = Hxdt = Fxdt + λfxdt.

−dλ: marginal cost of holding capital x from t to t + dt Hxdt: marginal revenue of investing the capital

Fxdt: direct marginal contribution

λfxdt: indirect marginal contribution

(33)

EXAMPLE 2.1

Consider the problem:

max J = Z 1 0 −xdt (33)

subject to the state equation

˙

x = u, x(0) = 1 (34)

and the control constraint

u ∈ Ω = [−1, 1]. (35)

Note that T = 1, F = −x, S = 0, and f = u. Because F = −x, we can interpret the problem as one of minimizing the (signed) area under the curve x(t) for 0 ≤ t ≤ 1.

(34)

SOLUTION OF EXAMPLE 2.1

We form the Hamiltonian

H = −x + λu. (36)

Because the Hamiltonian is linear in u, the form of the optimal control, i.e., the one that would maximize the Hamiltonian, is

u∗ (t) =        1 if λ(t) > 0, undefined if λ(t) = 0, −1 if λ(t) < 0. (37)

It is called Bang-Bang Control. In the notation of Section 1.4, u∗

(35)

SOLUTION OF EXAMPLE 2.1 CONT.

To find λ, we write the adjoint equation

˙

λ = −Hx = 1, λ(1) = Sx[x(T), T] = 0. (39)

Because this equation does not involve x and u, we can easily solve it as

λ(t) = t − 1. (40)

It follows that λ(t) = t − 1 ≤ 0 for all t ∈ [0, 1]. So u∗

(t) = −1, t ∈ [0, 1). Since λ(1) = 0, we can also set u∗(1) = −1, which defines u at the single point t = 1. We thus have the optimal control

u∗

(36)

Substituting this into the state equation (34) we have

˙

x = −1, x(0) = 1, (41)

whose solution is

x(t) = 1 − t for t ∈ [0, 1]. (42)

The graphs of the optimal state and adjoint trajectories appear in Figure 2.2. Note that the optimal value of the objective function is J∗

(37)

FIGURE 2.2 OPTIMAL STATE AND ADJOINT

(38)

EXAMPLE 2.2

Let us solve the same problem as in Example 2.1 over the interval [0, 2] so that the objective is to

maximize J = Z 2 0 −xdt . (43)

As before, the dynamics and constraints are (33) and (34),

respectively. Here we want to minimize the signed area between the horizontal axis and the trajectory of x(t) for 0 ≤ t ≤ 2.

(39)

As before the Hamiltonian is defined by (36) and the optimal control is as in (38). The adjoint equation

˙

λ = 1, λ(2) = 0 (44)

is the same as (39), except that now T = 2 instead of T = 1. The solution of (44) is easily found to be

λ(t) = t − 2, t ∈ [0, 2]. (45)

Hence the state equation (41) and its solution (42) are exactly the same. The graph of λ(t) is shown in Figure 2.3. The optimal

value of the objective function is J∗

(40)

FIGURE 2.3 OPTIMAL STATE AND ADJOINT

(41)

EXAMPLE 2.3

The next example is:

max J = Z 1 0 −1 2x 2_dt (46)

subject to the same constraints as in Example 2.1, namely,

˙

x = u, x(0) = 1, u ∈ Ω = [−1, 1]. (47)

Here F = −(1/2)x2 so that the interpretation of the objective function (46) is that we are trying to find the trajectory x(t) in order that the area under the curve (1/2)x2 is minimized.

(42)

SOLUTION OF EXAMPLE 2.3 The Hamiltonian is H = −1 2x 2 ₊ _λu, (48)

which is linear in u so that the optimal policy is u∗

(t) = bang [−1, 1; λ]. (49)

The adjoint equation is

˙

λ = −Hx = x, λ(1) = 0. (50)

Here the adjoint equation involves x so that we cannot solve it directly. Because the state equation (47) involves u, which

depends on λ, we also cannot integrate it independently without knowing λ.

(43)

A way out of this dilemma is to use some intuition. Since we

want to minimize the area under (1/2)x2 and since x(0) = 1, it is clear that we want x to decrease as quickly as possible. Let us therefore temporarily assume that λ is nonpositive in the interval

[0, 1] so that from (49) we have u = −1 throughout the interval. (In Exercise 2.5, you will be asked to show that this assumption is correct.) With this assumption, we can solve (47) as

(44)

Substituting this into (50) gives

˙

λ = 1 − t.

Integrating both sides of this equation from t to 1 gives

Z 1 t ˙ λ(τ)dτ = Z 1 t (1 − τ)dτ, or λ(1) − λ(t) = (τ − 1 2τ 2₎ _|1 t ,

which, using λ(1) = 0, yields

λ(t) = −1

2t

2 ₊ _t ₋ 1

(45)

The reader may now verify that λ(t) is nonpositive in the interval

[0, 1], justifying our original assumption. Hence, (51) and (52) satisfy the necessary conditions. In Exercise 2.6, you will be asked to show that they satisfy sufficient conditions derived in Section 2.4 as well, so that they are indeed optimal. Figure 2.4 shows the graphs of the optimal trajectories.

(46)

FIGURE 2.4 OPTIMAL TRAJECTORIES FOR

EXAMPLE 2.3 AND EXAMPLE 2.4

t (t)=1-t Â(t)=-t@/2+t-1/2 1 0 ₁ x, Â

.

x_*

1

-2

(47)

EXAMPLE 2.4

Let us rework Example 2.3 with T = 2, i.e., solve the problem:

max J = Z 2 0 −1 2x 2_dt (53)

subject to the constraints

˙

x = u, x(0) = 1, u ∈ Ω = [−1, 1].

It would be clearly optimal if we could keep x∗

(t) = 0, t ∈ [1, 2]. This is possible by setting u∗

(t) = 0, t ∈ [1, 2]. Note u∗

(t) = 0, t ∈ [1, 2], is a singular control, since it gives λ(t) = 0, t ∈ [1, 2]. See Fig. 2.4.

(48)

EXAMPLE 2.5

The problem is:

max J = Z 2 0 (2x − 3u − u2)dt (54) subject to ˙ x = x + u, x(0) = 5 (55)

and the control constraint

(49)

The Hamiltonian is

H = (2x − 3u − u2) + λ(x + u)

= (2 + λ)x − (u2 + 3u − λu). (57)

The optimal control is

u(t) = λ(t) − 3

2 . (58)

The adjoint equation is

˙

λ = −∂H

(50)

Using the integrating factor as shown in Appendix A.6, its solution is

λ(t) = 2(e2−t

− 1). The optimal control is

u∗ (t) =        2 if e2−t − 2.5 > 2, e2−t − 2.5 if 0 ≤ e2−t − 2.5 ≤ 2, 0 if e2−t − 2.5 < 0. (60) It can be written as u∗ (t) = sat [0, 2; e2−t − 2.5].

(51)

FIGURE 2.5 OPTIMAL CONTROL FOR EXAMPLE 2.5 6 -Y u∗₍_t₎ e2−t ₋ ₂_.₅ 2 t₁ _≈ 0.5 0 u t t₂ _≈ 1.1 2

(52)

2.4 SUFFICIENT CONDITIONS

H0(x, λ, t) = max

u∈Ω(t)

H(x, u, λ, t). (61)

H0(x, λ, t) = H(x, u0, λ, t). (62)

The Envelope Theorem gives

H_x0(x, λ, t) = Hx(x, u0, λ, t). (63) Here is why? H_x0(x, λ, t) = Hx(x, u0, λ, t) + Hu(x, u0, λ, t) ∂u0 ∂x . (64) But Hu(x, u0, λ, t) ∂u0 ∂x = 0, (65)

(53)

THEOREM 2.1

(

Sufficiency Conditions

)

Let

u

∗

(

t

)

, and the

corresponding

x

∗

(

t

)

and

λ

(

t

)

satisfy the maximum

principle necessary condition

(32)

for all

t

∈

[0

, T

]

.

Then,

u

∗

is an optimal control if

H

0

(

x, λ

(

t

)

, t

)

is

(54)

PROOF OF THEOREM 2.1

By definition of H0,

H[x(t), u(t), λ(t), t] ≤ H0[x(t), λ(t), t]. (66)

Since H0 is differentiable and concave, H0[x(t), λ(t), t] ≤ H0[x∗

(t), λ(t), t] (67)

+H_x0[x∗

(t), λ(t), t][x(t) − x∗

(t)].

Using the Envelope Theorem, we can write

H[x(t), u(t), λ(t), t] ≤ H[x∗ (t), u∗ (t), λ(t), t] (68) +Hx[x ∗ (t), u∗ (t), λ(t), t][x(t) − x∗ (t)].

(55)

PROOF OF THEOREM 2.1 CONT.

By the definition of H and the adjoint equation,

F[x(t), u(t), t] + λ(t)f[x(t), u(t), t] (69) ≤ F[x∗ (t), u∗ (t), t] + λ(t)f[x∗ (t), u∗ (t), t] −λ˙ (t)[x(t) − x∗ (t)].

Using the state equation,

F[x∗ (t), u∗ (t), t] − F[x(t), u(t), t] ≥ λ˙ (t)[x(t) − x∗ (t)] + λ(t)[ ˙x(t) − x˙∗ (t)]. (70)

(56)

PROOF OF THEOREM 2.1 CONT.

Since S is differentiable and concave S[x(T ), T] ≤ S[x∗ (T), T] + Sx[x ∗ (T ), T][x(T) − x∗ (T )] (71) or, S[x∗ (T ), T] − S[x(T), T] + Sx[x ∗ (T), T][x(T) − x∗ (T)] ≥ 0. (72)

Integrating (70) and adding (72),

J(u∗ ) − J(u) + Sx[x ∗ (T), T][x(T) − x∗ (T )] ≥ λ(T )[x(T) − x∗ (T)] − λ(0)[x(0) − x∗ (0)], (73) J(u∗ ) ≥ J(u). (74)

(57)

EXAMPLE 2.6

• _{Let us show that the problems in Examples 2.1 and 2.2}

satisfy the sufficient conditions. We have from (36) and (61),

H0 = −x + λu0,

where u0 is given by (37). Since u0 is a function of λ only, H0(x, λ, t) is certainly concave in x for any t and λ. Since S(x, T ) = 0, the sufficient conditions hold.

(58)

FIXED-END-POINT PROBLEM

The terminal values of the state variables are completely

specified, i.e., x(T) = k ∈ E, where k is a vector of constants. Recall that the proof of the sufficient conditions requires (73) to hold, i.e., J(u∗ ) − J(u) + Sx[x ∗ (T), T][x(T) − x∗ (T )] ≥ λ(T )[x(T) − x∗ (T)] − λ(0)[x(0) − x∗ (0)]. Since x(T) − x∗

(T) = 0, the RHS vanishes regardless of the value of λ(T) in this case. This means that the sufficiency result would go through for any value of λ(T ). Not surprisingly,

therefore, the transversality condition is

(59)

SOLVING A TPBVP BY USING SPREADSHEET

SOFTWARE

• _{Example 2.7. Consider the problem:}

max J = Z 1 0 −1 2(x 2 ₊ _u2₎_dt subject to ˙ x = −x3 + u, x(0) = 5. (76)

(60)

SOLVING A TPBVP BY USING SPREADSHEET SOFTWARE CONT. • _Solution: ˙ λ = x + 3x2λ, λ(1) = 0. (77) ˙ x = −x3 + λ, x(0) = 5. (78) x(t + △t) = x(t) + [−x(t)3 + λ(t)] △ t, x(0) = 5, (79) λ(t + △t) = λ(t) + [x(t) + 3x(t)2λ(t)] △ t, λ(1) = 0. (80)

(61)

SOFTWARE CONT.

(62)

(63)

(64)

(65)

(66)

(67)