Dynamic Programming Method For Impulsive Control Problems

(1)

ABSTRACT

BALKEW, TESHOME MOGESSIE. Dynamic Programming Method For Impulsive Control Problems. (Under the direction of Dr. Negash Medhin.)

In many control systems changes in the dynamics occur unexpectedly or are ap-plied by a controller as needed. The time at which a controller implements changes is not necessarily known a priori. For example, many manufacturing systems and flight op-erations have complicated control systems, and changes in the control systems may be automatically implemented as needed in response to possibly unexpected external factors affecting the operations of the systems.

Public policy makers in health-care have to launch timely and cost effective policies to deal with disease epidemics long term and short term. In drug dosing strategy for HIV treatment how treatment interruption strategies stimulate the bodys own immune response and also control cost of treatment are important.

Modeling of such systems lead to discontinuities in the dynamics governing the evolutions of the systems. We consider control problems governed by differential equa-tions with discontinuities in the states at a sequence of points. The differential equaequa-tions are referred to as impulsive differential equations, and the control problems as impul-sive control problems. We consider general deterministic and nondeterministic impulimpul-sive control problems. Then, we consider the analysis of models coming from HIV treatment strategies, production planning, and asset allocation.

(2)

(3)

Dynamic Programming Method For Impulsive Control Problems

by

Teshome Mogessie Balkew

A dissertation submitted to the Graduate Faculty of North Carolina State University

in partial fulfillment of the requirements for the Degree of

Doctor of Philosophy

Operations Research

Raleigh, North Carolina 2015

APPROVED BY:

Dr. Min Kang Dr. Dmitry Zenkov

Dr. Zhilin Li Dr. Negash Medhin

(4)

DEDICATION

(5)

BIOGRAPHY

I was born in Ethiopia. I got my bachelor’s degree in mathematics from Addis Ababa University, Ethiopia in 1989, and my master’s degree in mathematics from the same university in 1998. After having my master’s degree I taught four years at Haramaya University. From 2002 to 2004 I was chair of the department of mathematics at Haramaya University. In September 2002 I started teaching at Addia Ababa University which lasted till July 2008. In 2008 I joined East Tennessee State University (ETSU) for a second master’s degree. I completed my master’s degree study at ETSU in 2010 and joined North Carolina State University the same year.

(6)

ACKNOWLEDGEMENTS

First and foremost I would like to extend my deepest gratitude to the gracious, merciful Heavenly Father Almighty God, Who helped me in every way in my study at NCSU. My special thanks and great appreciation goes to my advisor Dr. Negash G. Medhin. He gave me all necessary advising and encouragement for my thesis to go in the right direction. He has been instrumental at all stages of my thesis development. From his excellent coaching I learned a life long lesson.

I would also like to thank Professor Min Kang, Professor Dmitry Zenkov and Professor Zhilin Li for their willingness to be in my thesis committee and all their help. I am grateful to Professor Jeffrey S. Scroggs, Director of Financial Mathematics program, for suggesting the asset allocation model and as well as bringing to my attention applications of my method in finance.

(7)

LIST OF TABLES

(10)

LIST OF FIGURES

Figure 3.1 Cell count of the simulation. . . 53

Figure 3.2 Cell count for the first interval. . . 67

Figure 3.3 Cell count for the second interval. . . 67

Figure 3.4 Cell count for the third interval. . . 68

Figure 6.1 Bond fund growth. . . 117

Figure 6.2 Stock fund growth. . . 117

(11)

Chapter 1 Introduction

1.1 Stability Analysis Of HIV Model.

There are two immune responses in the human body: the humoral immune response and the cellular immune response [18]. The humoral immune response employs antibodies produced by B cells to attack antigens in body fluids, while the cellular immune response employs CD4+T cells to destroy body cells that have been infected with virus[19]. When an alien enters our system T-helper cells (CD4 +T) identify it and alert our bodies defense system so that the system forms some kind of defense mechanism. Unlike many other common diseases HIV virus attacks CD4+T cells. By killing and converting the T-cell to hosts of the virus the disease weakens our immune mechanism. Eventually as the cell count is not high enough the patient shows symptoms of AIDS.

Even after the patient develops AIDS our body defense systems do not stop fighting the disease. As the virus enters T-cells and change it to host; it will be attacked by CTLes(cytotoxic Lymphocytes effectors)which are deployed by CTLPS( cytotic T-lymphocytes precursors) which is coded to memorize the disease and convert to CTLes cells which kills the infected CD4+T cells [19].

(12)

count[34].

We start by considering mathematical model presented in [20] and [21]. We follow the same argument given by [19]. Both models use cell counts of uninfected, infected, CTLP and CTLe cells to study ”Long Term Non-Progressor” (LTNP). Although it is usual for HIV infected patients to progress to AIDS after a certain latent period, less than 1% of them still have a sufficient amount of T-helper cells and never develop AIDS. Thus, their immune system is able to fight off other diseases in spite of the HIV infection. They are called long-term non-progressors (LTNP) and may provide clues to the control of HIV without continued drugs [19].

As function of medication the model has at least two equilibrium points in the state space where both of them are asymptotically stable. One of them corresponds to AIDS and the other to LTNP. Without medication it is a common phenomenon for the system to end up in the basin of attraction of AIDS. Hence we rely on drug treatment. The drug treatment helps to guide the system to enter into the basin of attraction of LTNP and once it is there we terminate the medication [19].

In [20] and [21] such possibility has been suggested by the use of the structured treat-ment interruption, which is basically a switching scheme between zero and maximum medication. Since then, this problem has been dealt with by the various methodologies such as model predictive control [22], [23], [24], optimal control [25], [26], and an ap-proximation method [27]. On the other hand, a control theoretic approach has been used to determine when to initiate HIV therapy [28], and to estimate the parameter of HIV models [29], [30].

(13)

1.2 HIV Infection Mathematical Model.

Mathematical model of HIV infection without drug application that we consider is [20] ˙

x=λ−dx−βxy

˙

y=βxy−ay−pyz

˙

w=cxyw−cqyw−bw

˙

z =cqyw−hz

˙

u=−u

(1.1) where,

x is number of healthy CD4+T cells. y number of unhealthy CD4+T cells. w number of CTLP cells (memory cells). z number of CTLe cells.

The description of the model is as follows:

Uninfected CD4+ T cells are produced at a rateλ, die at a rated, and become infected by free virus at a rate β. Infected cells decay at a ratea and are killed by CTL effectors at a rate p. Proliferation of the CTLp population is given by cxyw and is proportional to both virus load y and the number of uninfected T helper cells x. CTLp die at a rate

b and differentiate into effectors at a rate c. CTL effectors die at a rate h [21]. An HIV infection mathematical model with drug treatment is [19]

˙

x=λ−dx−(1−ηu)βxy

˙

y= (1−ηu)βxy−ay−pyz

˙

w=cxyw−cqyw−bw

˙

z =cqyw−hz

(14)

we use the notation η∗ = 1−ηu.

1.3 Stability Of Equilibrium Points Of The Model.

In this section we determine the equilibrium points of the model as a function of drug dose η∗. We set ˙x = ˙y = ˙w = ˙z = 0 to have the equilibrium points. We do have four equilibrium points as a function of drug dose where three of them have biological meaning but not the fourth one.

1. HIV free equilibrium point XA(η∗) = (xA, yA, wA, zA).

xA =

λ

d, yA =wA=zA= 0. (1.3)

2. AIDS stage XB(η∗) = (xB, yB, wB, zB).

xB =

a

η∗_β, yB =

λβ−da

aη∗_β , wB =zB= 0. (1.4)

3. Long term non-progressor XC(η∗) = (xC, yC, wC, zC). LetK := [c(λ+dq)−bη∗β]2₋₄_c2_λqd

xC =

[c(λ+dq)−bη∗β] +√K

2cd , yC =

b c(xC−q)

, wC =

hzC

cqyC

,

zC =

η∗βxC−a

p . (1.5)

4. The fourth equilibrium point XD(η∗) = (xD, yD, wD, zD).

xD =

[c(λ+dq)−bη∗β]−√K

2cd , yD =

b c(xD −q)

, wD =

hzD

cqyD

,

zD =

η∗βxD −a

p . (1.6)

(15)

1. Assumption 1:

d < a. (1.7)

b < h. (1.8)

q < λ

d. (1.9)

c > 4abd

(λ−dq)2. (1.10)

β < c(

√

λ−√dq)2

b . (1.11)

β > ac(λ+dq)−

p

a2_c2₍_λ₊_dq₎2₋₄_a2_cd₍_ad₊_cqλ₎

2(ab+cqλ) . (1.12)

As it is shown in [20] the death rate of uninfected cells is smaller than that of the infected cells and the life span of CTLP ( memory cells) is longer than that of CTLe cells. It reasonable to assume (1.7) and (1.8).

Inequality (1.9) requires that the differentiation of CTLp into CTL effectors occurs slowly. As a matter of fact, unless (1.9) is met, the LTNP equilibrium does not exist.

If the immune responsiveness clies below a threshold, treatment cannot result in the establishment of CTL memory [20] which establishes (1.10). Inequality (1.10) can be taken as a possible lower bound forc. The valueβ known as infection ratio, describes the efficacy of infection, including the rate at which virus particles find uninfected cells, the rate of virus entry, and the rate of successful infection [19]. When β is too small no treatment is needed for the system to go to LTNP. Also if it is too large then the virus replicates too fast and immune impairment occurs. So inequalities (1.11) and (1.12) follow.

2. Assumption 2:

(16)

We need the following values for stability analysis

η₁∗ := ad

βλ. (1.13)

η₂∗ := ac(λ+dq)−

p

(2(ab+cqλ)β) . (1.14)

η₃∗ := ac(λ+dq) +

p

2(ab+cqλ)β . (1.15)

Lemma 1.3.1 Assumption 1 implies the following:

1. A1(η∗) := [c(λ+dq)−bη∗β]2−4c2λqd > 0 for all η∗ ∈[0,1] 2. A2 :=a2c2(λ+dq)2−4a2cd(ab+cqλ)>0.

3. 0< η₁∗ < η₂∗ < η₃∗ and η∗₂ <1.

Proof:

(1): Since A1(η∗) = [c(λ+dq]2 −4c2λqd−2bη∗βc(λ+dq) + (bη∗β)2 = (bβ)2(η∗)2− (2bβc(λ +dq))η∗ +c2(λ −dq)2. A1(η∗) is a a polynomial of degree two in η∗, whose discriminantD:=b2β2c2[(λ+dq)2−(λ−dq)2], is always positive. Therefore, the equation

A1(η∗) = 0 has two distinct roots.

ηa : =

c(√λ−√dq)2

bβ .

ηb : =

c(√λ+√dq)2

bβ . (1.16)

Inequality (1.11) implies

c bβ(

√

(17)

It follows that ηb > ηa>1, which results in A1(η∗)>0 for all η∗ ∈[0,1]. (2): It easy to see that (1.10) implies A2 >0. In fact,

b < c(λ−dq)

2 4ad

⇔4abcd < c2(λ−dq)2

⇔4cd(ab+cqλ)< c2(λ+dq)2

⇔0< a2c2(λ+dq)2−4a2cd(ab+cqλ) (3): Clearly 0< η₁∗ and η∗₂ < η∗₃.

We show that η₁∗ < η₂∗. It is equivalent to showing

d λ <

c(λ+dq)−

√

K a

2(ab+cqd) (1.17)

Rearranging terms we have

λ a

√

K < cλ2−cλdq−2abd. (1.18)

We claim that the right hand side of (1.18) is positive. By (1.9), we have

c(λ−dq)2 4ad −

cλ(λ−dq) 2ad =

c(λ−dq)2₋₂_c₍_λ2₋_dqλ₎ 4ab

= c(−λ

2₊_d2_q2₎

4ad <0 (1.19)

which implies, together with (1.10),

b < c(λ−dq)

2 4ad <

cλ(λ−dq)

2ad (1.20)

Thus,

cλ2−cλdq−2abd > 0.

Therefore, it is sufficient to show the claim, by squaring both sides of (1.18)

(18)

In fact inequality (1.20) is proved because

(cλ2−cλdq−2abd)2−λ2(c2(λ+dq)2−4cd(ab+cqλ)) = 4a2b2d2+ 4abcd2qλ >0 Finally, η∗₂ <1 can easily be shown using (1.12).

Lemma 1.3.2 Under Assumption 1 the following hold for 0< η ≤1 1. xC > xD > q.

2. yD > yC >0.

3. zC ≶0 and wC ≤0 if and only if η≶η∗2.

4. If η∗₃ <1 , then zD ≶0 and wD ≶0 if and only if η ≶η3∗. If η

∗

3 ≥1, then zD ≤0 and

wD ≤0.

Proof:

(1): From (1.11), we have c(λ+dq)−2c√λdq−bβ >0 , which implies together with (1.9)

c(λ+dq)−bβ >2cdq (1.22)

By what we determined in (1.5) and (1.6) it is clear that xC > xD. We need only show

xD > q. Note that

dxD

dη =

−bβ

2cd(1−

[c(λ+dq)−bηβ]

[pc(λ+dq)−bηβ]2₋₄_c2_λdq) (1.23) From (1.21), and for all 0< η ≤1

(19)

which leads to dxD

dη >0. Thus, sincexD increases withη and

xD|η=0 =

[c(λ+dq)]−p[c(λ+dq)]2₋₄_c2_λqd 2cd

= [c(λ+dq)]−

p

[c(λ−dq)]2 2cd

= [c(λ+dq)]−[c(λ−dq)] 2cd

=q (1.25)

(2): From (1.5) and (1.6) we know thatyC = _c(xb

C−q) andyD =

b

c(xD−q). The claim follows

directly from part (1).

(3): We will show that zC is negative when η = 0, and becomes zero only at η2∗, and is positive for η slightly larger than η₂∗. This necessarily implies that zC ≶0⇔η≶η2∗. By the definition of zC and zD , it is obvious that they are negative when η= 0. To show thatzC is zero only at η2∗, observe that the equation

z(C,D) =

(ηβx(C,D)−a)

p = 0. (1.26)

is equivalent to ±√K = 2cd_ηβa −[c(λ+dq)−bηβ] ( (+) corresponds to (C) and (−) corresponds to (D)). By squaring both sides we obtain

[c(λ+dq)−bηβ]2−4c2λqd= (2cd a

ηβ −[c(λ+dq)−bηβ])

2

,

which can be rearranged as a quadratic equation forη,

[4c2λqd+ 4abcd]β2η2+ [−4ac2d(λ+dq]βη+ 4a2c2d2 = 0 (1.27) That is,

[c2λqd+abcd]β2η2+ [−ac2d(λ+dq]βη+a2c2d2 = 0 (1.28) whose roots are give by η₂∗ and η₃∗ given in (1.14) and (1.15). Thus it follows that either

(20)

it follows that, for η∗₂₊ slightly larger than η₂∗

[c2λqd+abcd]β2η2+ [−ac2d(λ+dq]βη+a2c2d2 <0 (1.29) which, in turn, implies zC(η2+∗ )>0.

Finally, the claim on wC directly follows from part (2).

(4): It suffices to show that, under the condition that η < ηa, zD ≶0 and wD ≶0 if and only if η_≶η∗₃, where ηa := c(

√

λ−√dq)2

bβ .

To see this observe thatzC|η=ηa >0 because of part (3) andη

∗

2 <1< ηa. Moreover, since

zD|η=ηa =zC|η=ηa >0 the proof is similar to part (3) and is omitted. Furthermore, since

zD|η=0 <0, zC|η=η∗₃ = 0 and zD|η=ηa >0, it is seen that η

∗

3 < ηa.

Theorem 1.3.3 Under Assumption 1

XA(η1∗) =XB(η1∗), XB(η∗2) =XC(η∗2) and XB(η3∗) =XD(η3∗). Moreover, no other equilib-rium point coalesce with each other elsewhere.

Proof:

Using (1.3) and (1.4) by setting xA = xB, yA = yB, wA = wB, zA = zB, we obtain

xB(η∗) = (λ_d,0,0,0) if and only if η∗ = _βλda, which impliesXA(η∗₁) = XB(η∗₁).

Next, from Lemma 1.3.2(3) zC = 0 andwC = 0 if and only if η=η∗2. Moreover, it can easily be shown that

xC(η∗2) =

a

η₂∗β =xB(η ∗

2) (1.30)

and

yC(η2∗) =

λη₂∗β−da

aη₂∗β =yB(η ∗

2) (1.31)

Hence xB(η∗2) = xC(η∗2). It can be shown similarly that xB(η∗3) = xD(η3∗). Finally, it follows from Lemma 1.3.2 that no other equilibrium points coalesce elsewhere.

Theorem 1.3.4 Under Assumption 1 we have the following.

1. XA(η∗) is locally exponentially stable if η∈[0, η1∗) and unstable if η ∈(η

∗

1,1]. 2. XB(η∗) is locally exponentially stable if η ∈ (η∗₁, η₂∗)∪(η₃∗,1] and unstable if η∗ ∈

(0, η₁∗)∪(η₂∗, η₃∗).

(21)

Proof:

(1): Let (1.2) be written as ˙X := F(X, η). The Jacobian matrix of F(X, η) at xA is upper triangular, and its diagonal elements are {−d,_(dηβλ₋_a),−b,−h}. Thus xA(η∗) is locally exponentially stable for 0≤η < η∗₁ = _βλda and unstable for η∗ > η∗₁.

(2): The Jacobian matrix of F(X, η) at X =XB(η) is given by

     

−ηβλ_a −a 0 0

ηβλ−da

a 0 0 −

p(ηβλ−da) aηβ

0 0 J33 0

0 0 cq(ηβλ_aηβ−da) −h

     

where we set

J33:=

c(ηβλ−da)

η2_β2 −

cq(ηβλ−da)

aηβ −b.

The characteristic equation for the upper left 2×2 block is

s2+ηβλ

a s+ (ηβλ−da) = 0

which is Hurwitz matrix if and only if η = η₁∗. Now, the stability of xB for η ∈ (η∗1,1) is determined by the (3,3)- component of the above Jacobian matrix, whose negativity implies stability. In fact, it is negative if and only if −aη2_β2_J

33 > 0, which is a second order polynomial in η. The discriminant of −aη2_β2_J

33= 0 is given by

D=a2c2(λ+dq)2β2−4a2cd(ab+cqλ)β2

which is positive, and hence −aη2_β2_J

33 = 0 has two distinct roots. However, a simple calculation yields that −aη2_β2_J

33|η=η∗₂ = −aη2β2J33|η=η∗₃ = 0, so that −aη2β2J33 > 0 if

η < η∗₂ or η < η₃∗

(3): We first observe that from (1.17) that

xC(ηa) =

c(λ+dq)−bβηa

2cd =

r

λq

(22)

and from the proof of Lemma 1.3.1 (1) that A1(η) is decreasing on 0 < η < ηa and

ηa>1. This implies that xC(η) is strictly decreasing for 0< η < ηa, so that

xC(η)> xC(ηa) =

r

λq

d , ∀ 0< η < ηa. (1.33)

Now for notational simplicityxC, yC, wC andzC are denoted byx, y, wandz, respectively. The Jacobian matrix of F(X, η) at x=xC(η) is given by



    

−d−ηβy −ηβx 0 0

ηβy 0 0 −py

cyw cxw−cqw 0 0

0 cqw cqy −h



    

whose characteristic equation is

s4+A1s3 +A2s2+A3s+A4 = 0, (1.34) where,

A1 =d+ηβy+h

A2 =η2β2yx+dh+cpqwy+ηβhy

A3 =dcpqwy+c2pq(x−q)wy2+ηβcpqwy2+η2β2hyx

A4 =dc2pq(x−q)wy2−ηβc2pq2wy3. (1.35) Sincex, y andware functions of η, it is very tedious to prove that the Jacobian matrix is Hurwitz for η∗₂ < η ≤ 1 by applying Routh-Hurwitz criterion to (1.33). Instead, we first prove that the Jacobian matrix is Hurwitz just for η slightly larger than η2, and then show that no eigenvalues cross the imaginary axis for η₂∗ < η ≤ 1. This completes the proof since the eigenvalues are continuous functions of η.

We first show that all theAi’s,i= 1,2,3,4 are positive forη2∗ < η ≤1. From Lemma 1.3.2, x > q, y >0, w > 0 and z > 0 if η₂∗ < η ≤1. So that A1 >0, A2 >0 and A3 >0. ForA4, it follows that from (1.32) that

A4

c2_pqwy2 = (x−q)d−ηβqy = (x−q)d−q(

λ

x−d) =

dx2₋_λq

(23)

for 0< η < ηa.

Hence,Lemma 1.3.2(3) implies that A4 >0 ifη2∗ < η ≤1 andA4 <0 if 0< η < η2∗. (The latter implies thatxC is unstable on (0, η2∗)).

Now, since ∂F_∂X(xC(η2∗)) = ∂F ∂X(xB(η

∗

2)) and J33|η=η∗₂ = 0. It follows from part (2) that ∂F

∂X(xC(η

∗

2)) has three eigenvalues in the open half-plane and one is at 0. Therefore, for η slightly larger than η₂∗, three eigenvalues except zero remain in the open left-half plane while the zero eigenvalue also moves into the open left-hand plane because all A0_is are positive in (1.33).

The fact that A4 >0 for all η2∗ < η ≤1 also implies that no eigenvalues pass through the origin for all η∗₂ < η ≤ 1. Therefore, it is left to show that no eigenvalues cross the imaginary axis. To show this, suppose not, i.e there exists a Ω > 0 such that s = jΩ satisfies (1.33). Then,

Ω4−A2Ω2+A4 = 0 and A1Ω3−A3Ω2 = 0 which leads to A23

A2 1 −A2

A3

A1 +A4 = 0. Multiply A

2

1 on both sides and collecting w-term, we obtain

B1w2+B2w+B3 = 0 where ( after some manipulation using ηβxy=λ−dx)

B1 = (cpqy)2(b−h)(b+ηβy+d)

B2 =cpqy[2η2β2hyx(b+ηβ+d + (db−ηβcqy2)(d+ηβy+h)2

−(d+ηβy+h)(η2β2yx+dh+ηβyh)(b+ηβy+d)

−(d+ηβy+h)η2β2hxy]

B3 = η2β2hxy[η2β2hxy−(h+ηβy+d)(η2β62yx+dh+ηβyh)]. (1.37) Since all variables are positive andb−h <0 by (1.8), it follows thatB1 <0 andB2 <0. Moreover, it can be shown that

(24)

where,

r=cpqηβy2_[_eta3_β3_y2_x₊_η2_β3_y2_h_{+ +}_ηβyh2_{+ 2}_dη2_β2_yx₊_d2_h_{+ 2}_dηβyh_{+ 2}_dh2₊_dηβxb₊

dηβxb+ηβyhb+h2_b₊_d2_ηβx_{+ 2}_cqydh₊_η2_β2_cry3_{+ 2}_ηβcqy2_h₊_cqyd2_{+ 2}_ηβcqy2_d₊_cqyh2_] and r >0.

Thus, B2 < 0 because b − h < 0. Thus, the positivity of w for η∗2 < η ≤ 1 implies

B1w2 +b2w+B3 <0 which is contradictory to the existence of Ω>0.

The proof of the following Corollary directly follows from Theorem 1.3.4 and the definition of transcritical bifurcation.

(25)

Chapter 2 Impulsive Control Problems.

2.1 Impulsive Control Problems.

Many evolution processes are characterized by the fact that at certain moments of time they experience a change of state abruptly. These processes are subject to short-term perturbation whose duration is negligible in comparison with the duration of the process. Consequently, it is natural to assume that these perturbations act instantaneously, that is in the form of impulse [2]. For such an idealization, it becomes necessary to study dynamical system with discontinuous trajectories, and they might be called differential equations with impulses or impulsive differential equations [1].

Impulsive control system (control system with impulse effects) is a control paradigm based on impulsive differential equation. In an impulsive control system, the underlying dynamic system should have at least one ”impulsively ” changeable state variable.

Theories involving impulsive control systems have been widely studied. In these sys-tems, the controls are inputs which occurs impulsively. Thus the state equation are or-dinary differential equations which are continuous except at certain instants where the impulses occur [34].

(26)

First, letti,i= 1,2,3...nbe the instants of applying the impulsive controls, where 0 = t0 < t1 < t2 < ... < tn−1 < tn.

In the initial interval [t0, t1] the dynamic equation is given by ˙

x1(t) =f1(x1(t), u1(t)), t0 < t < t1

x1(t0) =x0

and for i= 2,3, ...n the dynamics equation is ˙

xi(t) =fi(xi(t), ui(t)), ti−1 < t < ti

xi(ti−1) =hi(xi−1(ti−1))ci+xi−1(ti−1) We assume that,

fi : [ti−1, ti]×Rn×Rk−→R Φi : [ti−1, ti]×Rn×Rk−→R

hi :Rn −→Rn×Rn

Θ :Rn −→R (2.1)

The functions fi, Φi,hi and Θ are continuously differentiable with respect toxand ufor fixed time t.

The objective function is k

X

i=1

Z ti

ti−1

Φi(xi(ti), ui(ti))dt+ Θ(xn(tn)) (2.2)

The control problem is given by

min J(¯x1,u¯1, ...,x¯n,u¯n) = k

X

i=1

Z ti

ti−1

(27)

subject to ˙

xi(t) = fi(xi(t), ui(t)), ti−1 < t < ti

xi(ti−1) = hi(xi−1(ti−1))ci+xi−1(ti−1) (P1)

i= 1,2,3, ...n

2.2 Solution Method.

In this section we apply dynamic programming approach to solve the control problem (P1). This method has an advantage over considering the entire problem simultaneously. It reduces the number of variables significantly and avoids some mathematical difficulties of optimizing a non-continuous function over the entire interval.

We assume the control problem (P1) has solution (¯c1,¯c2, ...,¯cn,u¯1,u¯2, ...,u¯n). We de-note the corresponding trajectories by ¯xi(t),i= 1,2, ..., n. Let U =U1×U2×...×Un be the control set. Assume that U is a convex set.

1. We start with the last interval [tn−1, tn]. We also write tf for tn. The objective function in this interval is

Jn(¯x1,u¯1, ...,x¯n−1,u¯n−1, xn, un) =

Z tn

tn−1

Φn(xn(tn), un(tn))dt+ Θ(xf(tf))}.

The control problem is given by

min Jn(¯x1,u¯1, ...,x¯n−1,u¯n−1, xn, un) =

Z tn

tn−1

Φn(xn(tn), un(tn))dt+ Θ(xf(tf))

subject to ˙

xn(t) = fn(xn(t), un(t)) tn−1 < t < tn

xn(tn−1) = hn(¯xn−1(tn−1))cn+ ¯xn−1(tn−1) (P2)

(28)

this interval is given by

Jn−1(¯x1,u¯1, ..., xn−2,u¯n−1, xn−1, un−1, xn,u¯n) =

Z tn−1

tn−2

Φn−1(xn−1(t), un−1(t))dt

+

Z tn

tn−1

Φn(xn(tn),u¯n(tn))dt+ Θ(xn(tn)) (2.3) Control problem in this interval is

min Jn−1(¯x1,u¯1, ..., xn−1, un−1, xn,u¯n) =

Z tn−1

tn−2

+

Z tn

tn−1

subject to ˙

xn−1(t) = fn−1(xn−1(t), un−1(t)), tn−2 < t < tn−1

xn−1(tn−2) = hn−1(¯xn−2(tn−2))cn−1+ ¯xn−2(tn−2) (P3)

2.2.1 Controls Between Impulses (u

i

’s).

As shown inSection 2.2we have now nintervals andnoptimal control problems, one in each interval. We use dynamic programming methodology to study control problem (P1).

Let us perturb the optimal control ¯un in the last interval [tn−1, tn] .

(29)

xn,θ−x¯n(t) =x0 +

Z t

tn−1

fn(xn,θ, uθ)dt−x0 −

Z t

tn−1

fn(¯xn(t),u¯n(t))dt

=

Z t

tn−1

(fn,θ(xn,θ, un,θ)−fn(¯xn(t),u¯n(t)))dt

=

Z t

tn−1

Z 1

0

[fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))(xn,θ −x¯n)

+fn,un(¯xn+λ(xn,θ−x¯n), un+λ(¯un,θ−u¯n))(un,θ−u¯n)]dλ dt

Then, we have

xn,θ −x¯n(t)

θ =

Z t

tn−1

Z 1

0

[fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))

(xn,θ−xn)

θ

+fn,un(¯xn+λ(xn,θ−x¯n), un+λ(¯un,θ−u¯n))vn]dλ dt

The following Lemma is used to determine the limit of xn,θ−¯xn(t)

θ as θ tends to zero.

Lemma 2.2.1 We define a function δxn(t) by the system of differential equations

dδxn(t) = {fn,xn(¯xn(t),u¯n(t))δxn(t) +fn,un(¯xn(t),u¯n(t))vn(t)}dt

δxn(tn−1) = 0

Then,

xn,θ −x¯n

θ −→δxn(t)

uniformly.

Proof:

Define

yθ :=

xn,θ(t)−x¯n(t)

(30)

Then, we have

yθ(t) =

Z t

tn−1

(fn,θ(xn,θ, un,θ)−fn(¯xn(t),u¯n(t)))

θ

−fn,xn(¯xn,u¯n)δxn(t)−fn,un(¯xn,u¯n)vn)dt

=

Z t tn−1

Z 1 0

{(fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))yθ

+ [(fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))−fn,xn(¯xn,u¯n))]δxn(t)

+ (fn,un(¯xn+λ(xn,θ −x¯n), un+λ(un,θ−u¯n)))−fn,un(¯xn,u¯n)vn}dλ dt (2.4)

Since the functions δxn and v belongs to L2, we can take sufficiently large M, such that both of their 2−norms are less than or equal to M. Also assume that |fxn| ≤ M. By

Holder’s inequality, we have

|yθ|2 ≤(

Z t

tn−1

Z 1

0

{|(fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))||yθ|

+ [|(fn,xn(¯xn+λ(xn,θ −x¯n),u¯n+λ(un,θ−u¯n)−fn,xn(¯xn,u¯n))|]|δxn(t)|

+|(fn,un(¯xn+λ(xn,θ−x¯n), un+λ(un,θ−u¯n)))−fn,un(¯xn,u¯n)||vn|}dλ ds)

2

≤2(

Z t

tn−1

Z 1

0

{|(fn,xn(¯xn+λ(xn,θ −x¯n),u¯n+λ(un,θ−u¯n))||yθ|dλ ds)

2

+ 2(

Z t tn−1

Z 1 0

[|(fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ −u¯n)−fn,xn(¯xn,u¯n))|]|δxn(t)|dλ ds)

2

+ 2(

Z t

tn−1

Z 1

0

|(fn,un(¯xn+λ(xn,θ −x¯n), un+λ(un,θ−u¯n)))−fn,un(¯xn,u¯n)||vn|}dλ ds)

2

≤2M2t2_n

Z t

tn−1

|yθ|ds

+ 2M2(

Z t

tn−1

[|(fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ −u¯n)−fn,xn(¯xn,u¯n))|]dλ ds)

2

+ 2M2

Z t

tn−1

(|(fn,un(¯xn+λ(xn,θ−x¯n), un+λ(un,θ−u¯n)))−fn,un(¯xn,u¯n)|

2_{dλ ds}

≤2M2t2_n

Z tn

tn−1

(31)

where,

Γ(θ) =

Z tn

tn−1

Z 1 0

|fn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n)−fn,xn(¯xn,u¯n)|dλ ds

+

Z tn

tn−1

Z 1

0

|fn,un(¯xn+λ(xn,θ −x¯n), un+λ(un,θ−u¯n))−fn,un(¯xn,u¯n)|})

2

dλ ds

Clearly Γ(θ) tends to zero when θ tends to zero.

By Gronwall’s inequality, we have

|yθ(t)|2 ≤2M2Γ(θ)e2M

2_t2

nt_, ∀_t ∈ _[_t

n−1, tn]

The right hand side tends to zero as θ tends to zero. So, yθ tends to zero. That is,

xn,θ−x¯n

θ −→δxn(t).

Now we can write the variation of the dynamics with respect to the variation of the optimal control ¯un as

d

dtδxn(t) = fn,xn(¯xn(t),u¯n(t))δxn(t) +fn,un(¯xn(t),u¯n(t))vn

Under the same perturbation of the optimal control ¯un the cost Jn({u¯n}) varies as

Jn(un,θ)−Jn(¯un) =Θ(xθ,f(tf))−Θ(¯xf(tf)) +

Z tn

tn−1

Φn(xn,θ(t), un,θ(t))−Φn(¯xn(t),u¯n(t)))dt

=

Z 1

0

Θxn(¯xf(tf) +λ(xθ,f(tf)−x¯n))(xn,θ(tf)−x¯n(tf))dλ

+

Z tn

tn−1

Z 1

0

(Φn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ −u¯n)(xn,θ−x¯n)dλ dt

+

Z tn

tn−1

Z 1

0

(32)

Then we divide both sides by θ Jn(un,θ)−Jn(¯un)

θ =

Z 1 0

Θxn(¯xn(tn) +λ(xθ,n(tn)−x¯n))

(xn,θ(tn)−x¯n(tn))

θ dλ

+

Z tn

tn−1

Z 1

0

Φn,xn(¯xn+λ(xn,θ−x¯n),u¯n+λ(un,θ−u¯n))

(xn,θ−x¯n)

θ dλ dt

+

Z tn

n−1

Z 1 0

Φn,un(¯xn+λ(xn,θ −x¯n), un+λ(un,θ−u¯n))vndλ dt

Letting θ tend to zero, by Lemma 2.2.1, we obtain the Gˆateaux derivative of

Jn(un(.)) along the direction of vn.

Proposition 2.2.2 The Gˆateaux derivative ofJn(un(.)) in the direction of vn is

δJn(¯x1,u¯1, ...,x¯n−1,u¯n−1, xn, un) =

Z tn

tn−1

(Φn,xnδxn(t) + Φn,unvn)dt+ Θxn(xn(tn))δxn(tn)

(2.5) Now we are in a position to define dual variable for the costate equation. Consider the equation

dΨ

dt =fn,xnΨ +ξ (2.6)

Ψ(tn−1) = 0

where,ξ is a function in L2_{. For each}_ξ _{we can find one and only one Ψ in} _L2_{. Then, for} each ξ, we consider the map

ξ 7−→ Z tn

tn−1

Φn,xn.Ψ(t)dt+ Θn,xn(x(tn)).Ψ(tn)

This map is a continuous linear functional. By Riesz representation theorem, there is one and only one function Pn(t) in L2 such that for each ξ ∈ L2 we have

Z tn

tn−1

Pn(t).ξ(t)dt =

Z tn

tn−1

Φn,xn.Ψ(t)dt+ Θn,xn(x(tn)).Ψ(tn) (2.7)

(33)

We define the Hamiltonian in the last interval [tn−1, tn] by

Hn(tn, xn, un, Pn) = Φn+fn.Pn (2.8) Let,

γn = Θxn(xn(tn)) (2.9)

Let us perturb ¯un as ¯un+θvn in the Hamiltonian. Setting un,θ = ¯un+θvn we have

Z tn

tn−1

(Hn(t, xn,θ, un,θ, Pn)−Hn(t,x¯n,u¯n, Pn))dt =

Z tn

tn−1

(Φn,θ(xn,θ(t), un.θ) +fn,xn(xn,θ(t), un,θ(t))

−Φn(¯xn(t),u¯n(t))Pn(t)

−fn(¯xn(t),u¯n(t))).Pn(t)dt (2.10)

Dividing both sides of (2.10) by θ and letting θ−→0+ we obtain

Z tn

tn−1

(Φn,unvn+fn,unvn).Pndt (2.11)

Using (2.7) we have

Z tn

tn−1

Pnfn,unvndt=

Z tn

tn−1

Φn,xnδxn(t)dt+ Θxfδxn(tf) (2.12)

Comparing with

dΨ

dt =fn,xnΨ +ξ (2.13)

Ψ(tn−1) = 0

in (2.6) where ξ =fn,xnvn and Ψ =δxn we conclude that

Z tn

tn−1

Hn,un(¯x(t),u¯n(t), Pn(t)).v(t)dt =

Z tn

tn−1

Φn,unvndt+

Z tn

tn−1

Φn,xnδxn(t)dt+ Θδxn(tn)

(34)

From (2.14) we see that for any control v

Z tn

tn−1

(Hn(t, xn, v, Pn)−Hn(t, xn,u¯n, Pn))dt ≥0 (2.15)

Proposition 2.2.3 For (P2), we have the minimum principle

Hn,un(t,x¯n(t),u¯n(t), Pn(t))(v−u¯n(t))≥0

For all possible v ∈Un and t ∈[tn−1, tn].

Proof:

Letv be any fixed vector in Un, t a Lebesgue point of ¯U in (tn−1, tn), and >0, such that (t, t+)∈[tn−1, tn]. Then, we consider the function

¯

v(s) =

(

v, s∈(t, t+) ¯

un, otherwise

So, ¯v(.) belongs to L2 _{and it is an admissible control. We have}

Z t+

t

Hn,un(t,x¯n(t),u¯n(t), Pn(t))(v−u¯n(t))dt

=

Z tn

tn−1

Hn,un(t,x¯n(t),u¯n(t), Pn(t))(¯v −u¯n(t))dt≥0

Letting tends to zero 1

Z t+

t

Hn,un(t,x¯n(t),u¯n(t), Pn(t))(v−u¯n(t))dt−→Hn,un(t,x¯n(t),u¯n(t), Pn(t))(v−u¯n(t)), a.e

So, we conclude that

Hn,un(t,x¯n(t),u¯n(t), Pn(t))(v−u¯n(t))≥0, a.e

for t∈[tn−1, tn].

(35)

Proposition 2.2.4 Pn(t) is the adjoint variable we defined by Riesz representation the-orem, if and only if it satisfies the costate equation

dPn=−Hn,xn(¯xn,u¯n, Pn)dt =−{Φn,xn(¯xn,u¯n) +fn,xn(¯xn,u¯n)

t

.Pn}dt

Pn(tf) = γn

Proof:

First, we supposePn(t) is a solution of the costate equation and prove that it satisfies the Riesz representation theorem.

Using integration by parts, we have

d(Pn(t).Ψ(t)) =Ψ(t).dPn(t) +Pn(t).dΨ(t)

=Ψ(t).− {Φn,xn(¯xn(t),u¯n(t))dt+ (fn,xn(t))

t_.P n(t)} +Pn(t).{fn,xn(¯xn(t),u¯n(t))Ψ(t)dt+ξ(t)dt}

=−Φn,xn(¯xn(t),u¯n(t)).Ψ(t) +Pn(t).ξ(t)dt

Since Ψ(tn−1) = 0 and Pn(tn) = γn , we have Θxn(¯xn(tn)).Ψ(tn) =Pn(tn).Ψ(tn)

=Pn(tn−1).Ψ(tn−1) +

Z tn

tn−1

{−Φn,xn(¯xn(t),u¯n(t)).Ψ(t) +Pn(t).ξ(t)}dt

=

Z tn

tn−1

{−Φn,xn(¯xn(t),u¯n(t)).Ψ(t) +Pn(t).ξ(t)}dt

Then,

Z tn

tn−1

Pn(t).ξ(t)dt=

Z tn

tn−1

Φn,xn(¯xn(t),u¯n(t)).Ψ(t)dt+ Θxn(¯xn(tn)).Ψ(tn)dt

(36)

and prove that it must satisfies the costate equation. Consider the matrix equation

dZ(t) =fn,xnZ(t)dt

Z(tn−1) =I where, I is the identity matrix. Then, from (2.6)

Ψ(t) =Z(t)Ψ(tn−1) +Z(t)

Z t

tn−1

Z(s)−1(s)ξ(s)ds

=Z(t)

Z t

tn−1

Z(s)−1(s)ξ(s)ds

By Integration by parts we have

Z tn

tn−1

Pn(t).ξ(t)dt=

Z tn

tn−1

Φn,xn(¯xn,u¯n(t))Ψ(t)dt+ Θxn(¯xf(tf)).Ψ(tf)

=

Z tn

tn−1

Φn,xn(¯xn,u¯n(t)).{Z(t)

Z t

tn−1

Z(s)−1(s)Ψ(s)ds}dt

+ Θxn(¯xf(tf)){Z(tf)

Z tn

tn−1

Z(s)−1(s)ξ(s)ds}

=

Z tn

tn−1

Z t

tn−1

Φn,xn(¯xn,u¯n(t)).{Z(t)Z(s)

−1₍_s₎_ξ₍_s₎_}_ds + Θxn(¯xf(tf)).{Z(tf)Z(t)

−1₍_t₎_ξ₍_t₎_}_dt =

Z tn

tn−1

Z tn

t

Φn,xn(¯xn(t),u¯n(t)).{Z(s)Z

−1₍_t₎_ξ₍_t₎_}_ds + Θxn(¯xf(tf)).{Z(tf)Z(t)

−1₍_t₎_ξ₍_t₎_}_dt =

Z tn

tn−1

Ψ(t).(Z(t)−1)t{ Z tn

t

Zt(s)Φn,xn(¯xn(s),u¯n(s))

+Z(tn)Θxn(¯xf(tf))}dt

=

Z tn

tn−1

ξ(t).Z(t)−1)t{− Z t

tn−1

Zt(s)Φn,xn(¯xn(s),u¯n(s))ds

(37)

for all ξ(t)∈L2_{. Then,}

Pn(t) =(Z−1(t))t{−

Z t

tn−1

(Ψt(s))Φn,xn(¯xn(s),u¯n(s))ds

+ Ψ(tf)Θxn(¯xf(tf))}

By letting t=tn, we obtain the boundary condition

Pn(tn) = (Ψ−1(t))t{−

Z tn

tn−1

(Ψt(s))Φn,xn(¯xn(s),u¯n(s))ds+ Ψ(tf)Θxn(¯xf(tf))}

=Θxn(¯xf(tf))

Since dZ(t)_dt =fn,xnZ, we have

dZt

dt =Z t₍_f

n,xn)

t_{. Then}

ZtPn(t) =−

Z t tn−1

ZtΦn,xnds+Z(tf)

t_Θ

xn(¯xf(tf))

Then,

dZ(t)

dt Pn(t) +Z

tdPn(t)

dt =−Z

t_Φ

n,xn(¯xn,u¯n)

Therefore,

ZtdPn(t)

dt =−Z

t

Φn,xn −

dZ(t)

dt Pn(t)

=−ZtΦn,xn −Z

t_ft n,xnPn

Thus,

dPn(t)

dt =−Φn,xn(¯xn(t),u¯n(t))−fn,xn(¯xn(t),u¯n(t))

t

Pn(t) = −Hn,xn(t,x¯n,u¯n, Pn).

Let us move one step backwards to the interval [tn−2, tn−1]. The control problem is

min Jn−1(¯x1,u¯1, ..., xn−1, un−1, xn,u¯n) =

Z tn−1

tn−2

+

Z tn

tn−1

(38)

subject to ˙

xn−1(tn−2) = hn−1(¯xn−2(tn−2)cn−1+ ¯xn−2(tn−2) (2.16) Perturbing ¯un−1 as ¯un−1+θvn−1 in the dynamics gives us

d

dtδxn−1(t) =fn−1,xn−1δxn−1(t) +fn−1,un−1vn−1. (2.17)

Perturbing ¯un−1 as ¯un−1+θvn−1 in the cost results in

δJn−1(¯un−1) =

Z tn−1

tn−2

(Φn−1,xn−1δxn−1(t) + Φn−1,un−1vn−1)dt

+

Z tn

tn−1

Φn,xnδxn(t)dt+ Θn,xn(xn(tn))δxn(tn)

Since the initial condition of the state in the interval [tn−1, tn] is related to the state in the interval [tn−2, tn−1] we note that perturbation of ¯un−1 affects the dynamics in the last interval. We do not perturb ¯un when we perturb ¯un−1. Then, the resulting perturbation of the state in the last interval is given by

d

dtδxn(t) =fn,xn(xn(t),u¯n(t))δxn(t), (2.18)

where,

δxn(tn−1) = [hn(xn−1(tn−1))cn]xn−1δxn−1(tn−1) +δxn−1(tn−1). (2.19)

Equation (2.19) can be rewritten as

δxn(tn−1) = [(hn(xn−1(tn−1))cn]xn−1 +I]δxn−1(tn−1).

We can also write (2.19) as

δxn(tn−1) = Qn(tn−1)δxn−1(tn−1) (2.20)

where,

(39)

Consider the following linear ODE with given initial condition

d

dtLn(t, tn−1) =fn,xn(xn(t),u¯n(t))Ln(t, tn−1), (2.21)

Ln(tn−1, tn−1) =I.

LetLn(t, tn−1) be fundamental matrix solution of (2.21). Then

δxn(t) = Ln(t, tn−1)Qn(tn−1)δxn−1(tn−1) (2.22) The variation of the costJn−1(¯x1,u¯1, ..., xn−1, un−1, xn,u¯n) with respect to the perturba-tion of the optimal control ¯un−1 in the current interval [tn−2, tn−1] is given by

δJn−1(¯x1,u¯1, ..., xn−1, un−1, xn,u¯n) =

Z tn−1

tn−2

(Φn−1,xn−1δxn−1(t) + Φn−1,¯un−1vn−1)dt

+

Z tn

tn−1

Φn,xnLn(t, tn−1)Qn(tn−1)δxn−1(tn−1)dt

+ Θxn(xn(tn))Ln(tn, tn−1)Qn(tn−1)δxn−1(tn−1)

(2.23) Let,

γn−1 =

Z tn

tn−1

Φn,xnLn(t, tn−1)Qn(tn−1)dt+ Θxn(xn(tn))Ln(tn, tn−1)Qn(tn−1) (2.24)

We rewrite (2.26) as

γn−1 =

Z tn

tn−1

Φn,xnLn(t, tn−1)Qn(tn−1)dt+γnLn(tn, tn−1)Qn(tn−1) (2.25)

where,

γn = Θxn(xn(tn))

Thus, the Gˆateaux derivative of the cost Jn−1(¯x1,u¯1, ..., xn−1, un−1, xn,u¯n) is given by

δJn−1(¯x1,u¯1, ..., xn−1, un−1,x¯n,u¯n) =

Z tn−1

tn−2

(Φn−1,xn−1(t)δxn−1(t) + Φn−1,¯un−1vn−1)dt

(40)

The proof of the following proposition which characterizes the costate variablePn−1(t) is similar to the proof of Proposition 2.2.4.

Proposition 2.2.5 Pn−1(t) is adjoint variable if and only if it satisfies the costate equa-tion

dPn−1 =−Hn−1,xn−1(¯xn−1,u¯n−1, Pn−1)dt=−{Φn−1,xn−1(¯xn−1,u¯n−1) +fn−1,xn−1(¯xn,u¯n−1).Pn−1}dt

Pn−1(tn−1) =γn−1

Let us move one step backwards to the next interval [tn−3, tn−2]. The control problem in this interval is

min Jn−2({xn−2(.), un−2(.)}) =

Z tn−2

tn−3

Φn−2(xn−2(t), un−2(t))dt+

Z tn−1

tn−2

+

Z tn

tn−1

subject to ˙

xn−2(tn−3) = hn−2(¯xn−3(tn−3))cn−1+ ¯xn−3(tn−3) (2.27) Perturbing ¯un−2 as ¯un−2+θvn−2 in the dynamics gives us

d

dtδxn−2(t) =fn−2,xn−2δxn−2(t) +fn−2,un−2vn−2. (2.28)

Perturbing ¯un−2 as ¯un−2+θvn−2 in the cost results in

δJn−2 =

Z tn−2

tn−3

(Φn−2,xn−2δxn−2(t) + Φn−2,un−2vn−2(t))dt+

Z tn−1

tn−2

Φn−1,xn−1δxn−1(t)dt

+

Z tn

tn−1

The dynamics in the interval [tn−2, tn−1] and [tn−1, tn] are affected by perturbation of ¯

(41)

in the intervals, [tn−2, tn−1] and [tn−1, tn]:

d

dtδxn−1(t) = fn−1,xn−1(xn−1(t),u¯n−1(t))δxn−1(t) (2.29)

and

d

with

δxn−1(tn−2) =Qn−1(tn−2)δxn−2(tn−2), (2.31) where,

Qn−1(tn−2) = [hn−1(¯xn−2(tn−2))cn−1]xn−2 +I,

We also have

δxn(tn−1) =Qn(tn−1)δxn−1(tn−1), (2.32)

where,

Qn(tn−1) = [h(¯xn−1(tn−1))cn]xn−1 +I,

Consider the following linear ODEs with respective initial conditions

d

dtLn−1(t, tn−2) = fn−1,xn−1(xn−1(t),u¯n−1(t))Ln−1(t, tn−2), (2.33)

Ln−1(tn−2, tn−2) = I, and

d

dtLn(t, tn−1) = fn,xn(xn(t),u¯n(t))Ln(t, tn−1) (2.34)

Ln(tn−1, tn−1) = I. Then, we have

δxn(t) = Ln(t, tn−1)Qn(tn−1)δxn−1(tn−1) (2.35) and

(42)

Then, the Gˆateaux derivative of the cost is

δJn−2 =

Z tn−2

tn−3

(Φn−2,xn−2δxn−2(t) + Φn−2,un−2vn−2(t))dt+γn−2δxn−2(tn−2) (2.37)

where,

γn−2 =

Z tn−1

tn−2

Φn−1,xn−1Ln−1(t, tn−2)Qn−1(tn−2)dt

+γn−1Ln−1(tn−1, tn−2)Qn−1(tn−2) (2.38) The following proposition characterizes the adjoint variable. The proof is identical to proof of Proposition 2.2.4.

Proposition 2.2.6 Pn−2(t) is adjoint variable if and only if it satisfies the costate equa-tion

dPn−2 =−Hn−2,xn−2(¯xn−2,u¯n−2, Pn−2)dt =−{Φn−2,xn−2(¯xn−2,u¯n−2)

+fn−2,xn−2(¯xn,u¯n−2)Pn−2}dt

Pn−2(tn−2) =γn−2.

At theithstage, we are in interval [tn−(i+1), tn−i]. Thus, we have the control problem

min Jn−i(¯x1,u¯1, ...,x¯i−1,u¯i−1, xi, ui, xi+1,u¯i+1, ..., xn,u¯n) =

i−1

X

i=0

Z tn−i

tn−(i+1)

Φn−i(xn−i(tn−i), un−i(tn−i)))dt+ Θ(xf(tf))

subject to ˙

xn−i(t) =f(xn−i(t), un−i(t)), tn−(i+1) < t < tn−i

xn−i(tn−(i+1)) =hn−i(¯xn−(i+1)(tn−(i+1)))cn−i + ¯xn−(i+1)(tn−(i+1))

(43)

that is un−i,θ = ¯un−i+θvn−i, is given by

dδxn−i = (fn−i,xn−i(xn−i(t),u¯n−i(t))δxn−i(t) +fn−i,un−i(xn−i(t),u¯n−i(t))vn−i(t))dt (2.40)

We know that the perturbation of ¯un−i affects the dynamics in the intervals [tn−i, tn−(i−1)], [tn−(i−1), tn−(i−2)], ..., [tn−1, tn]. The variation of the dynamics in the respective intervals is

dδxn−i =fn−i,xn−i(xn−i(t),u¯n−i(t))δxn−i(t)dt (2.41)

i= 0,1,2, ...i−2 Consider the following linear ODEs

dLn−i(t, tn−i) = fn−i,xn−i(xn−i(t),u¯n−i(t))Ln−i(t, tn−i)dt (2.42)

i= 0,1,2, ...i−2

Let the fundamental matrix solution of the system (2.42) be Ln−i(t, tn−i), where,

i= 0,1,2, ...i−2.

The Gˆateaux derivative of the cost is

δJn−i =

Z tn−i

tn−(i+1)

(Φn−i,_xn₋_iδxn−i(t) + Φn−i,_un₋_ivn−i)dt+γn−iδxn−i(tn−i) (2.43)

where,

γn−i =

Z tn−(i−1)

tn−i

Φn−(i−1),xn−(i−1)Ln−(i−1)(t, tn−i)Qn−(i−1)(tn−i)dt

+γn−(i−1)Ln−(i−1)(tn−(i−1), tn−i)Qn−(i−1)(tn−i) (2.44)

The proof of the following proposition is the same as Proposition 2.2.4.

(44)

equa-tion

dPn−i =−H(n−i),xn−i(¯xn−i,u¯n−i, Pn−i)dt

=−{Φn−i,xn−i(¯xn−i,u¯n−i) +fn−i,xn−i(¯xn,u¯n−i).Pn−i}dt

Pn−i(tn−(i−1)) = γn−i

After having determined the adjoint variables then we proceed to formulate the Hamil-tonian in each of the intervals, and use steepest decent method to numerically optimize the controls in each interval.

2.2.2 The Decision Variables

(

c

0_i

s

)

.

In this section we consider only the decision variables c0_is. We keep the control’s be-tween impulse instances to remain unchanged. That is, we keep ¯ui, i= 1, ..., nunchanged and perturb the c0_is, i= 1, ..., n.

We begin our study in the last interval [tn−1, tn]. The control problem is

min cn

Jn(¯x1,u¯1, ...,x¯n,u¯n) =

Z tn

tn−1

Φn(¯xn(tn),u¯n(tn)))dt+ Θ(¯xf(tf))

subject to ˙

xn(t) =fn(xn(t),u¯n(t)), tn−1 < t < tn

xn(tn−1) =hn(¯xn−1(tn−1))cn+ ¯xn−1(tn−1) (2.45) Consider the last decision variable ¯cn applied at time tn−1 and we perturb it as cn,θ = ¯

cn+θδcn. Then, the variation of the dynamics with respect to optimal decision variable ¯

cn is

dδxn

dt =fn,xn(xn,u¯n)δxn(t) (2.46)

(45)

Consider the following linear ODE with the given initial condition

d

dtLn(t, tn−1) = fn,xn(xn,u¯n)Ln(t, tn−1) (2.47)

Ln(tn−1, tn−1) = I

LetLn(t, tn−1) be fundamental matrix solution of the linear ODE. Then,

δxn(t) =Ln(t, tn−1)hn(¯xn−1(tn−1))δcn (2.48) Let,

˜

Qn(tn−1) =hn(¯xn−1(tn−1)) Then, we have

δxn(t) =Ln(t, tn−1) ˜Qn(tn−1)δcn (2.49) Thus, the variation of the cost becomes

δJn(¯x1,u¯1, ...x¯n,u¯n) =

Z tn

tn−1

Φn,xnLn(t, tn−1) ˜Qn(tn−1)δcndt

+ Θ(¯xf(tf))Ln(tn, tn−1) ˜Qn(tn−1)δcn

(2.50) Let,

˜

γn= Θx¯n(¯xn(tn))Ln(tn, tn−1) ˜Qn(tn−1)

Thus, the Gˆateaux derivative of the cost is

δJn(¯x1,u¯1, ...x¯n,u¯n) =

Z tn

tn−1

Φn,xnLn(t, tn−1) ˜Qn(tn−1)δcn+ ˜γnδcn (2.51)

Moving one step backward the next optimal control problem to discuss is in interval [tn−2, tn−1]. The control problem in this interval is

min cn−1

Jn−1(¯x1,u¯1, ...,x¯n−1,u¯n−1,x¯n,u¯n) =

Z tn−1

tn−2

Φn−1(¯xn−1(t),u¯n−1(t))dt

+

Z tn

(46)

subject to ˙

xn−1(t) = fn−1(xn−1(t),u¯n−1(t)), tn−2 < t < tn−1

xn−1(tn−2) = hn−1(¯xn−2(tn−2)cn−1+ ¯xn−2(tn−2) (2.52) Perturbing the decision variable ¯cn−1, the variation of the dynamics is

d

dtδxn−1 =fn−1,xn−1(xn−1(t),u¯n−1(t))δxn−1(t), tn−2 < t < tn−1

δxn−1(tn−2) =hn−1(¯xn−2(tn−2))δcn−1

Consider the following ODE

d

dtLn−1(t, tn−1) =fn−1,xn−1(xn−1(t),u¯n−1(t))Ln−1(t, tn−1)

Ln−1(tn−1, tn−1) =I

LetLn−1(t, tn−1) be fundamental matrix solution. Then, we have

δxn−1(t) =Ln−1(t, tn−1)hn−1(¯xn−2(tn−2))δcn−1. Thus,

δxn−1(t) =Ln−1(t, tn−1) ˜Qn−1(tn−2)δcn−1 (2.53) where,

˜

Qn−1(tn−2) =hn−1(¯xn−2(tn−2))

The variation of the cost with respect the perturbation of the decision variable ¯cn−1 is

δJn−1(¯x1,u¯1, ...,x¯n−1,u¯n−1,x¯n,u¯n) =

Z tn−1

tn−2

+

Z tn

tn−1

(47)

We can write (2.54) as

δJn−1 =

Z tn−1

tn−2

Φn−1,xn−1Ln−1(t, tn−2) ˜Qn−1(tn−2)δcn−1dt

+

Z tn

tn−1

Φn,xnLn(t, tn−1)Qn(tn−1)Ln−1(tn−1, tn−2) ˜Qn−1(tn−2)δcn−1dt

+ Θxn(xn(tn))Ln(tn, tn−1)Qn(tn−1)Ln−1(tn−1, tn−2)Qn−1(tn−2)δcn−1 (2.55)

where, Jn−1 =Jn−1(¯x1,u¯1, ...,x¯n−1,u¯n−1,x¯n,u¯n) Thus Gˆateaux derivative of the cost is

δJn−1 = [

Z tn

tn−2

Φn−1,xn−1Ln−1(t, tn−2)Qn−1(tn−2)dt+ ¯γn−1]δcn−1 (2.56)

where,

˜

γn−1 =

Z tn

tn−1

Φn,xnLn(t, tn−1) ˜Qn(tn−2)Ln−1(tn−1, tn−2) ˜Qn−1(tn−2)dt

+ ˜γnLn−1(tn−1, tn−2) ˜Qn−1(tn−2) (2.57) If ¯cn−1 is an interior point we have

Z tn

tn−1

Φn,xnLn(t, tn−1) ˜Qn(tn−2)Ln−1(tn−1, tn−2) ˜Qn−1(tn−2)dt

+ ˜γnLn−1(tn−1, tn−2) ˜Qn−1(tn−2) = 0

Next we move to the interval [tn−3, tn−2]. The control problem is

min cn−2

Jn−2(¯x1,u¯1, ...,x¯n−1,u¯n−1,x¯n,u¯n)}=

Z tn−2

tn−3

+

Z tn−1

tn−2

+

Z tn

tn−1

(48)

subject to ˙

xn−2(tn−3) = hn−2(¯xn−3(tn−3)cn−2+ ¯xn−3(tn−3) (2.58) Let,

Jn−2 =Jn−2(¯x1,u¯1, ...,x¯n−1,u¯n−1,x¯n,u¯n). Perturbing ¯cn−2 as ¯cn−2+θδcn−2 in the dynamics gives us

d

dtδxn−2(t) = fn−2,xn−2δxn−2(t). (2.59)

Perturbing ¯cn−2 as ¯cn−2+θδcn−2 in the cost gives us

δJn−2 =

Z tn−2

tn−3

Φn−2,xn−2δxn−2dt+

Z tn−1

tn−2

+

Z tn

tn−1

The dynamics in the interval [tn−2, tn−1] and [tn−1, tn] are affected by perturbation of ¯

cn−2. The decision variables ¯un−1,c¯n−1, ¯un,¯cn remain the same. This observation leads us to the following perturbation of the dynamics in the interval [tn−2, tn−1] and [tn−1, tn].

d

dtδxn−1(t) =fn−1,xn−1(xn−1(t),u¯n−1(t))δxn−1(t), (2.60)

and

d

with,

δxn−1(tn−2) = ˜Qn−1(tn−2)δxn−2(tn−2), (2.62) ˜

(49)

We also have,

δxn(tn−1) = ˜Qn(tn−1)δxn−1(tn−1) (2.63)

˜

Qn(tn−1) = h(xn−1(tn−1)).

Consider the following linear ODE’s with their respective initial conditions

d

dtLn−1(t, tn−2) =fn−1,xn−1(xn−1(t),u¯n−1(t))Ln−1(t, tn−2) (2.64)

Ln−1(tn−2, tn−2) =I and

d

dtLn(t, tn−1) = fn,xn(xn(t),u¯n(t))Ln(t, tn−1) (2.65)

Ln(tn−1, tn−1) = I Then, we have

δxn(t) =Ln(t, tn−1) ˜Qn(tn−1)δxn−1(t) (2.66)

and

δxn−1(t) = Ln−1(t, tn−2) ˜Qn−1(tn−2)δxn−2(t) (2.67) Then the Gˆateaux derivative of the cost Jn−2 is

δJn−2 = [

Z tn−2

tn−3

Φn−2,xn−2Ln−2(t, tn−3) ˜Qn−2(tn−3)dt+ ˜γn−2]δcn−1 (2.68)

where,

˜

γn−2 =

Z tn−1

tn−2

Φn−1,xn−1Ln−1(t, tn−2) ˜Qn−1(tn−2)Ln−2(tn−2, tn−3) ˜Qn−2(tn−3)dt

+ ˜γn−1Ln−2(tn−2, tn−3) ˜Qn−2(tn−3) If ¯cn−2 is an interior point we have

Z tn−2

tn−3

(50)

Continuing the same way at theithstage, that is, in the interval [tn−(i+1), tn−i] the control problem is

min cn−i

Jn−i(¯x1,u¯1, ...,x¯n,u¯n) = i−1

X

i=0

Z tn−i

tn−(i+1)

Φn−i(xn−i(tn−i),u¯n−i(tn−i)))dt+ Θ(¯xf(tf))

subject to ˙

xn−i(t) =f(xn−i(t),u¯n−i(t)), tn−(i+1) < t < tn−i

xn−i(tn−(i+1)) =hn−i(¯xn−(i+1)(tn−(i+1)))cn−i + ¯xn−(i+1)(tn−(i+1))

(2.69) The variation of the dynamics with respect to variation of ¯cn−i is

d

dtδxn−i(t) =fn−i,xn−i(xn−i(t),u¯n−i(t))δxn−i(t)

δxn−i(tn−(i−1)) =hn−i(¯xn−(i−1)(tn−(i−1)))δcn−i (2.70) The perturbation of ¯cn−i affects the dynamics in the succeeding intervals. The variation of the dynamics in those affected intervals is

dδxn−i =fn−i,xn−i(xn−i(t),u¯n−i(t))δxn−i(t)dt (2.71)

i= 0,1,2, ...i−1 Consider the following linear ODE’s

dLn−i(t, tn−i) = fn−i,xn−i(xn−i(t),u¯n−i(t))Ln−i(t, tn−i)dt (2.72)

i= 0,1,2, ...i−1 The Gˆateaux derivative of the cost is

δJn−i = [

Z tn−i

tn−(i+1)

(51)

where,

˜

γn−i =

Z tn−(i−1)

tn−i

Φn−(i−1),xn−(i−1)Ln−i(tn−i) ˜Qn−i(tn−i)Ln−(i−1)(t) ˜Qn−(i−1)(tn−(i−1))dt

+ ˜γn−(i−1)Ln−(i−1)(tn−(i−1), tn−(i−2)) ˜Qn−(i−1)(tn−(i−2)) (2.74) If ¯cn−i is an interior point we have

Z tn−i

tn−(i+1)

Φn−i,_xn−iLn−i(t, tn−i) ˜Qn−i(tn−i)dt+ ˜γn−i = 0.

(52)

Chapter 3 Applications.

In this chapter we give two applications of the method development in chapter two. We consider two HIV-treatment models. The first model is one which has only impulsive controls. The model is the one we studied in chapter one with cost added. The second application is one which has both impulsive controls and controls in between impulsive times.

3.0.3 Application 1

Consider the HIV treatment model we considered in Chapter 1. ˙

x=λ−dx−(1−ηu)βxy

˙

y= (1−ηu)βxy−ay−pyz

˙

w=cxyw−cqyw−bw

˙

z =cqyw−hz

˙

u=−u

Xi+1(t+i ) =Xi(ti−) +h(Xi(t−i ))ci where,

(53)

z number of CTLe cells,

u amount of medication delivered to the patient.

The term 1 −ηu, 0 ≤ 1−ηu ≤ 1 represents the efficacy of the drug. The drug is 100% effective if 1−ηu = 0 and ineffective if 1−ηu = 1. The cost is

J(¯c1,¯c2, ...,¯cn) = n

X

i=1

R¯c

2 i 2 +Sx

(xf+1(tf+1)−xf)2

2 +Sy

(yf+1(tf+1)−yf)2 2

+Sw

(wf+1(tf+1)−wf)2

2 +Sz

(zf+1(tf+1)−zf)2

2 (3.1)

Here R is the cost of associated with the intake of the drug, which includes the cost of the drug as well as the amount of damage to the health due to the drug taken. Let

J =J(¯c1,¯c2, ..., cn). Our control problem is

min J

subject to

˙

X(t) =f(X(t))

Xi+1(t+i ) =Xi(ti−) +h(Xi(t−i ))ci

h(Xi) =diag(0 0 0 0 1) (3.2) where,

xf, yf, wf, zf are chosen so that if the system is left without further medication after this point, it converges to the desired equilibrium point which is high healthy cell count and low unhealthy cell count.

1. Consider the optimal control problem in the last interval [tn−1, tn]. Let the objective function in this interval be

Jn(¯c1,c¯2, ..., cn) = n X i=1 Rc 2 i 2 +Sx

(xf+1(tf+1)−xf)2

2 +Sy

(yf+1(tf+1)−yf)2 2

+Sw

(wf+1(tf+1)−wf)2

2 +Sz

(zf+1(tf+1)−zf)2

(54)

LetJn=Jn(¯c1,¯c2, ..., cn). The control problem is min

cn

Jn

Subject to

˙

Xn(t) = fn(Xn(t)), tn−1 < t < tn

Xn(tn−1) = ¯Xn−1(tn−1) +hn( ¯Xn−1(tn−1))cn

hn(Xn(t)) =diag(0 0 0 0 1) (3.4) The variation of the dynamics with respect to variation of the optimal impulsive control ¯cn is

d

dtδXn=fn,Xn(Xn(t))δXn

δXn(t+n) = hn( ¯Xn−1(t−n))δcn

LetLn(t, tn−1) be fundamental matrix solution of the following linear ODE

d

dtLn(t, tn−1) = fn,Xn(Xn)Ln(t, tn−1)

Ln(tn−1, tn−1) = I Then,

δXn(t) = ˜Qn(tn−1)Ln(t, tn−1)δcn (3.5) where,

˜

Qn(tn−1) =hn(tn−1)

Variation of the cost Jn with respect to variation of the optimal impulsive control ¯

cn is

δJn= ΘXn(Xn(t))δXn(t) (3.6)

Using (3.5) the variation of the cost is

(55)

Then the derivative of the cost is

δJn= Θxn(tn)Ln(tn, tn−1)Qn(tn−1)δcn (3.8)

2. Moving one step backward to interval [tn−2, tn−1]. The cost is

Jn−1(¯c1,¯c2, ...,c¯n−1, cn−1,¯cn) = n

X

i=1

Rc

2 i 2 +Sx

(xf+1(tf+1)−xf)2

2 +Sy

(yf+1(tf+1)−yf)2 2

+Sw

(wf+1(tf+1)−wf)2

2 +Sz

(zf+1(tf+1)−zf)2 2

(3.9) We letJn−1 =Jn−1(¯c1,c¯2, ...,c¯n−1, cn−1,c¯n). In this interval the control problem is

min cn−1

Jn−1

Subject to

˙

Xn−1(t) =fn−1(Xn−1(t)), tn−2 < t < tn−1

Xn−1(tn−1) = ¯Xn−2(tn−2) +hn−1( ¯Xn−2(tn−2))cn−1

hn−1( ¯Xn−1(t)) =diag(0 0 0 0 1) The variation of the dynamics is

d

dtδXn−1 =fn−1,Xn−1( ¯Xn−1(t))δXn−1(t)

δXn−1(t+n−2) = hn−1( ¯Xn−2(t−n−1))δcn−1

LetLn−1(t, tn−2) be fundamental matrix solution of the following linear ODE

d

dtLn−1(t, tn−2) =fn,XnLn−1(t, tn−2)

Ln−1(tn−1, tn−2) =I

(56)

Consider the following linear ODE in interval [tn−1, tn]

d

dtLn(t, tn−1) = fn,Xn(Xn)Ln(t, tn−1) (3.10)

Ln(tn−1, tn−1) = I

LetLn(t, tn−1) be fundamental matrix solution (3.10) and

δXn(t) =IδXn−1(tn−1) (3.11) Also,

δXn−1(tn−2) = ˜Qn−1(tn−2)δcn−1 (3.12) where ˜Qn−1(tn−2) = hn−1 =h. Then the Gˆateaux derivative of the cost is

δJn−1 = ΘXn(tn)Ln(tn, tn−1)ILn−1(tn−1, tn−2)Qn−1(tn−2)δcn−1 (3.13)

3. Continuing the same way, at the ith interval [tn−(i+1), tn−i]. The control problem is min

cn−i

Jn−i

Subject to

˙

Xn−i(t) = fn−i(Xn−i(t)), tn−(i+1) < t < tn−i

Xn−i(tn−(i−1)) = hn−i( ¯Xn−(i−1)(tn−(i−1)))cn−i+ ¯Xn−(i+1)(tn−(i+1)) (3.14)

hn−i(Xn−i(t)) = diag(0 0 0 0 1) where,

Jn−i =Jn−i(¯c1,¯c2, ...,¯cn−(i+1)), cn−i,¯cn−(i−1, ...,c¯n). The variation of the dynamics is

d

dtδXn−i(t) = fn−i,Xn−i(Xn−i(t))δXn−i(t)

(57)

variation of the dynamics in those affected intervals is

dLn−i(t, tn−(i+1)) = fn−i,Xn−i(Xn−i(t))Ln−i(t, tn−(i+1))dt (3.16)

i= 0,1,2, ...i−1

Let Fundamental matrix solution of the systems in (3.16) beLn−i(t, tn−(i+1)), where,

i= 0,1,2, ...i−1. Then the variation of the cost is

δJn−i = ΘXn(tn)Ln−i(t, tn−(i+1)) ˜Qn−i(tn−(i+1))...Ln(tn, tn−1)δcn−i. (3.17)

The Gˆateaux derivative of the cost is

δJn−i(¯c1,¯c2, ...,¯cn) = ΘXn(tn) ˜Qn−i(tn−(i+1))Ln−i(t, tn−(i+1))...Ln(tn, tn−1)δcn−i.

(3.18)

3.0.4 Numerical Computation And Simulation.

We use four intervals [t0, t1],[t1, t2],[t2, t3],[t3, t4] where t3 = tf for the simulation. With impulsive controls applied at t1 and t2. That means we do have four impulsive control problems one in each interval. The fourth interval is included to give the system enough time to come closer to the intended cell count. The dynamics in each interval is given by

1. In interval [t0, t1] we consider ˙

x1 =λ−dx1 −(1−ηu1)βx1y1 ˙

y1 = (1−ηu1)βx1y1−ay1−py1z1 ˙

w1 =cx1y1w1−cqy1w1−bw1 ˙

z1 =cqy1w1 −hz1 ˙

u1 =−u1

(58)

x2 =λ−dx2 −(1−ηu2)βx2y2 ˙

y2 = (1−ηu2)βx2y2−ay2−py2z2 ˙

u2 =−u2

X2(t1) = ¯X1(t1) +h( ¯X1(t1))c1 (3.20)

x3 =λ−dx3 −(1−ηu3)βx3y3 ˙

y3 = (1−ηu3)βx3y3−ay3−py3z3 ˙

u3 =−u3

X3(t2) = ¯X2(t2) +h( ¯X2(t2))c2 (3.21)

x4 =λ−dx4 −(1−ηu4)βx4y4 ˙

y4 = (1−ηu4)βx4y4−ay4−py4z4 ˙

u4 =−u4

X4(t3) = (2,2,0.4,0.25)t

The cost is

J(¯c1,¯c2) =

R

2((¯c1) 2

+(¯c2)2)+Sx

(x4(4)−2)2 2 +Sy

(y4(4)−2)2 2 +Sw

(w4(4)−0.4)2

2 +Sz

(59)

The next step is to determine the fundamental matrix solutions, L1, L2,L3, and L4. 1. L4 is determined from the following equation

dL4

dt =f4,X4( ¯X4(t))L4(t)

L4(t3) =I

t3 < t < t4

2. L3 is determined from the following equation

dL3

dt =f3,X3( ¯X3(t))L3(t)

L3(t2) =I

t2 < t < t3

dL2

dt =f2,X2( ¯X2(t))L2(t)

L2(t1) =I

t1 < t < t2

dL1

dt =f1,X1( ¯X1(t))L1(t)

L1(t0) =I

t0 < t < t1

Also,

1. in the interval [t3, t4] ˜Q4 be defined by ˜Q4 = (diag(0,0,0,1))t

(60)

The numerical simulation we are going to carry out is based on the state equations and the impulsive controls. We use steepest descent method for optimization purpose. We follow the following procedure for numerical simulation

1. We solve the ODEs in each interval starting from the one in the first interval forward.

2. Determine ˜Q4 and L4 in interval 4.

3. In the third interval calculate ˜Q3, L3 and gradient of the objective function. Use the information to update c2. After updating c2 use the improved information to update the state variables both in the third and forth interval.

4. Calculate ˜Q2 and L2. Using the information improvec1. After improvingc1 use the improved information to update the state variables both in the second, third and fourth interval.

5. Finally go to first interval. Solve the ODE in the interval. Changes in the state variable in this interval affects the value of the state variables both in second third and fourth interval.

The following is a pseudo code for our numerical simulation

BEGIN psuedocode

Solve ODE’s in all intervals

Fourth interval:

Calculate \tilde{Q}4, L4 and gradJ4

Third interval:

Calculate \tilde{Q}3 L3 and gradJ3

use steepest decent method to improve c2

solve ODE’s in third and fourth interval using the improved c2

Second interval:

Calculate \tilde{Q}2 L2 and gradJ2

(61)

Solve ODE’s in second, third and fourth interval using the improved c1

First interval:

Calculate \tilde{Q}1 L1 gradJ1

Update states variables in all intervals

END psuedocode

The parameters value we use are [21]

λ= 1, e= 0.1, d= 0.1, a= 0.2, η= 0.5, β = 0.42, p= 1, b= 0.1, h= 0.1, q= 0.5, and the initial amount of medicine in the body chosen to be u0(0) = 0.

The numbersSx, Sy, Sw, Szare measures of importance attached to the difference between the final cell numbers and desired cell counts. These values are all chosen to be 2.

Conclusion.

Taking R= 0.1, we have the following cost and optimal impulsive values:

Cost= 0.042833315801935, c1 = 0.523975314189902, c2 = 0.349153540962085.

(62)

Table 3.1: Cell counts.

Healthy cells Unhealthy cells Memory cells Effectors Control 3.411988 0.9208089 0.3096524 0.1031824 0.1120595 3.063783 1.203528 0.3222986 0.1078693 0.9651733 2.799905 1.417936 0.3290286 0.1113703 0.8556089 2.024841 2.038927 0.3335831 0.1216719 0.5363530 2.024838 2.038922 0.3335830 0.1216724 0.5363640 2.024835 2.038917 0.3335829 0.1216729 0.5363750

(63)

Figure 3.1: Cell count of the simulation.

3.0.5 Application 2.

The HIV-immune system model we study here is considered in [41]. Here we study it as an impulsive control problem. The rational for this model is given in [34]. The model is

˙

x1(t) = −a1x1−a2x1x2(1−u2) +a3a4x4(1−u1) ˙

x2(t) =

a5 1 +x1

−a2x2(1−u2)(1−u4)−a6x2 +a7(1− 1

a8

(x2+x3+x4))x2(1 +u3) ˙

x3(t) = a2x1x2(1−u2)(1−u4)−a9x3−a6x3 ˙

x4(t) = a9x3−a4x4

X(0) = (x1(0), x2(0), x3(0), x4(0))T

Dynamic Programming Method For Impulsive Control Problems

ABSTRACT

DEDICATION

BIOGRAPHY

ACKNOWLEDGEMENTS

TABLE OF CONTENTS

LIST OF TABLES

LIST OF FIGURES

Chapter 1

Introduction

1.1

Stability Analysis Of HIV Model.

1.2

HIV Infection Mathematical Model.

1.3

Stability Of Equilibrium Points Of The Model.

Chapter 2

Impulsive Control Problems.

2.1

Impulsive Control Problems.

2.2

Solution Method.

2.2.1

Controls Between Impulses (u

’s).

2.2.2

The Decision Variables

(

c

s

)

.

Chapter 3

Applications.

3.0.3

Application 1

3.0.4

Numerical Computation And Simulation.

3.0.5

Application 2.