A New Lagrangian Multiplier Method on Constrained Optimization

(1)

A New Lagrangian Multiplier Method on Constrained

Optimization

*

Youlin Shang#, Shengli Guo, Xiangyi Jiang

Department of Mathematics, Henan University of Science and Technology, Luoyang, China

Email: #_{[email protected]}

Received July 8, 2012; revised August 8, 2012; accepted August 15, 2012

ABSTRACT

In this paper, a new augmented Lagrangian function with 4-piecewise linear NCP function is introduced for solving nonlinear programming problems with equality constrained and inequality constrained. It is proved that a solution of the original constrained problem and corresponding values of Lagrange multipliers can be found by solving an uncon-strained minimization of the augmented Lagrange function. Meanwhile, a new Lagrangian multiplier method corre-sponding with new augmented Lagrangian function is proposed. And this method is implementable and convergent.

Keywords: Nonlinear Programming; NCP Function; Lagrange Function; Multiplier; Convergence

1. Introduction

Considering the following nonlinear inequality constrained optimization Problem (NLP):

 

min

. 0

f x

s t H x  G x 0, (1)

where : n

f R R and

 



   

 



 



   

 



1 2

, , , :

T _n _m

m

T _n _p

p

G x g x g x g x R R

H x h x h x h x R R

 





 

are continuously differentiable functions. We denote by

 







 







0, 1, 2, , ,

0, 1, 2, ,

n i

j

D x R g x i m

h x j p

   

 



the feasible set of the problem (NLP).

The Lagrangian function associated with the problem (NLP) is the function



, ,



 

T

 

+ T

 

,

L x   f x  H x  G x

where



1, ,



,



1, ,



T

T _p _m

P R

       _m R



are the multiplier vectors, For simplicity, we use



x, ,  to denote the column vector



xT, T, T



T

Defintion 1.1. A point



x, ,  



RnRpRm is

called a Karush-Kuhn-Tucker (KKT) point or a KKT pair of Problem (NLP), if it satisfies the following condi-tions:





 

, , 0, 0 , 0,

0, 0, ,

x

i i

L x G x H x

g x i I

 

 

  

   



(2)

where I  



1 i m



, we also say x is a KKT point if

there exists a

 

 , such that



x, , 



satisfies (2). For the nonlinear inequality constrained optimization problem (NLP), there are many practical methods to solve it, such as augmented Lagrangian function method [1-6], Trust-region filter method [7,8], QP-free feasible method [9,10], Newton iterative method [11,12], etc. As we know, Lagrange multiplier method is one of the effi-cient methods to solve problem (NLP). Pillo and Grippo in [1-3] proposed a class of augmented Lagrange func-tion methods which have nice equivalence between the unconstrained optimization and the primal constrained problem and get good convergence properties of the re-lated algorithm. However, a max function is used for these methods which may be not differentiable at infinite numbers of points. To overcome this shortcoming, Pu in [4] proposed a augmented Lagrange function with Fischer- Burmeister nonlinear NCP function and Lagrange multi-plier methods. Pu and Ding in [6] proposed a Lagrange multiplier methods with 3-piecewise linear NCP function. In this paper, a new class augmented Lagrange function with 4-piecewise linear NCP function and some La-grange multiplier methods are proposed for the minimi-zation of a smooth function subject to smooth inequality

*_{This paper was partially supported by the National Natural Science}

Foundation of China under Grant No. 10971053, and NNSF of Henan Province under Grant No. 094300510050.

(2)

constraints and equality constrains.

The paper is organized as follows: In the next section we give some definitions and properties about NCP func-tion, and then define a new augmented Lagrange function with 4-piecewise NCP function. In Section three, we give the algorithm. In Section four, we prove convergence of the algorithm. Some conclusions are given in Section five.

2. Preliminaries

In this section, we recall some definitions and define a new Lagrange multiplier function with 4-piecewise NCP function.

Definition 2.1 (NCP pair and SNCP pair). We call a pair (a, b) to be an NCP pair if and ab = 0;

and call (a, b) to be an SNCP pair if (a, b) is a pair and

.

0, 0

a b

2 2 ₀

a b 

Definition 2.2 (NCP function). A function

is called an NCP function if if and only is an NCP pair.

2

:R R

 

0

 

a b,

 



a b,



In this paper, we propose a new 4-piecewise linear NCP function 



a b, is as follows:

 

2

2 2

2

, ,

2 ,

,

2 +2 ,

4 , 0

k a if b k a

kb b a if a b k a b

k a kb b a if a b k k a kb if b k a



 



 

  

  



 _ _{ }

 



0

(3)

If

  

a b,  0, , then

 

2

2 2

2 2 2

2 , 0

,

2 2

,

2

,

2 2

, 4

k

if b k a b a

if a b k k b a

a b

k b a

if a b k k b a

k

if b k a k



 

  

 



 

_ _ _

 _

 

 _{ }

  

 _{ }

 

 _

 

  

_ _  

  

(4)

and

 









2 2

2 2 0,0

2 1

1 ,

2 1 2 1

A

k t k t

t

k t _k _t

  

_ _ _ _ _ _

 _ _

_ _{ } _  

 _

 _{ } _ 

 



0

(5)

It is easy to check the following propositions: 1) 

 

a b,   0 a 0,b0,ab ;

2) The square of  is continuously differentiable; 3)  is twice continuously differentiable everywhere except at the origin but it is strongly semi-smooth at the origin.

Let



, ,





,

 



, 1 ,

i x C i Cg xi i

      m

where is a parameter. if and

only if

0

C _i



x, , C



0

 

0

i

g x  , _i0 and for any

.

 

0

ig xi

 

0

C

We construct function:







1



1







Ф x, , C   x, , C , ,m x,m,C

Clearly, the KKT point condition (2) is equivalently reformulate as the condition:





 





Ф x, , C 0,H x  0, L x, ,  0.

If



g xi

 

,i





 

0,0 , then i



x, , C



is

continu-ously differentiable at



x,



_Rn m _{. We have}

 





 





 





 

2

2 2 2

2

2 2 2 2

2 , 0

,

if

2 2

,

if

2 2

, 2

if

4

,

if 0

i i

i

i i

i i i

i i

i

i i

i i i

i i

i

i i

Cg x

k

Cg x k

C g x

kC g x

C g x e

C g x k

C g x

kC g x

k C g x e

C g x k

kC g x

k e

Cg x k

 



  



 

      

  



  

 _  _ 

_ _ 

 

 

 



 _ _

 

  

_ _ 

  

 

  

 

 

 

 

 

 

   

              

 (6)

where is the ith column of

the unit matrix, its jth element is 1, and other elements

are 0, in this paper take k = 1.



0, ,0,1, ,0



T m

i

e    R

If



gi

 

x , i



 

0,0 , and then i



x, , C



is

strongly semi-smooth and direction differentiable at



_x_,



__Rn m _{. We have}









 





 

2 2

2 2 ,

2 1

1 ,

2 1 2 1

i

i i

x

k t e

k t e

t

Ck t g x Ck t g x

 



_ _  _  

 _ _ 

_ _{ } _  

  _{ }

 _{ } _ 

 

 (7)

(3)





 



 













  

 





2 1 2 2 1 2 , , , , 2

, , 2 2 2

, , 2,

p T

j j m

i i i

i

T x

S x C D

f x H x D h x

x C C

G x L x

                 _   _     



(8)

where



1, ,





1, ,



T m T p

m R P R

     ,    are the Lagrange multiplier, C and D are positive parameters.

In this section, we gave some assumptions as follows: Assumpion 1 f, hj

 

x , j1, , p, g xi

 

1, ,

i  m are twice Lipschitz continuously

differen-tiable.

Define index set I₀ and I1 as follows:









 



 











 



 



0

1

, 0,0 , 1, 2, ,

i i

I x i g x i m

I x i g x i

            ，， m

for any iI₀, according to definition of _i, have









2





2

, , 2 2 0

xi xi C i i 

 _   _

 

for any iI1, we have

1) if  i C gi

 

x , we have























 



 









 

2 2

, , 2 2 2

2 2 , ,

2

x i i i i

i i i i i i

i i i i i

i i i i

x C C

x C Cg x g x

Cg x x C g x

g x Cg x g x

               __   __              (9)

The gradient of S x



, , , , C D



is





 



 





 











 





  









 





 



 









 





  



1 1 1 2 2 1 1 2 , , , ,

, , 2 2

, , , ,

, ,

2 2 , ,

2 , , , , x p T j j j

i i i

i I T p T j j j

i i i i i

i I m

i i

i

T

S x C D

f x H x D h x h x

x C C

G x L x G x L x

L x H x D h x h x

Cg x x C g x

Cg x g x

G x L x G x L x

                                                        



(10)

The Henssian matrix of S x



, , , ,C D



KKT point











   



   





  

  









2 2 1 1 2 2 , , , , , , 6 , , , , , p _T j j j m T i i i T

S x C D

L x D h x h x

C g x g x

G x L x G x L x

                      



 (11)

2) if _i

 

i

C g x

   , then























 



 









 

2 2

, , 2 2 2

, , 2 2 2 ,

2 2 , ,

2

x i i i i

i i i i i i

i i i i i

i i i i

x C C

x C Cg x

Cg x x C g x

g x Cg x g x

               __   __             

g x ₍₁₂₎

The gradient of S x



, , , ,  C D



is





 



 





 











 





  









 





 



 









 





  



1 1 1 2 2 1 2 , , , ,

, , 2 2

, , , ,

, ,

2 2 , ,

2 , , , , x p T j j j

i i i

i I T p T j j j

i i i i i

i I

m

i i

i

T

S x C D

f x H x C h x h x

x C C

G x L x G x L x

L x H x C h x h x

Cg x x C g x

Cg x g x

G x L x G x L x

                                                       



(13)

The Henssian matrix of S x



, , , ,  C D



at KKT point



x, , 



is











   



   





  

  









2 2 1 1 2 2 , , , , , , 4 , , , , , p _T j j j m T i i i T

S x C D

L x D h x h x

C g x g x

G x L x G x L x

                      



 (14)

Definition 2.3 A point



x, , 



is said to satisfy the

strong second-order sufficiency condition for problem (NLP) if it satisfies the first-order KKT condition and if





2 _{, ,} ₀

T xx

d  L x  d for all

  

 





0, 1, 2, , ;

0, 1, 2, , , 0 ,

T j

T

i i

d p x d d h x j p

d g x i i i m

    

    



at

(4)

and d0.

Assumption 2 At any KKT point



x, , 



satisfied

strong second-order sufficiency condition. Lemma If An n

Ad

is a positive semi-definite matrix, for

any , , matrix satisfied ,

then exist ₁, for any , is positive definite matrix (see [4]).

n

dR 0 Bn n

1

m

0 T

d Bd 

m m BmA

Theorem 2.1 If



x, , 



is KKT point of problem

(1), then for sufficiently large C and D, S x



, , , ,  C D



is strong convex function at point



x, , 



. Proof: Let B 2L x



, , 



   





 



 



1 1

p T m T

j j i i

j i

A h x h x g x g x

 





  



 

for dp x

 

, we have

   





 



 



 





 



 



1 1

0

p _T m

T

j j i i

j i

p m

T T

j j i i

j i

Ad h x h x d g x g x d

h x d h x g x d g x

 

     





from A2,wehave . Furthermore there is 1

if , for any ₂ ₁,

0 T

d Bd



m1

m A



min 6 , 4 ,C C D m m Bm₂ is

positive definite matrix. And then for any x0 and sufficiently large C and D have











 



 









2

2 2

1 2

2 1

1

, , , ,

, ,

6

0 T

p

T T

j j

m

T T

i i

T

x S x C D x

x L x x D x h x

C x g x x L x x

m x Ax

 

, , 





   

   

 



by its continuously, we may obtained that there is 0, for all



















, , , , , ,

, , , ,



,

x B x C D

x x



   

    



  

we have T



, , , ,



0 the theorem hold.

x S x  C D x 

3. Lagrange Multiplier Algorithm

Step 0 Choose parameters C00, D00, 01,

1

0  1 2, given point

0 n

x R ,and









0 0 0

1

0 0 0

1

, , ,

p p

m m

R

  

  

 



Let k0.

Step 1 Solve following, we will obtain xk1.





 

















 





2 1

2 2

1

2 min , , , ,

2

, , 2 2

2

, , 2

def

T

p

j j

m

i i i i

i

T x

S x C D

f x H x

D h x

x C

C

G x L x

  

   

 



 



 _ _ 

 

 



  



if

 

k 1

H x  

  and





1_, _,

k k k

x  C 



  then

stop.

Step 2 Forj1, 2, , p, h_j

 

xk1 ₁h_j

 

xk , then 1

k k

D  D or Dk1₂Dk, for i1, 2, , m, if



1







1

, , , ,

k k k k k k

i x C i x

   _  

C ,

then k 1 k, or

C  C 1 2

k k

C   C

1 k

Step 3 Compute   and k1

 

1 1 1

k k k k k k

j j Dh _j x i i Cg xi

  _ _    _ _ 1 Step 4 Let k = k + 1, go to Step 1.

4. Convergence of the Algorithm

In this section, we make a assumption follow as: Assumption 3 For any k, k, Ck, Dk,



k 1_, k_, k_, k_, k



S x    C D exists a minimizer point xk1.

Theorem 4.1 Assume feasible set of problem (NLP) is non-empty set and f x

 

is bounded, then algorithm is

bound to stop after finite steps iteration.

Proof: Assume that the algorithm can not stop after finite steps iteration, by the sack of convenience, we de-fine index set as following

 



 



 



 



lim 0 , lim

k k

e e

i

x x

k k

i

i i

x x

J i h x J i D

J i  x J i C

 

    

according to assumption A3, it is clearly that JeJi or

e i

J J are non-empty set. for any k, obtain



  

 





















 





1 1

2 2

1 1

2 2

1 1

2 1

S , , , , =

+ 2

+ , , 2 + 2

+ , , 2

k k k k k k

p

k k k k k k

i i i

i m

k k k k k k k k k

i i i

i

k k k

x C D f x

D h x D D

C x C C C C

G x L x

 

 

   

 

 

 



 _ _ 

 

 

2

 _ 

 

 

 



from above assumption, we obtain that for any a k

there is kk, for any iJe and 0,

and

1

k k

D  D

 

k 1 i

(5)

difficult to see that

 



₁



2





2 _{2 2}

2 2

k k k k k k

i i i

D _ h x   D   D _ k

 

Or for any iJ and  0, k 1 and

C  Ck

 

k 1

i

g x  , for sufficiently large k, have







₁



2





2

2 2

, , 2 2 2

2

k k k k k k k k k

i i i i

C x C C C C

k

   

 

 _ _ 

 

 



When k  we can hold



1



S k , k, k, k, k

x   C D  

Which contradicts A3, the theorem holds. Theorem 4.2 Let X P M Rn p m +



k , k, k



is a compact set, sequence 1

x  







are generated by the algo-rithm, and k 1_, k_, k int





x     X P M , in algorithm,

0 take the place of , either algorithm stops at its

and k 1 kth

x is solution of problem(NLP), or for any an

accumulation *

x of sequence

 

xk1 ,x* is solution of

problem (NLP).

Proof: Because the algorithm stops at its , then

we have kth







 

+1 +1

+1

, , 0, 0,

0, 0

k k k k

i i i

k k k

i i i

x C g x

g x

 

 



 

  (15)

for g xi

 

0,  i 0, it is easy to see that for any have

0



C



k+1_, k_,



₀ k 1 k

 

k 1

i x C i i Cg xi

k i

  _   _ _  __

It is from Step 2 of the algorithm that we have



x, , C



_ 0, H x

 

0

  _ 



(16)

putting (15) (16) into (10) or (13), we can obtain









  



 





  



 









1

1 2 1 1

1

1 2 1 1

1

, , , ,

, ,

+ , ,

, ,

= , ,

, , 0

k k k k k

k k k

k k k k T k

k k k

k k k k T k

k k k

S x C D

L x

G x L x G x

L x

E G x L x G x

L x

   



  



 





 

  



   

 

for gi

 

x 0, i0, according to definition of



, ,



Ф x C , we can obtain, that



k 1_, k_, k



₀

L x   

 

First part of the theorem holds，_xk1_{is solution of} problem (NLP).

On the other hand, if the algorithm is not stop at , for any accumulation point



_x*_,_{ }*_, *



_{of sequence}kth



k 1_, k_, k



x   , from theorem 4.1, we can obtain, for any

positive number C, that



*_, *_,



_0,

 

* ₀

x C H x

   

for any xX , have

 





















  



 

  



 

1

2 2

1 1

2 2

1 1

2

1 1

2

1 1 1

+ 2

, , 2 + 2

+ , , 2

, , 2 1

k

p

k k k k k k

i i i

i m

k k k k k k k k k

i i i

i

k k k k

k k k k k

f x f x

D h x D D

C x C C C C

G x L x

f x G x L x

 

   

 

  









 

  



 _ _ 

 

 

2

 

 _  _

 

 

  



= +o

Let k ,have

 

  



2

 

* * *_, *_, * ₂ *

f x  f x  G x L x    f x

Clearly, second part of the theorem holds. *

x is

solu-tion of problem (NLP).

5. Conclusion

A new Lagrange multiplier function with 4-piecewise linear NCP function is proposed in this paper which has a nice equivalence between its solution and solution of original problem. We can solve it to obtain solution of original constrained problem, the algorithm correspond-ing with it be endowed with convergence.

6. Acknowledgements

This paper was partially supported by the NNSF of China under Grant No. 10971053, and NNSF of Henan under Grant No. 094300510050.

REFERENCES

[1] G. Pillo and L. Grippo, “A New Class of Augmented Lag- rangian in Nonlinear Programming,” SIAM Journal on Control and Optimization, Vol. 17, No. 5, 1979, pp. 617- 628.

[2] G. Pillo and L. Grippo, “An Augmented Lageangian function for Inequlity Constrains in Nonlinear Program-ming Problems,” Optimization Theorem and Application, Vol. 36, No. 4, 1982, pp. 495-520.

doi:10.1007/BF00940544

[3] G. Pillo and S. Lucidi, “An augmented Lagrange Func-tion with Improved Exactness Properties,” SIAM Journal on Control and Optimization, Vol. 12, No. 2, 2001, pp. 376-406.

(6)

[5] K. D. Li and D. G. Pu, “A class of New Lagrange Multi-plier Methods,” OR Transaction, Vol. 10, No. 4, 2006, pp. 9-24.

[6] X.-W. Du and Y.-L. Shang, “Exact Augmented Lagrange Function for Nonlinear Programming Problems with Ine-quality and EIne-quality Constraints,” Applied Mathematics and Mechanics, Vol. 26, No. 12, 2005, pp. 1651-1656. [7] K. Su, “Trust-Region Filter Method with NCP Function,”

Journal of Systems Science and Mathematical Science, Vol. 28, No. 12, 2008, pp. 1525-1534

[8] D. G. Pu, “A New Filter Method,” OR Transaction, Vol. 1, No. 15, 2011, pp. 46-58.

[9] W. Hua and D. G. Pu, “A Modified QP-Free Feasible Method,” Applied Mathmatical Modelling, Vol. 35, No. 4, 2011, pp. 1696-1708. doi:10.1016/j.apm.2010.10.002 [10] K. Li and D. G. Pu, “Filter QP-Free Method with 3-Piece-

Wise Linear NCP Function,” OR Transaction, Vol. 12, No. 2, 2008, pp. 49-57.

[11] Y. X. Yuan, “Numerical Method for Nonlinear Program-ming,” Shanghai Science and Technology Press, Shang-hai, 1993.

doi:10.1007/BF00940544

1708. doi:10.1016/j.apm.2010.10.002