a ni < s1 < �) , a2) 1 aa

I n m any probl ems , thi s xi cannot be eas i l y eval uated

ar e therefore us uall y sl o..rer

( 5. 2. 7 )

and than

N RL -el imi nati on i ter at i ons

NRL-adj us tment i ter at i ons . When the s ystemati c part ni i s nonl i near ,

N RL-el imi nat i on and NRL-adj ustment are usuall y not equi val ent . I n

models wi th two or mor e s yst ematic parts , N RL- el imination can be def i ned i n a simil ar way .

Another commonl y us ed al ternati ve to NR is F isher ' s scori ng

techni que ( FS) i n whi ch the second deri vati ve matrix in

( 5. 2. 1 )

i s repl aced by i ts expected val ue . The FS-adj ustment i ter ations have t he

SECTION

5. 2

_{1 43}

same f orm as NR-el imi nat i on/ aclj ustm ent i ter at i ons , but wi th all second deri vat i v es replaced by their expe cted values . L i ke j oi nt FS i ter at i ons , they ar e al ways i n an as cent direct i on . FS-el imi nati on i s usuall y di stinct fr om FS-a clj ustment and i s rar ely pra cti c al since ther e ar e no gener al form ul ae corres ponding to thos e i n Theor em

5. 2. 1

f or

2

.. 2

eval uat i ng E[

a i( 81 < s2) , a2) 1 aa2J ,

and al so the expectat i on us ually

i nvol ves

a,

as well as

s2•

Si nce adj ustm ent i ncr eas es the l i kel i hood in an i ter at i on ,

NR- el imi nation!adj us tment i s l ess l i kely to di ver ge than j oi nt NR and , when bot h al gori thms conver ge , cons i derabl y f e..r er i ter at i ons ar e often required . N RL-adj us tment and FS-adj ustment have simil ar advanta ges over j oi nt NRL and j oi nt FS . However conver gence probl an s ar e often encount ere d wi th all al gori thns , es pe ci ally if poor starting val ues are used . F irs tl y , i n NR al gori thms the matri x of second der i vati ves may

not be negati ve def i ni t e an d the i t er ati ons may then approach a mi nim um

or s addl e negat i ve

probl em

poi nt of the l i kel i roo d .

def i nite whenever

a2 i( 8)

I

as2

i s more canmon wi th

2 ..

2

Since

a i( 81 ( 82) , �) / ola2

i s negat i ve j oi nt NR def i ni te , than

NR- el imi nati on/ adj ustment . A simi l ar result hol ds f or the

i s thi s with

NRL al gori thns. If the probl an ari s es , a def i ni t enes s mo difi cation must be

used . Thi s t ype of modifi cat i on i s easi er and f ast er for NR

el imi nation/ adj ustment than f or j oi nt NR since the second deri vati ve matri x i s sm al l er i n t he f ormer i ter at i ons . For exampl e , if

�

is a

si ngl e parameter , the s econd der i vati ve of the elimi nat ed

log-l i kel i hood can be r e placed by minus i ts absol ute val ue to get an

as cent di recti on . Simi l arl y , f or NRL-adj ustm ent , a def i ni teness modif i cat i on need onl y be appl i ed to

B22

in

(5. 2. 5)

rather t hen to

a,

and i s theref ore easi er to appl y . This t ype of probl an does not ar i se wi th j oi nt FS or FS-aclj us tment .

A second probl an that m ay be encount ered wi th all al gori thms i s

I

SECTION 5 . 2

_{1 44}

I f t hi s happens , a stepsi ze mo difi cation must be used . T he probl em i s less l i kel y f or adj ustm ent al gori thms than f or t he corres ponding j oi nt

al gori ttms since adj us tment increas es the l i kel i hood . H owever

-

adj us tm ent and el imi nat i on al gor i thms mus t reeval uat e

a1 ( 82)

each time a new s tep f or

_�

i s tri ed so that step r educt i on i s us uall y sl ow er than i n the corres pondi ng j oi nt al gori thm s .

T he choi ce of whether or not t o us e el imination or a dj us tment woul d be bas ed on whether the reduct i on i n the number of i terations i s

-

outw ei ghed by the extr a time requi red t o eval uat e

a,

( 82) i n each

i ter at i on . The al gori thms ar e compar ed i n more det ail f or the pro bl em of nonl i near l e as t s quar es i n Se ction 5 . 3 .

To concl ude thi s sect i on , we not e that the i nvers e o f

- a2H 8) / a82

can be use d to consistentl y es timat e the vari ance of the ma ximum

-

l i kel i oood es timat e

₈

of

8

( Secti ons 3. 8 and 4. 3) • I f

NR-el iminati on/ adj ustm ent i s used , thi s corres ponds to estimat i ng

_-

2 -

2

- 1

var (

�

) wi th v22

_{-[a H 81 < a2) , a2) 1 a82J}

f rom the l as t i ter at i on of

the al gor i thm . The covar i ance between

_-

a,

and and the

var i an ce- covar i an ce matri x of

a,

can be es timat ed by Y1 2 .. AY22 and

w1 1 .. -[ a2 H 81 , a2) 1 aa�r 1 -v1 �' ,

res pe cti vel y , where

A =

( a2H 81 , a2)

)_,

( a22.< s, . �) )

as, a�

A i s al so cal cul at ed i n the l ast i ter at i on . I f FS-adj us tment i s us ed , simi l ar var i an ce estimat es can be f ound , whi eh are bas ed on expe cted s e cond deri vat i ves . I f NRL-adj us tment is used , then the vari ances can be estimat ed by

(R1 1 &1 )- 1

i n the us ual way , wher eas f or NRL-el imi nat i on

-

(I'IIX)- 1

can be used to es timat e var ( 82) wher e I is def i ned here t o

SECTI ON 5 . 3 1 45

5. 3 N ONLI NEAR LEAST SQUARES

In nonl inear 1 east s quar es , t he l og-l i kel i hood I S

Since o2 can be f actori zed out , i t can be treat ed as a constant , say

i

= 1 , f or t he pur pose of estimat i ng 8. A maj or appl i cat i on of

el imination and a dj ustment i s i n

s om e par amet ers occur l i nearl y

where t he ve ctor 11 wi th el ements

11 =

nonl i near i n ni ( 8) . ni ( 8) can

I1 ( 82 ) 81

leas t s quar es probl ems where

W e t heref ore exam i ne mode l s

be wri tten as

For e as e of exposition , we restri et o ur attention to the most canmon

cas e , w hi eh i s wher e each of the p2 e l em ents of

�

i s i nvol ve d i n onl y

one of the p1 col umns of I1 ( 82) . R es ul ts simil ar to thos e di scuss ed

bel o.,r al so hol d f or the mor e gener al case . Suppos e t hat t he i ' th

el ement of � , s2 i , i s used i n t he c ( i )th column of I1 < 82) , Ic ( i ) ( 82) . We def i ne t he m atri x

Y( �)

t o have i th column ( oxc ( i ) < 82) 1 o B2i ) f or

i =1 , • • • , p2 and t he p2 xp1 m atri x K to have a one i n i ts ( i , c ( i ) )th

pos i ti on f or i .. 1 , • • • , p2 , and zeros el sewher e . The not at i on di ag (z) i s

used t o denot e a p2xp2 di agonal matri x whos e di agonal el ements ar e gi ven by t he p2- vector

z.

The NR-el imi nati on/ adj ustm ent al gor i thm is parti cul arl y attracti ve when compar e d wi th j oi nt NR here , since

whi ch i s needed i n T heor em 5 . 2 . 1 , is al so used i n t he f orm ul a

s

1 C 82) = [:X1 ( 82) 'I1 ( 82) J-1I1 ( 82) • y . T he C hol es ki f actori zat i on ( 5. 2 . 3 ) i s oft en f ound dur i ng t he eval uat i on of thi s l east squares f orm ul a (see

·Se ction 2 . 3) . A NR- el imination/ a dj ustment i t er ation i s as f ast as

SECTION 5 . 3

1 46

The most wi dely used al gorithm for nonlinear least s quar es is the

NRL al gori thm whi ch , in the context of nonlinear l east squares , is

called the Gauss-Newton (GN ) al gorithn.

A

j oint GN iter ation consists

of the l inear least squares

probl em

of

minimi zi ng

S( 8� . t;)

=

£(1�. 1;) ' £(8;, a;) wi th res pect to a; and a;, where

( 5. 3. 1 )

is a Tayl or s eries for

('J'

- I1 ( 8;) a;) around the previ ous estimat e ,

< t, . �) . The matri ces I1 an d I2 i n ( 5. 2. 4) are equi valent t o I1 C 82)

and Y( � ) . di ag(E81 ) respecti vely . Note that the l at t er is j ust a

res cal i ng of the columns of V( � ) . Joint GN is i denti cal to j oi nt FS .

In document General algorithms based on least squares calculations for maximum likelihood estimation in multiparameter models : a thesis presented in partial fulfilment of the requirements for the degree of Ph D in Statistics at Massey University (Page 150-154)