Nonlinear Statistical Models

Nonlinear Statistical Models

y_i =β0+β1x1i+· · ·+βkxki +εi, i =1, . . . ,n, written in matrix form as: y =Xβ +ε , OR       y₁ .. . y_n       =         1 x₁₁ · · · x_p−1,1 1 x₁₂ · · · x_p−1,2 .. . ... ... 1 x_1n · · · x_p−1,n                 β₀ β1 .. . β_p−1         +         ε₁ ε2 .. . ε_n         .

Specifically, we discussed estimation of model parameters, measurements of uncertainty in these estimates and the predicted values, and the use of confidence intervals or significance tests to assess the significance of parameters. Through the matrix formulation, we saw that in linear models (model is alinear function of the parameters), the estimates of linear functions of the response values and their standard errors have simple closed-form expressions and can thus be analytically computed. Unfortunately, this will notbe the case for nonlinear models.

Definition: As before, consider a set ofn observed values on a response variable y assumed to be dependent on k

explanatory variables x =(x₁, . . . ,x_k). This situation can be described through the following nonlinear statistical model:

y_i = f(x0

i,θ) +εi, i=1, . . . ,n ,

where x0

i =[x1i,x2i, . . . ,xki]is the row vector of

observations on kexplanatory variables x₁, . . . ,x_k for the

ithobservational unit,θ =(θ₁, . . . ,θ_p)is a vector of p

parameters, andεi is theitherror term. In a matrix setting, this model can be expressed as:

        y₁ y₂ .. . y_n         | {z } y =         f(x0 1,θ) f(x0 2,θ) .. . f(x0 n,θ)         | {z } f(θ) +         ε₁ ε₂ .. . ε_n         | {z } ε .

• Typically, we assumeε_i iid∼ (0,σ2) and that the explanatory variables in x are fixed.

• If f(θ)=Xθ, where X is the design matrix as defined earlier, then the model reverts to a linear model.

• As Leonid has argued, nonlinear models commonly arise as solutions to a system of differential equations which describe some dynamic process in a particular application. Some common examples of nonlinear models are given below:

1. Exponential Growth/Decay Model: y_i =αeβxi +ε

i, where x is typically time.

– This arises as a solution of the differential equation: d E(y)

d x =βE(y).

2. Two-term Exponential Model:

y_i= θ1

θ₁−θ₂ ”

e−θ2xi −_e−θ1xi—+ε

i.

– This model results when two processes lead to exponential growth or decay. For example, if we are monitoring the concentration of a drug in the bloodstream, which enters the bloodstream from muscle tissue and is removed by the kidneys, then this model is a solution to the differential equation:

d E(y)

d x =θ1E(ym)−θ2E(y),

where y_mis the concentration of the drug in the muscle tissue and y is the concentration in the blood.

3. Mitscherlich Model: y_i=α”1−e−β(xi+δ)—+ε

i. – As an example of where this model is used, let y

be the yield of a crop, let x be the amount of some nutrient, and letα be the maximum

attainable yield. Then if we assume the increase in yield is proportional to the difference between αand the actual yield, we get the following differential equation, to which the model is a solution: d E(y)

d x =β

α−E(y),

whereδis the equivalent nutrient value of the soil.

4. Logistic Growth Model:

y_i = M u

u+ (M−u)∗exp(−kt_i) +εi .

– This model results from a differential equation where we assume the population growth rate decays as the population increases, so that it

reaches acarrying capacity M. The form of this differential equation was discussed in some detail in the past weeks, and is given by:

d E(y) d t =kE(y) 1− E(y) M , E(y|t =0) =u.

where M is the carrying capacity,u is the

population size at time 0, and kcontrols the rate of growth. The form of this model is shown as a fit to the AIDS data (1981-1992) below.

! " # "$ % & ' % ( % & & )

Fitting Nonlinear Models via Least Squares : In principle, fitting a nonlinear statistical model using least squares is no different than fitting a linear one. That is, for the model

y_i= f(x0

i,θ) +εi, we seek to find that parameter vectorθb

that minimizes the sum of the squared errors:

Q(θ|y,X) = n X i=1 ε2_i = n X i=1 [y_i−f(x0 i,θ)] 2 = [y ₋ f(θ)]0[y ₋ f(θ)] =ε0_ε where f(θ)is then×1 vector of f(x0

i,θ)values.

DifferentiatingQ with respect to theθ_j (in an effort to minimizeQ), setting the resulting equations to zero and evaluating atθ =θbgives: ∂Q(θ|y,X) ∂ θ_j θ=θb =−2 n X i=1 [y_i−f(x0 i,θb)]  ∂ f(x 0 i,θb) ∂θb_j  =0,

∀j=1, . . . ,p. These resulting equations are p nonlinear equations in the pparameters, which in general cannot be solved analytically to obtain explicit solutions for θ. As a result, numerical methods are used to solve these

nonlinear systems. In MatLab, there are several numerical methods that can be used, although the default is a

generalization of the standard Gauss-Newton method, known as the Levenberg-Marquardt Algorithm.

Gauss-Newton Method: This method, like others that may be used, is iterative in nature, and requires starting values for the parameters. These starting values are then

continually adjusted to home in on the least squares

solution. The method is conducted via the following steps: 1. Consider taking a 1st-order Taylor expansion of f(θ)

about some starting parameter vectorθ0_:

f(θ)≈ f(θ0_{) +F(θ}0_)(θ

−θ0), (1)

where F(θ0)is the n×pmatrix of partial derivatives evaluated atθ0 and the n data values x0

i, as given below: F(θ0₎₌              ∂[f(x0 1,θ 0_)] ∂ θ1 ∂[f(x0 1,θ 0_)] ∂ θ2 · · · ∂[f(x 0 1,θ 0_)] ∂ θ_p ∂[f(x0 2,θ 0_)] ∂ θ₁ ∂[f(x0 2,θ 0_)] ∂ θ₂ · · · ∂[f(x0 2,θ 0_)] ∂ θ_p .. . ... ... ∂[f(x0 n,θ0)] ∂ θ1 ∂[f(x0 n,θ0)] ∂ θ2 · · · ∂[f(x 0 n,θ0)] ∂ θ_p              .

2. Calculate f(θ0_{) and F(θ}0_{). Recognizing f(θ)as an}

approximatelylinearized model inθ from equation

(1), linear least squares is used to update the starting values and obtain the solution at the next iteration. 3. Repeat steps 1 & 2 until the change in the parameters

is below some tolerance.

Some Key Results: Based on this same Taylor expansion, Gallant (1987) established the following asymptotic (large sample) results:

1. If we assumeε∼N(0,σ2I), then the least squares parameter vectorθbisapproximatelynormally distributed with meanθ and variance matrix Var(θb) =(F0_F)−1_σ2_{, written:}

b

θ ∼• N[θ,(F0_F)−1_σ2_{] .}

• In practice, we compute F(θb)as an estimate of F(θ)and estimate σ2 with MSE=SSE/(n−p) to give the asymptotic variance-covariance matrix for

b θ as:

• This gives an approximate 100(1−α)% confidence interval for a given parameterθ_i as:

b θ_i±t_1−α,n−p·s(θb)_ii = θb_i±t_1−α,n−p·ÆMSE(Fb0_Fb₎−1 i i , where(bF0_Fb₎−1 i i is the i th _{diagonal entry of(Fb}0_Fb₎−1_.

2. Again assuming the errorsε∼N(0,σ2I), then: Var(_by) =F(F0_F)−1_F0_σ2_{, which can be estimated} as: s2_(by)=d_{Var(by) =}”_Fb(Fb0_Fb₎−1_Fb0—_{·MS E.}

• This technique of quantifying uncertainty through the use of a first-order Taylor series expansion is known as the Delta Methodand was the most commonly used technique for finding approximate standard errors before the advent of modern computing.

• Gallant (1987) illustrated several cases where these approximations are poor, giving confidence intervals and hypothesis tests which are completely wrong. • Some more modern techniques for estimating

standard errors and subsequently performing statistical inferences in nonlinear models are the method of bootstrapping, andMarkov chain Monte Carlo (MCMC) methods.

Example: Data were collected on the number of big horn sheep on the Mount Everts winter range in Yellowstone between 1965 and 1999 (obtained from Marty Kardos). Consider the time plot of these data below. The population suffered a die-off resulting from the infectious kerato-conjunctivitis in 1983. Marty indicates that this is fairly typical for bighorn sheep populations: logistic-like population growth up to a carrying capacity followed by disease-related die-offs, and subsequent recovery. For now, consider modeling the initial population growth from 1965 to 1982 using a logistic growth model. Here, we will fit the model, and use it to illustrate the calculation of standard errors for the parameter estimates from this nonlinear model. * + , - * + . / * + . - * + 0 / * + 0 - * + + / * + + - 1 / / / / - / * / / * - / 1 / / 1 - / 2 3 45 6 7 8 9 : ; < = > ? @ A < ; B C A : : D EF G H I JK L M IN OPQ RF M H S R L L T UF G H IN V W X Y Z [ W X X X

The logistic growth model is a 3-parameter nonlinear model of the form:

y_i = M u

u+ (M−u)exp(−kt_i) +εi, where:

y_i = the number of bighorn sheep at time t_i,

M = the carrying capacity of the population,

u = the number of sheep at time 0 (1964 here),

k = a parameter representing the growth rate. Fitting this model to the 1965-1982 data with nonlinear least squares using the nlinfitfunction in MatLab gives the following fitted model:

by_i = Mbub b

u+ (Mb −u_b)exp(−bkt_i), where Mb =221.0513, u_b=27.4643, bk=0.3279, as computed using the Gauss-Newton method outlined earlier. The fitted model is shown in overlay below. Is this a good model? \ ] ^ _ \ ] ^ ^ \ ] ^ ` \ ] a b \ ] a c \ ] a _ \ ] a ^ \ ] a ` \ ] ` b \ ] ` c b d b \ b b \ d b c b b c d b e f gh i j k l m n o p q r s t o n u v t m m w xy z { | } ~  { € €  ‚ | ƒ ~ „ … † ‡ ˆ ‰ Š ‹ ‡ ˆ Œ  Ž  y „ {  | z y … „y ‘ ’y„

The code for fitting the logistic model is given below and can be found on the course webpage under the files sheep.mandlogistic.m. The first file is the driver program and the logistic.mfile defines the logistic function.

% ============================================= % % Fits a logistic growth model to the 1965-1982 %

% sheep data, and plots the fitted model. %

% ============================================= % sheep82.year = sheep.year(sheep.year<=1982);

% Take only years before 1983

sheep82.count = sheep.count(sheep.year<=1982); % Take only cases before 1983

time = sheep82.year-1964;

% Defines the year-1964 variable

beta = [221 27 0.33]; % Parameter starting values [betahat,resid,J] = nlinfit(time, ...

sheep82.count,@logistic,beta);

% Performs NLS, returning the parameter

% estimates, residuals, & Jacobian

yhat = sheep82.count-resid; % Computes predicted counts

plot(sheep82.year,sheep82.count,’ko’,... sheep82.year,yhat);

% Plots sheep counts vs. year w/ fitted curve xlabel(’Year’,’fontsize’,14);

ylabel(’Number of Bighorn Sheep’,’fontsize’,14); title(’Bighorn Sheep Counts: 1965-1982’,...

’fontsize’,14,’fontweight’,’b’);

• I tried a few different things to try to improve the fit, including a “Weibull-like” modification to the logistic growth model, where I added a parameter in the exponent of the model yielding ageneralized logistic growth model:

y_i = M u

u+ (M−u)exp[−ktγ_i] +εi , whereγis a parameter controlling the degree of curvature in the logistic shape. Ifγ=1, this reverts to the usual logistic growth model.

• This change is “Weibull-like” in the sense that we have added a parameter in the exponent of an exponential function, much like the addition of theγ-parameter in going from exponential growth to a Weibull growth model.

• This “Weibull-like” curvature-adjustment to the logistic model has a corresponding differential equation to which it is the solution. This equation has the form:

d E(y) d t =γt γ−1_ku₁₋ u M ‹ ,

where M is the carrying capacity,u is the population size at time 0, kcontrols the rate of growth, andγis this curvature piece.

• When parameterized this way, it turns out that the resulting parameter estimates for k &γare unstable because they are highly correlated. An alterative, but equivalent way to parameterize this model is as follows:

y_i = M u

u+ (M−u)exp[−(kt_i)γ_] +εi .

This is the form of the model we will use for the remainder of our treatment of this generalized logistic model.

• Essentially,γcontrols thesteepnessof the S-shape in the logistic model, where larger values ofγcorrespond to steeper curves. Since our logistic model for the sheep data appears to not be steep enough, we would expect this parameter to be larger than 1.

• Fitting this generalized logistic model produced the fitted curve and parameter estimates as given on the next page. “ ” • – “ ” • • “ ” • — “ ”˜ ™ “ ”˜ š “ ”˜ – “ ”˜ • “ ”˜ — “ ” — ™ “ ” — š • ™ — ™ “ ™ ™ “ š ™ “ – ™ “ • ™ “ — ™ š ™ ™ š š ™ š – ™ › œ ž Ÿ   ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª ¥ ¤ « ¬ ª £ £ ­ ®¯ ° ± ² ³ ´ µ ± ¶ ¶ · ¸ ² ¹ ´ º » ¼ ½ ¾ ¿ À Á ½ ¾  ÃÄ Å ¯º ± Æ ¶ ´¶ ³ Ç È¯ É ¶ Ê Ë ² ° ¯ » º¯ Ì Í¯º b M = 207.6399 b u = 67.9731 bk = 0.1380 b γ = 3.4542

Does this seem like a better fit?

The MatLab code used to fit this generalized logistic model is shown below:

% ===================================== %

% Fits a generalized logistic growth %

% model to the 1965-1982 sheep data, %

% and plots the fitted mode in overlay. % % ===================================== % beta2 = [221 27 0.3 2];

% Parameter starting values

[betahat2,resid2,J2] = nlinfit(time, ... sheep82.count,@genlogistic,beta2); % Performs NLS, returning the parameter

% estimates, residuals, & Jacobian

yhat2 = sheep82.count-resid2; % Computes predicted counts

plot(sheep82.year,sheep82.count,’ko’,... sheep82.year,yhat2);

% Plots the sheep counts vs. year with

% fitted curve in overlay

xlabel(’Year’,’fontsize’,14); ylabel(’Number of Bighorn Sheep’,

’fontsize’,14);

title(’Bighorn Sheep Counts (1965-1982) with Generalized Logistic Fit’,’fontsize’,14,... ’fontweight’,’b’);

Which of these parameters do you expect to be the least stable (i.e.: which do you expect to have the highest

variability?) To get the standard errors of these parameters inθ =(M,u,k,γ), we need to compute the Delta Method approximate variance matrix, given as:

s2(θb)=Var(Ó θb) =(Fb0_Fb₎−1_{·MSE .}

To compute the Fb-matrix, we first compute the partial derivatives of

f_i(θ) = f(x0

i,θ) =

M u

u+ (M−u)exp[−(kt_i)γ_]

with respect to M,u, k, and γrespectively. Doing so with repeated use of the product (quotient) rule gives:

∂ f_i(θ) ∂M = u2{1−exp[−(kt_i)γ]} {u+ (M−u)exp[−(kt_i)γ]}2, ∂ f_i(θ) ∂u = M2exp[−(kt_i)γ] {u+ (M−u)exp[−(kt_i)γ_]}2, ∂ f_i(θ) ∂k = M u(M−u)γkγ−1tγ_i exp[−(kt_i)γ] {u+ (M−u)exp[−(kt_i)γ]}2 , ∂ f_i(θ) ∂ γ = M u(M−u)(kt_i)γexp[−(kt_i)γ]log(kt_i) {u+ (M −u)exp[−(kt_i)γ]}2 .

Plugging the least squares estimates θb= (Mb,u_b,bk,γ)_b into these derivatives yields the Fb-matrix shown below: Fb=             ∂[f(t₁,θb)] ∂Mb ∂[f(t₁,θb)] ∂u_b ∂[f(t₁,θb)] ∂bk ∂[f(t₁,θb)] ∂γ_b ∂[f(t₂,θb)] ∂Mb ∂[f(t₂,θb)] ∂u_b ∂[f(t₂,θb)] ∂bk ∂[f(t₂,θb)] ∂γ_b .. . ... ... ... ∂[f(t₁₇,θb)] ∂Mb ∂[f(t₁₇,θb)] ∂u_b ∂[f(t₁₇,θb)] ∂bk ∂[f(t₁₇,θb)] ∂γ_b             =                        0.0001 1.0004 0.0012 −0.0969 0.0053 1.0161 0.0553 −1.9492 0.0153 1.0415 0.1532 −3.6359 0.0371 1.0816 0.3439 −5.0945 0.0825 1.1242 0.6710 −5.0515 0.1729 1.1274 1.1459 −1.5678 0.3349 1.0106 1.6292 6.4654 0.5652 0.7222 1.7487 15.1716 0.7904 0.3683 1.2833 16.5367 0.9293 0.1276 0.6181 10.3187 0.9834 0.0303 0.1980 3.9937 0.9973 0.0050 0.0427 0.9987 0.9997 0.0006 0.0061 0.1615 1.0000 0.0000 0.0006 0.0165 1.0000 0.0000 0.0000 0.0010 1.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000                        ,

where the actual values were computed in MatLab. Finally, computing MSE·(Fb0_Fb₎−1 _{to get the}

variance-covariance matrix of the parameter estimates, extracting the diagonal entries of this

4x4 matrix, and taking square roots yields the standard errors of the parameter estimates. The MatLab code to do this is given below:

% ============================================= %

% Computes the Delta Method parameter estimate %

% standard errors for a 4-parameter generalized %

% logistic model. %

% ============================================= % n = length(sheep82.year); % Sample size

p = length(beta2); % No. model parameters

mse = sum(resid2.^2)/(n-p);

% Computes the mean squared error (MSE) Mhat = betahat2(1); uhat = betahat2(2); khat = betahat2(3); ghat = betahat2(4);

% Renames the three parameter estimates den = (uhat+(Mhat-uhat)*exp(-(khat* ...

time).^ghat)).^2;

dfdM = uhat^2*(1-exp(-(khat*time).^... ghat))./den;

% Computes df/dM (nx1 vector)

dfdu = Mhat^2*exp(-(khat*time).^ghat)./den; % Computes df/du (nx1 vector)

dfdk = Mhat*uhat*(Mhat-uhat).*(time.^... ghat).*ghat.*khat.^(ghat-1).*... exp(-(khat*time).^ghat)./den; % Computes df/dk (nx1 vector) dfdg = Mhat*uhat*(Mhat-uhat).*(khat*... time).^ghat.*log(khat*time).*... exp(-(khat*time).^ghat)./den; % Computes df/dg (nx1 vector) Fhat = [dfdM dfdu dfdk dfdg];

% Combines derivatives into nx4 matrix varmat = inv(Fhat’*Fhat).*mse;

% Inverts estimated (F’F)*MSE matrix separ = sqrt(diag(varmat));

% Standard errors are square root

% diagonal elements of varmat

tstar = tinv(1-.025/p,n-p); % Computes Bonferroni t-value

ci = [betahat2’-tstar*separ betahat2’+ ... tstar*separ];

% Computes Bonferroni CIs for the 3 parameters

betahat2 = 207.640 67.973 0.1380 3.4542

separ = 4.178 7.166 0.0091 0.8093

• This provides approximate Bonferroni-adjusted

simultaneous 95% confidence intervals for the four

parameters (using t_1−.025/4(n−p) =t_.99375(17−4) = tinv(.99375,13)=2.8961) as: b M±t_.99375(n−p)·SE(Mb) =207.640±2.8961(4.178) = (195.54, 219.74). b u±t_.99375(n−p)·SE(u_b) =67.973±2.8961(7.166) = (47.22, 88.73). bk±t_.99375(n−p)·SE(bk) =0.1380±2.8961(0.0091) = (0.1116, 0.1645). b γ±t_.99375(n−p)·SE(_bγ) =3.4542±2.8961(0.8093) = (1.11, 5.80). • In MatLab, the Delta Method of computing standard

and can be easily obtained after calling the nlinfit function, by typing:

ci2 = nlparci(betahat2,resid2,J2)

% Computes NLS Delta Method CIs

This returns approximate 95% individualconfidence intervals for the three model parameters based on the Delta Method standard errors.

• We will next explore the use of bootstrapping as an alternative to using Delta Method-based standard errors. Both techniques have their problems and consider parameter uncertainty for each parameter

separately rather than jointly. Later, we will briefly

explore the use of Markov chain Monte Carlo (MCMC) methods which account for the dependence among model parameters in measuring uncertainty.