2.3.1
Comparing models with major structural differences
In this case, a researcher may want to compare models with different numbers of compartments, such as a one-compartment model with a two-compartment model. This method is designed to detect differences in models that affect the overall shape of the curve.Consider a dataset with subjects i, i = 1, ..., n. Each subject has observations yij
forj = 1, ...,ti (tibeing the number of time points or discrete values of the independent
variable for which there are observations for subject i). The statistic can be calculated as follows.
For i= 1 to n:
1. Remove subject i from the dataset
2. Fit a mixed effects model to the subset of the data
3. Accept all parameter estimates from the last run, and freeze the parameters to those values
4. Fit the same model to the whole dataset, without any major iterations, estimat- ing only the post hoc values of the random effects (Phoenix NLME: NITER=0. NONMEM: MAXITER=0, POSTHOC=Y)
5. Calculate predicted values for subject i (the subject that was left out)
6. Take the average of the squared individual residuals for the subject that was left out (over all time points or over all values of the independent variable ti)
Take the average of the quantity in step 6 over all subjects.
This sequence of steps can also be represented by the equation
mP RESS = 1 n n X i=1 ti P j=1 (yij−yˆij,−i)2 ti (2.1)
whereyij is the observed value for theith subject at the jth time point or indepen-
dent variable value. ˆyij,−i is the predicted value for theith subject at thejth time point
or independent variable value in a model where subject i is left out and post hocs are obtained. The number of time points or independent variable values for which there
For purposes of exploration, another statistic that takes into account the weighting can be calculated wtmP RESS = 1 n n X i=1 ti P j=1 W T IRES2 ij,−i ti , W T IRESij,−i = √ wtij,−i(yij −yˆij,−i) ˆ σ−i (2.2)
where W T IRESij,−i is the individual weighted residual for subject i at time or
independent variable value j in a model where subject i is left out and post hocs are obtained, and wtij,−i is the weight defined by the residual error model (equal to the
squared reciprocal of ˆyij,−i for constant CV error models or 1 for additive error models),
and ˆσ−2i is the estimated residual variance.
When comparing models, the following steps should be applied. If the model with less parameters has a value of the statistic less than or equal to that of the model with more parameters, the model with less parameters should be chosen. For cases where the statistic for the model with more parameters is smaller than that of the model with less parameters, and furthermore, if the statistic for the model with less parameters is within one standard error of the statistic of the model with more parameters, the model with the smaller number of parameters should be chosen. Otherwise, if the model with more parameters has a value of the statistic that is more than one standard error below that of the model with less parameters, the model with more parameters should be chosen. The standard error employed should be that of the model with the smallest value of the statistic.
Alternatively, one may follow the same procedure, removing more than one subject at a time. For example, remove 10 percent of subjects at a time, fit a model, obtain predictions for the subjects left out including the post hoc values of the parameters. Square the individual residuals, average those over the independent variable for each
subject, average over subjects.
This method is similar to, or possibly the same as, cross validation methods already established, though it’s not clear whether current methods include calculating the post hoc parameter values to obtain predictions for the subjects that are left out.
2.3.2
Comparing covariate models
In this case, a researcher may want to compare models with and without covariate effects, such as a model with an age effect on clearance versus a model without an age effect on clearance. This method is designed to detect differences in models that affect the equations for the parameters.
Consider a dataset with subjectsi,i= 1, ...,n. Each subject has observationsyij for
j = 1, ..., ti (ti being the number of time points or discrete values of the independent
variable for which there are observations for subject i). The question of interest is whether or not a fixed effect dPdV for a covariate V should be included in an equation for a parameter P, having fixed effect tvP and random effect ηP. The equation for P
could have any of the typical forms used in population PK/PD modeling, for example,
P =tvP ·(V /mean(V))dP dV ·exp(ηP) (2.3)
and one wishes to compare it with a model having no covariate effect
P =tvP ·exp(ηP) (2.4)
If a covariate, V, has an effect on a parameter, P, the unexplained error in P, modeled by
ηP, when V is left out of the model tends to have higher variance. By including covariate
V in the model, we wish to reduce the unexplained error in P, which is represented by
is needed. While the distribution of ηP under the null and alternative hypotheses is
unknown, cross validation can be performed. We propose a statistic for determining whether a covariate, V, is needed for explaining variability in a parameter, P, when P is modeled with a random effect “eta”,ηP.
The statistic can be calculated as follows.
For i= 1 to n:
1. Remove subject i from the dataset
2. Fit a mixed effects model to the subset of the data
3. Accept all parameter estimates from the last run, and freeze the parameters to those values
4. Fit the same model to the whole dataset, without any major iterations, estimat- ing only the post hoc values of the random effects (Phoenix NLME: NITER=0. NONMEM: MAXITER=0, POSTHOC=Y)
5. Square the post hoc eta estimate for the subject that was left out for the parameter of interest
Take the average of the quantity in step 5 over all subjects.
This sequence of steps can also be represented by the equation
nP RESS = 1 n n X i=1 (ˆηPi,−i) 2 (2.5)
Where ˆηPi,−i is the post hoc “eta” estimate for the ith subject for parameter P in
a model where the ith subject was removed, and n is the number of subjects. 29
When comparing models, the following steps should be applied. If the model with less parameters has a value of the statistic less than or equal to that of the model with more parameters, the model with less parameters should be chosen. For cases where the statistic for the model with more parameters is smaller than that of the model with less parameters, and furthermore, if the statistic for the model with less parameters is within one standard error of the statistic of the model with more parameters, the model with the smaller number of parameters should be chosen. Otherwise, if the model with more parameters has a value of the statistic that is more than one standard error below that of the model with less parameters, the model with more parameters should be chosen. The standard error employed should be that of the model with the smallest value of the statistic.
Alternatively, one may follow the same procedure, removing more than one subject at a time. For example, remove 10 percent of subjects at a time, fit a model, calculate the post hoc values for the subjects left out, square the post hoc etas, average them over subjects.