of how well the model predicts the values at unknown locations. Cross-validation and validation help you make an informed decision as to which model provides the best predictions. The calculated statistics serve as diagnostics that indicate whether the model and/or its associated parameter values are reasonable. Cross validation and validation withhold one or more data samples and then make a prediction to the same data location. In this way, you can compare the predicted value to the observed value and from this get useful information about the kriging model (e.g., the semivariogram parameters and the searching neighborhood). The difference between cross-validation and validation will be discussed next.
Cross-validation uses all of the data to estimate the autocorrela- tion model. Then it removes each data location, one at a time, and predicts the associated data value. For example, the diagram below shows 10 randomly distributed data points. Cross- validation omits a point (red point) and calculates the value of this location using the remaining nine points (blue points). The predicted and actual values at the location of the omitted point are compared. This procedure is repeated for a second point, and so on. For all points, cross-validation compares the measured and predicted values. In a sense, cross-validation cheats a little by using all of the data to estimate the autocorrelation model. After completing cross-validation, some data locations may be set aside as unusual, requiring the autocorrelation model to be refit.
And so on for all points
Validation first removes part of the datacall it the test dataset and then uses the rest of the datacall it the training datasetto develop the trend and autocorrelation models to be used for prediction. In the Geostatistical Analyst, you create the test and training datasets using the Create Subset tools. Other than that, the types of graphs and summary statistics used to compare predictions to true values are similar for both validation and cross-validation. Validation creates a model for only a subset of the data, so it does not directly check your final model, which should include all available data. Rather, validation checks whether a protocol of decisions is valid, for example, choice of semivariogram model, choice of lag size, choice of search neighborhood, and so on. If the decision protocol works for the validation dataset, you can feel comfortable that it also works for the whole dataset.
Geostatistical Analyst gives several graphs and summaries of the measurement values versus the predicted values.Starting with the plots, a scatter plot of predicted versus measurement values is given. One might expect that these should scatter around the 1:1 line (the black dashed line below). However, the slope is usually less than one. It is a property of kriging that tends to
underpredict large values and overpredict small values, as shown in the following figure.
The fitted line through the scatter of points is given in blue with the equation given just below the plot. The error plot is the same as the prediction plot, except here the true values are subtracted from the predicted values. For the standardized error plot, the measurement values are subtracted from the predicted values and then divided by the estimated kriging standard errors. All three of these plots help to show how well kriging is predicting. If all the data was independent (no autocorrelation), all predictions will be the same (every prediction would be the mean of the measured data), so the blue line would be horizontal. With autocorrelation and a good kriging model, the blue line should be closer to the 1:1 (black dashed) line. You can also see the scatter about the line (a few are given in the figure above as green lines). The tighter the scatter about the 1:1 line, the better.
The final plot is a QQPlot. This shows the quantiles of the difference between the predicted and measurement values divided by the estimated kriging standard errors and the corre- sponding quantiles from a standard normal distribution. If the errors of the predictions from their true values are normally distributed, the points should lie roughly along the dashed line. If the errors are normally distributed, you can be confident of using methods that rely on normality (e.g., quantile maps in ordinary kriging).
See Chapter 4, Exploratory Spatial Data Analysis, for more on QQPlots.
Finally, some summary statistics on the kriging prediction errors are given in the lower left. You use these as diagnostics for three basic ideas:
1. You would like your predictions to be unbiased (centered on the measurement values). If the prediction errors are unbiased, the mean prediction error should be near zero. However, this value depends on the scale of the data, so to standardize these the standardized prediction errors give the prediction errors divided by their prediction standard errors. The mean of these should also be near zero.
2. You would like your predictions to be as close to the measurment values as possible. The root-mean-square prediction errors are computed as the square root of the average of the squared distances of the green lines in the prediction plot above. The shorter the green lines, the closer the predictions are to their true values, and the smaller the root-mean-square prediction errors. This summary can be used to compare different models by seeing how closely they predict the measurement values. The smaller the root-mean- square prediction error, the better.
3. You would like your assessment of uncertainty, the prediction standard errors, to be valid. Each of the kriging methods gives the estimated prediction kriging standard errors. Besides making predictions, we estimate the variability of the predic- tions from the measurement values. It is important to get the correct variability. For example, in ordinary kriging (assuming
USING ANALYTICAL TOOLS WHEN GENERATING SURFACES 191
If the average standard errors are greater than the root-mean- square prediction errors, then you are overestimating the variability of your predictions; if the average standard errors are less than the root-mean-square prediction errors, then you are underestimating the variability in your predictions. Another way to look at this is to divide each prediction error by its estimated prediction standard error. They should be similar, on average, and so the root-mean-square standardized errors should be close to one if the prediction standard errors are valid. If the root-mean-square standardized errors are greater than 1, you are underestimating the variability in our predictions; if the root-mean-square standardized errors are less than 1, you are overestimating the variability in your predictions.
The Cross Validation and Validation dialog box
Line of best fit 1:1 Line
Results from cross-validation or validation Summary statistics Cross-validation scatter plot
USING ANALYTICAL TOOLS WHEN GENERATING SURFACES 193