SLAC PUB-2504 STAN-CS-80-799 May 1980
(M)
MULTIDIMENSIONAL ADDITIVE SPLINE APPROXIMATION* Jerome H. Friedman
Stanford Linear Accelerator Center Stanford, California 94305
Eric Grosse
Computer Science Department Stanford University
Stanford, California 94305 Werner Stuettle
Stanford Linear Accelerator Center and
Department of Statistics, Stanford University Stanford, California 94305
ABSTRACT
We describe an adaptive procedure that approximates a function of many . variables by a sum of (univariate) spline functions sm of selected linear combinations am-x of the coordinates
y(x) = c
l~~;rrts;M smbm4
The procedure is nonlinear in that not only the spline coefficients but also the linear combinations are optimized for the particular problem. The sample need not lie on a regular grid, and the approximation is affine
ion invariant, smooth, and lends itself to graph
values, derivatives, and integrals are cheap
ical interpretation. Funct to evaluate.
*Work partially supported by the Department of Energy under contract DE-AC03-76SF00515, and by the National Science Foundation under grant MCS 78-17697.
1 1. Introdktion
Mu&dimensional surface approximation is recognized as an important prob- lem for which several methodologies have been developed. The aim is to con- struct an approximation 4(x) to a p-dimensional surface y = f(x) on the basis
of (possibly noisy) observations { (yi, xi)}l<icn. -- Most existing methods, such as
tensor product splines, kernels, and thin plate splines (for a survey, see Schumaker [1976]), are linear in that.
4(x) = c wyi,
l<i<n
where the weights ( wi} depend only on x and { xi}l<i< n, but not on { yi},< i< ,.,: --
These methods have the advantage that they are straightforward to compute and their theory is tractable. In pract.ice, however, they are limited because they
cannot take advantage of special properties of the surface. Due to the inherent
sparsity of high-dimensional sampling, procedures successful in high dimensions must be adaptive and thus nominear.
In this paper we describe an adaptive procedure that approximates f(x) by a sum of (univariate) spline functions sm of selected linear combinations a, * x of the coordinates
4(x) = C Sm(am s 4.
l<m<M
51)
The ‘procedure is nonlinear in that not only the spline coefficients but also thelinear combinations are optimized for the particular problem. *
2. The algorithm
The spline function sm along am s x is represented as a sum of j,,, B-splines
[de Boor, 1979) of order q
2
The approximation 4(x) (g’ iven by equations (1) and (2)) is specified by the direc-
tions tam)l<m<M, the knot sequences along am s x for 1 5 m 5 M, and the
B-spline^coefficients (Pmj}lCmCM,l<j<j,. For particular {a, ), the knots are ’
placed heuristically and then-the{ Pmj } are determined by (linear) least squares.
The residual sum of squares from this fit is taken to be the inverse figure of merit
for (Bmll<m<M*
- -
Following Friedman, Jacobson, and Stuetzle [1980], the approximation is con-
structed in a stepwise manner: given ( a, }1< m< M--l, find aM to optimize the
figure of merit of ( am >I< m< M. Terminate when the figure of merit - - is not
significantly improved.
3. Imp’lementation
The most difficult part of the algorithm is finding each direction am. We per- form a numerical search using a Rosenbrock’method [Rosenbrock, 1966) modified for the unit sphere, starting at the best coordinate direction. On any given search, there is no guarantee that the global optimum will be found. If the local optimum
is insignificant, the search is restarted at random dire.ctions. This guards against
premature termination, If the local optimum is significant but not identical to the global optimum, no great harm is done because a new search is performed in the next iteration on an object function for which the previous optima have been deflated. Each iteration of the optimizer requires 3.5 p function evaluations,
on the average, where p is the dimension of x. Two iterations are nearly always
sufficient.
For high dimensionality, the computation is dominated by the evaluations of the object fu.nction. Since it is not crucial to find the precise optimum, con-
siderable savings can be achieved by substituting a similar;but much less ex-
pensive figure of merit during the search for a new direction. For this figure of
merit not only the previously foun’d directions but also the corresponding spline
coefficients are held fixed. The new direction can thus be found by considering
3
residuals are modelled by a smooth based on local linear fits [Cleveland, 19791,
[Friedman, Jacobson, and Stuetzle, 19SOJ. The characteristic bandwidth (that is,
the aver^age fraction of observations used in each local fit) is taken to be inversely
proportional to the number of knots, The residual sum of squares from the smooth is the figure of merit used for the smooth. Solving the least squares problem for the original figure of merit requires
operations, while the new figure of merit can be evaluated in roughly n operations using updating formulas for the moving average. The least squares problem has
to be solved only once for each iteration to determine the new model after am has
been found.
To solve the least squares problem, we form the normal equations and use a
pseudo-inverse, since the design matrix might not:have full rank. The singularity which arises form the inclusion of a constant term for each direction is remedied by simply dropping one column per direction from the design matrix. Higher order singularities caused, for example, by the linear terms for three co-planar directions, are not explicitly taken care,of, but are handled by the pseudo-inverse.
Our knot placement procedure is motivated by the sequential nature of the
algorithm. At each iteration, the knot positions are required for the least squares fit, after the new direction has been found. Our model at this point is the spline fit of the previous iteration, plus the moving average smooth along the newly found direction. The’knot placement is based on the residuals { ri} from this model. Multidimensional structure in these residuals due to incompleteness of the model manifests itself as high lo.cal variability in the scatterplots of ri against a,,, . Xi. In order to preserve the ability of fitting this structure in further iterations, it is important to avoid accounting for it by spurious fits along existing directions. For
this reason we place fewer knots in regions of higher local variability. Since the
4
The knots along a direction am are placed as follows: the smooth described
above is applied to { (ri, am** xi) }l<i<n and the local variability vi at each point is --
taken to^be the average squared residual from its local linear fit. The Winsorized local variabilities are defined by
if Ui > 2U
if .Vi < it,
otherwise
(where u = ixl<i<, Vi), and then are scaled so that zlCiKn & = 1. The --
knots ( tr } are placed to divide the line into intervals with equal content of &:
for each I, 1 c 1
j,---q+l= am~xi~(wl+l) -. Wi
4. Examples
In this section we present and discuss the results of applying the Multidimensional Spline Approximation method (MASA) to four examples. (A
FORTRAN program implementing MASA is available from the authors.) The first three examples were suggested elsewhere for testing surface approximation procedures. The function in the fourth example was studied in connection with a problem in mathematical genetics.
The first exam.ple is taken from Friedman [1979]. In this example uniformly distributed random points { xi 1 1 5 i 5 200) were generated in the six- dimensional hypercube [O, I]“, Associated with each point Xi was a surface value
yi = 10 sin(Xzi(l)zi(2)) + 2O[Zi(3) -- 0.512 j- lOZi(4) + 5zi(5) + OZi(6) + Ci,
where the { ci} were independent identically distributed standard normal. The inverse figures of merit for the approximation with M = 1,. . ., 4 terms were 6.71,, 4.29,1.87,0.97. In three restarts, the figure of merit did not decrease below
5
0.86,. so h = 4 was chosen. The four linear combinations and the corresponding
spline functions are shown in figures 1.1-1.4. (For completeness, the program paramerers are also listed; see the program comments for a detailed explanation.) The spline along the first linear combination (figure 1.1) is seen to model the linear part of the surface. The second term in the approximation (Egure 1.2) models
the additive quadratic d.ependence on ~(3). The Enal two terms (figures 1.3, 1.4)
, model the interaction between ~(1) and z(2). The F2 norm of the error ]]f - $112 was 0.57.
Although the full advantages of MASA compared to other procedures are realized in higher dimensional or noisy settings, we applied it to two bivariate examples used by Franke [1979] to compare a number of interpolatory surface approximation schemes.. For both examples 100 uniformly distributed random points in. the unit square [0, 1]2 were generated. The function in Franke’s first example’ is - - f(s, y) = 0.75 exp[--- (” 2)2 I(” 2)2 ] + 0,75exp[- (9x + 1)2 - 9y + l].’ 1o + 0.5 exp[-(” ““1)1+(9y-3)21 -1 0.2 exp[-(92 - 4)2 ” (9y - 7)2].
Considerations similar to those in the previous example led to an approximation
with three terms. The linear combinations and corresponding spline functions are
shown in figures 2.1-2.3.
The function in Franke’s second example is
f(z, y) = i[tanh(Sy - 92) + I].
For this case the approximation used only one term, shown in figure 3.1.
Since different random points were used in Franke’s and our tests, precise
I
6
of magnitude larger errors than the best methods in Franke’s trials (global basis function methods) while on the second example, MASA gave an order of mag- nitude s%aller errors than the best methods. These results are not surprising.since the peak-shaped basis functions of the global basis methods are especially suited
for representing the peaks of the first example, whereas the ridge-shaped basis
functions of MASA are especially suited to the second example. Unfortunately, peak-shaped basis functions are not appropriate for moderate or higher dimen- sionality. The difficulty is that in order to achieve a smooth fit, the width of the basis peaks. needs to be comparable to the distance between data points. For
n uniformly distributed random points in a p-dimensional hypercube (0, l]“, the
typical nearest neighbor distance is (k)*. In particular for n = 1000 and p =
10, this distance ‘is 0.5, and.for p = 20 is 0.7. Thus variation of the surface
over distances small,compared to such large interpoint distances cannot be well approximated with these global basis functions methods.
Our fin.al example is a 19-dimensional function encountered by Carmelli and
Cavalli [1979]. An important question is the structure .of this function near its
minimum. We sam.pled the function at 200 points uniformly distributed in a small
hypercube centered at the minimum found by numerical optimization and applied
MASA. Th.e inverse figure of merit for the best constant Et was 13.3. The inverse
figure of merit for M = 1 was 0.78. In 30 restarts, the Egure of merit did not
decrease below 0.42. Figure 4.1 gives the linear combination and corresponding
spline function. This picture shows considerable structure that was not revealed
in. the original study.
5. Discussion
MASA can be expected to work well to the extent that the, surface can be
approximated by a function of the form (1). Of course in the limit M --+ co all
smooth su.rfaces can be represented by (l), but even for moderate M functions of this form constitute a rich class.
As seen in the previous section, an advantage of using essentially one-
7
entire mddel can be represented by graphing Sm(am s x) against a, - x and by
specifyin.g { a, } l<m<M ( per h p g phically for a s ra p = 2 or 3). Additionally, ade-
quacy o?’ thti knot placement can be judged using the M plots of ,the residuals from the final model against a, a x. Proper termination of the algorithm can be
assured by monitoring at each iteration*th.e plot of the residuals from the model
of the previous iteration along the newly found direction.
The problem of sparse sampling in high dimensions is not encountered, since
MASA is fitting one-dimensonal projections of the e&ire sample. The sample need
not’ lie on a regular grid, and the approximation is affine invariant ai;ld smooth.
Function values, derivatives, and integrals are cheap to evaluate. In addition, since the approximation is locally quadratic for Q = 3, optimization algorithms can be expected to converge rapidly.
Carl de Boor
[1978] A practical guide to spjines, Springer-Verlag.
1 Dorit Carmelli and L L Cavalli-Sforza
[1979] 772e genetic origin of the Jews: a multivariate approach, Human Biology
51, 41-61.
William S Cleveland
[1979] Robust locally weighted regression and smoothing scatterplots, J Amer
Stat Assoc 74, 829-836.
Richard Franke
[1979] A critical comparison of some methods for interpolation of scattered.
8
Jerome H’ Friedman, Mark- Jacobson, and Werner Stuetzle
[1980] Projection pursuit regression, submitted to,J Amer Stat Assoc.
H H Rosenbrock
[1960] An automatic method for finding the greatest or least value of a func-
tion, Computer J 3, 175-184.
‘Larry L Schumaker
[1976] Fitting surfaces to scattered data, in Approximation Theory, (G G
SP;I;;6;UNC$IC);gND
. . ~~s6ALO~G7~I~ECT~O~5~~ . . . --A. 0146 PLC%&TATI&TICS = 0. 0. 0. / 0. 9. 0. / 0. 0. 0. PRCJ ON Y 3000000000000000010~00000100000000001000000000000000000003 ---+--- --- +---+- ---4-- -+----.-- &XI3 7 ****I LEFT BIN ECGE **** j *** I *** I *** *** z ** *** z ** I ** + *** ** ; ** I ** *** z ** *** ** i *** I ** + t** ** II *** ** : *** I ** ** z ; *** I ** + *** I ** I *** I *** *** I *** I*** 1* t :3 + + + 3: -I-- ----_-- --- --- --- 00000000~l;t;;;0000000~~~~~~~~~~lllllll~~ii~i~llllllll . . . ...* 333334444555556666777778888999990000111112222333334444455556 135793468035791468025791368024791358024691357024681357924680 02468~246802468025791357913579135791357913579135791357913579 888888888999999990080000001111111112222222233333333344444444 ? figure 1.1"'~1;~9"~1m9qND . . PLOT&TATASTICS = ON X AXIS 0.0006 I;'$; -0:4848 -3.0737 13,. ;;g -3:5591 r;* g;z -5:6626 :;-;;g -6:1481 -6.3099 -6.4717 LEFT BIN ECGE ? 10 0. 0. 0. / 0. NR ii.0 9. 0. / 0. 0. 0. 300000000000000000001000000000000001000000000000100000000003 $** ; T * T I .. I ** T * *t *T * *i; * *1 ** ** + * * I ** ** I * * I ** ** T ** * * ** ** * * ** ** ** ** * * ** ** * ** ** * ** ** * ** ** ** x* ** * ** ** ** ** ** ** ** ** *** ** *** ** *** *** ***** *** ******** + + + +3
+ --- ---em +- ---_ --- -I-- ---me 0000000~0000000000000000000000000000000000000000000000000001 . . . ..*...*... 000000011111122222333333444444555555666667777778888889999990 102457912468913568013578024579124689134680135780245790246791 074196318630852074297419631853085207429641963185307529742964 314825936047148259360~71482592603714815926037048159~69370481 PRQJ ON Y AXIS figure 1.2
11 SP~Ibk&JNC~IQ'JD k$JF5lAL!C$G0;;;ECT;Oi NR l . . . . 03.00*3 PLOTJTTATLSTICS = 0. 0. 0. / 0. 9. 0. / 0. 0. 0. ON X AXIS 300000000000000001000000000100~000000001~0000000~00000003 ?. ,222 2 $---t---3--- --- i I J..Z7”ktd 1.6641 I. 'I 83640 1.0638 7 x-3; ":1634 -~,.1368 2*%?
3;
:
1* I*** -I- *** ---f---y -- ******* *** *** *** *** ** *** *** ** **** *** ** * ** ** * ** * ** * ** * ** * * ** * * ** * * ** * * ** . * * I,.,J,A I -1.0373 I *** **** -A.3374 I ---- - ************* -l.bJ-lb I z$';;;g i: -2:5381 I r;=8,g ; -3:4386 I -3.7387 + -a;0389 I I~*~~~~ ; -4:9394 I I;-;;;: ; -5:8398 I -6.1400 I -6.4402 I -9.1416 I -9.4418 I -9.7419 +3 + + + 3-t+- ---_--_ e-_-e- e-_--v-- ---_ --- ---es LEFT 000000000~0E10000"0~00000000~0000000~00011111~111~1111 BIN . . . EJXE 000111122222333334444555556666677777888899999000001111222223 579135802468035791357024680257913579246802469135791468024681 345789123467801245689023467801235679013457891235679012456890 147036036926925825814714703703693692582581481470470369369259 ? figure 1.3
12 S'~I~;O;UNC;IWND ~O&ALf;G0~~;ECT;O;6J . . . i.0072 PL;CY&TAT3&TICS = 0. 0. 0. / 0. 9. 0. / 0. 0. 0. PROJ ?00000000000000010000000000010000000001000000000000000000003 i&z ON X AXIS 5.1486 5.0070 4.8654 4.7238 t*2% 4:2991 t---l---t---f---3--- **********- ** *** *** *** ; ** *** :: ** *** ** *** * ** i ** *** : ** ** I ** ** + * *** ** ** * ** : ** ** * ** I ** ** ** ** II * ** I * ** ** ** + z * ** ** * I * ** ** 3; * ** I ** **I * *1 * ** z I EE 1:4674 SE; 1:0426 ;w PI:6179 ix% 0:1931 0.0515 -0.0900 z;- ;+g -0:5148 LEFT BIN ECGE ? + * I ** 7 1* I ,I,: i 1* 1* : 1* I z I' +3 + + + 3: -I- --- +- --- 1-1-- -l- ---e--+---e ---- --- 000000000000000000000000000000000000000~00000000000000000000 . . . . 222221111100000000001111122222333334444455555666667777788888 975319753197531024680246802468024680246802468024680246802468 444444333333333666677777777777777788888888888888899999999999 332110998776544788900122344566788900122344566789901123345567 figure 1.4
13
MULTIDIMENSIONAL ADDITIVE SPLINE APPROXIMATION (4/19/80) ;$V&METERS FOR2;;IS RUN
NPRED MODE MAXTRY MAXPRO PPCONV MAXIT KORDER EEE? IPRINT 6 2
74
.150000 4 93 2-002000KR%
1$VERAGE SQ"ARi? RESIDUAL AROUND THE MEAN 26.7495 .
14
?
KNOTS ALONG DIRECTION NR 1 PLOT STATI5TICS PROJ ON X AXIS 0.4576 FEE 0:4188 = 0. 0. 0. / 0. 11. 0. / 0. 0. 0. PRCU Ah, v "LY I 3000000~0000000000001000000000010010000010000000000100000003 AXIS + ---I-- -t- ---+---+----+--- I***** i ** *** i I ** T ** T 0.3412
+
---I-- ---+---+----+---
-t-
I*****
i **
***
i
:
**
**
:
:
**
**
z
:
****
****
z
:
**
+
****
:
****
i
**
i
****
:
****
I
**
****
z
**
:
I ** ** I * * + ** ** * * z ** ** I * * z I ** ** ** ** s :: * * ** ** I I ** ** I + *** *** **** **** + I **** ** **** ** ** ** : I * * ********* *1 II ** ** ** ********I * * *** II
**
i
:**
I I : : i I I t 0.0564 E'%z 0:0176 0.0047 -0.0083 :;* ;g -0:0471 -0.0600z
+
i
:
z
:
I + z z s I I + : ********* *1 ** ********I *** I T ** ** f****
1
T . . . . . . . . 111112222233333344444555556666677777788888999990000011111122 134680246802357913579134680246802457913579134680246802457913 009876543210987665432109876543211098765432109877654321098765 900122344566788900122344556778990112334556778990112334556778 figure 2.115 PLOT STATISTICS = PRoJ 0. 0. 0. / 0. 11. 0. / 0. 0. 0. PROJ ON X AXIS 30000000000000010000000~010000000100000001000000000010000003 &: 0.4584 $---t---+---3---+---+---~-
KNOTS ALONG DIRECTION NR 2
0.4454 I ****** C%E : ** ** 0:4066 I ** ** * * 0.3936 I ** ** z*z% : ** * * 0:3548 I ** ** * 0.3418 I * ** ** * * * ** ** * * ** * * * * ** * * ** * * ** * ** * ** ** **** pm; f *,* **** PI:0568 I * *** ** ** ***i ** ** I ** *** ** *** I *** *** I ***** z + + + + + 3: -I-- --- -t- --- ---+---+---+--- LEFT --- BIN 000000000000000000000000000000000000000000000000000000000000 ECGE . . . ..*...*.... 000000000001111122222233333344444555555666666777777888889999 75420124689135680235790246791346801357802457912468913568~235 792470752075308530853085308530853085308530853186318631863186 272724949495050505161616272727283838384949495050505161616272 ? figure 2.2
16
SPL01bJ;g;UNC;I~;5ND KNOTS ALONG DIRECTION NR 3
. . PLOT&TATpjTICS = 0. 0. 0. / 0. 11. 0. / 0. 0. 0. --a- ON X AXIS 0.2831 EE 0: 2509 PKUJ ON Y 300000000001000000001000000001000000000000010001000000000003 AXIS ? 0.1223 LEFT BIN ECGE +- --- --- +- ---+- --- t _--- ---- : ****** ** ** : I ** ** i * * ** ** ; 3; * * 3E ** ** II * * ** * i t * ** + ** II ; * * : I ** ** : * * ; ** ; .z * * : g : ** * z + ** ** ****** + I 1* * ***** *** I" ** ***** *** 1** ** **** ** : 1* ****** ** I* * i 1* . ** 1* ** I z * i ** I + ** + II * ** I I : *1 *1 I **1 I *1 *1 : i +3 + + + + + 3+ --- -e-D-- ---+---+---+-we- 000000000~0000000~00000000000000000000000000000000000000 . . . ..*... 000000111111222222333333444444555555666666777777888888999999 134679124679124679124679124679124679124679124679124679124679 306396396296295295285285185185184184174174074073073073063063 306284062840639517395173952840628406284173951739517306284062 figure 2.3
17
MULTIDIMENSiONAL ADDITIVE SPLINE APPROXIMATION (4/19/80) PARAMETERS FORIT;IS RUN
NOBS gg? a liEsoy : 12 PFCONV MAXIT
l 15?000
KORDER !EE? 3 IPRINT 1.~~000 kzE&i 13GVERAGE SQ"ARi! RESIDUAL AROUND THE MEAN .109703 .
5 ;;: w c
W + + + ; + + + W W $ + + + + + QJ tHHHHHHHHH+HHHHHHHHH+HHHHHHHHH+HHHHHHHHH
19
MULTIDIMENSIONAL ADDITIVE SPLINE APPROXIMATION (J/19/80) NOBS. PARAMETERS - FORIT;IS RUN
z PFCONV f MAXIT KORDER l IsYaO 3 .*2000 !ERS 13
eVERAGE SQUAti: RESIDUAL AROUND THE MEAN
20
S';I;;2;UNd;I;N AND ~~22AL~~G2~~~EC~~O~3~~ -;:;242 : -E039 PI:0556 -A 5495 -0:0384 -0:0141 0:0 fE351 . . -0.0660 -0.0153 -0.0235 PLOT&TA&TICS = 0. 0. 0. / 0. 11. 0. / 0. 0. 0. ON X PROJ AXIS 400000000000000000010000000000000001000001000000000000000004 %I: 29.8919 t---t---t---1---(----1-1+---1--- 29.6401 I* ***** 29.3884 29.1366 28.8848 28.6331 1* *** I* ** 1** ** 1* ** I ** * ** *** I *;**;* I + I ** ** * ** * ** * * ** * I I 27.1225 26.8707 f ** * * ** * * ** * * ** * * ** * * ** * ** * * ** I I I I ; ; z z + + : : : : : : : : I I + + jE jE i i z z i i *+ *+ ***I ***I ,***** ,***** *** *** 1 1 ***** ***** : : ****
;m; II
*
*** 21:0802 I**
** ** ** %E: E I ** ***T
20:3249 I **t I 20.0731 I 19.8214 +4 -_-- --- + ---+--- + + 4+ I --- --- LEFT ---I--- BIN EIXE 111111111111111111111111110000000000000000000000000000000000 . . . ...*... 111110000000000000000000009999999999999999999998888888888888 211009988777665544332211009988777665544332211009988877665544 272838394940505161627273838494950506161727283839495050616172 257025803580358036813681368146914691469247924792570257025703 ? figure 4.121
MULTIDIMENSiONAL ADDITIVE SPLINE APPROXIIMATION (4/19/80) PARAMETERS FOR2;;IS RUN
NO%S gg? -cI 19 MAXTRY MAXPRO z PPCONV MAXIT .15:000 KORDER i 1.+;000 EiEE 13
$VERAGE SQ"ARh: RESIDUAL AROUND THE MEAN 13.2975 .
