ECE 3040 Lecture 18: Curve Fitting by Least-Squares-Error Regression Prof. Mohamad Hassoun

ECE 3040

Lecture 18: Curve Fitting by Least-Squares-Error Regression

This lecture covers the following topics:



Introduction



Linear least-squares-Error (LSE) regression: The straight-line model



Linearization of nonlinear models



General linear LSE regression and the polynomial model



Polynomial regression with Matlab:

polyfit



Non-linear LSE regression



Numerical solution of the non-linear LSE optimization problem:

Gradient search and Matlab’s

fminsearch

function



Solution of differential equations based on LSE minimization



Appendix: Explicit matrix formulation for the quadratic regression

problem

Introduction

In the previous lecture, polynomial and cubic spline interpolation methods were introduced for estimating a value between a given set of precise data points. The idea was to (interpolate) “fit” a function to the data points so as to perfectly pass through all data points. Many engineering and scientific observations are made by conducting experiments in which physical quantities are measured and recorded as inexact (noisy) data points. In this case, the objective would be to find the best-fit analytic curve (model) that approximates the underlying functional relationship present in the data set. Here, the best-fit curve is not required to pass through the data points, but it is required to capture the shape (general trend) of the data. This curve fitting problem is referred to as regression. The following sections present formulations for the regression problem and provide solutions.

The following figure compares two polynomials that attempt to fit the shown data points. The blue curve is the solution to the interpolation problem. The green curve is the solution (we seek) to the linear regression problem.

Linear Least-Squares-Error (LSE) Regression:

The Straight-Line Model

The regression problem will first be illustrated for fitting the linear model (straight-line), 𝑦(𝑥) = 𝑎₁𝑥 + 𝑎₀, to a set of 𝑛 paired experimental observations:

(𝑥₁, 𝑦₁), (𝑥₂, 𝑦₂), … , (𝑥_𝑛, 𝑦_𝑛). So, the idea here is to position the straight-line (i.e., to determine the regression coefficients 𝑎₀ and 𝑎₁) so that some error measure of fit is minimized. A common error measure is the sum-of-the-squares (SSE) of the

residual errors 𝑒_𝑖 = 𝑦_𝑖 − 𝑦(𝑥_𝑖), 𝐸(𝑎₀, 𝑎₁) = ∑ 𝑒_𝑖2 𝑛 𝑖=1 = ∑[𝑦_𝑖 − 𝑦(𝑥_𝑖)]2 𝑛 𝑖=1 = ∑[𝑦_𝑖 − (𝑎₁𝑥_𝑖 + 𝑎₀)]2 𝑛 𝑖=1

The residual error 𝑒_𝑖 is the discrepancy between the measured value, 𝑦_𝑖, and the approximate value 𝑦(𝑥_𝑖) = 𝑎₀ + 𝑎₁𝑥_𝑖, predicted by the straight-line regression model. The residual error for the 𝑖th data point is depicted in the following figure.

A solution can be obtained for the regression coefficients, {𝑎₀, 𝑎₁}, that minimizes 𝐸(𝑎₀, 𝑎₁). This criterion, 𝐸, which is called least-squares-error (LSE) criterion, has a number of advantages, including that it yields a unique line for a given data set. Differentiating 𝐸(𝑎₀, 𝑎₁) with respect to each of the unknown regression model coefficients, and setting the result to zero lead to a system of two linear equations,

𝜕 𝜕𝑎₀𝐸(𝑎0, 𝑎1) = 2 ∑(𝑦𝑖 − 𝑎1𝑥𝑖 − 𝑎0)(−1) 𝑛 𝑖=1 = 0 𝜕 𝜕𝑎₁𝐸(𝑎0, 𝑎1) = 2 ∑(𝑦𝑖 − 𝑎1𝑥𝑖 − 𝑎0)(−𝑥𝑖) 𝑛 𝑖=1 = 0

After expanding the sums, we obtain

− ∑ 𝑦_𝑖 + 𝑛 𝑖=1 ∑ 𝑎₀ 𝑛 𝑖=1 + ∑ 𝑎₁𝑥_𝑖 𝑛 𝑖=1 = 0 − ∑ 𝑥_𝑖𝑦_𝑖 𝑛 𝑖=1 + ∑ 𝑎₀𝑥_𝑖 𝑛 𝑖=1 + ∑ 𝑎₁𝑥_𝑖2 𝑛 𝑖=1 = 0

Now, realizing that ∑𝑛_𝑖=1𝑎₀ = 𝑛𝑎₀, and that multiplicative quantities that do not depend on the summation index 𝑖 can be brought outside the summation (i.e., ∑𝑛_𝑖=1𝑎𝑥_𝑖 = 𝑎 ∑𝑛_𝑖=1𝑥_𝑖), we may rewrite the above equations as

These are called the normal equations. We can solve for 𝑎₁ using Cramer’s rule and for 𝑎₀ by substitution (Your turn: Perform the algebra) to arrive at the following LSE solution: 𝑎₁∗ = 𝑛 ∑ 𝑥𝑖𝑦𝑖 − ∑ 𝑥𝑖∑ 𝑦𝑖 𝑛 𝑖=1 𝑛 𝑖=1 𝑛 𝑖=1 𝑛 ∑𝑛_𝑖=1𝑥_𝑖2 − (∑𝑛_𝑖=1𝑥_𝑖)2 𝑎₀∗ = ∑ 𝑦𝑖 𝑛 𝑖=1 𝑛 − 𝑎1 ∗ ∑ 𝑥𝑖 𝑛 𝑖=1 𝑛

The value 𝐸(𝑎₀∗, 𝑎₁∗) represents the LSE value and will be referred to as 𝐸_𝐿𝑆𝐸 and expressed as

𝐸_𝐿𝑆𝐸 = ∑(𝑦_𝑖 − 𝑎₁∗𝑥_𝑖 − 𝑎₀∗)2 𝑛

𝑖=1

Any other straight-line will lead to an error 𝐸(𝑎₀, 𝑎₁) > 𝐸_𝐿𝑆𝐸.

Let the value of the sum-of-the-square of the difference between the 𝑦_𝑖 values and their average value, 𝑦̅ = ∑ 𝑦𝑖

𝑛 𝑖=1 𝑛 , be 𝐸_𝑀 = ∑(𝑦_𝑖 − 𝑦̅)2 𝑛 𝑖=1

Then, the (positive) difference 𝐸_𝑀 − 𝐸_𝐿𝑆𝐸 represents the improvement (where the smaller 𝐸_𝐿𝑆𝐸 is, the better) due to describing the data in terms of a straight-line, rather than as an average value (a straight-line with zero slope and 𝑦-intercept equals to 𝑦̅). The coefficient of determination, 𝑟2, is defined as the relative error between 𝐸_𝑀 and 𝐸_𝐿𝑆𝐸,

𝑟2 = 𝐸𝑀 − 𝐸𝐿𝑆𝐸

𝐸_𝑀 = 1 − 𝐸_𝐿𝑆𝐸

𝐸_𝑀

For perfect fit, where the regression line goes through all data points, 𝐸_𝐿𝑆𝐸 = 0 and 𝑟2 = 1, signifying that the line explains 100% of the variability in the data. On the other hand for 𝐸_𝑀 = 𝐸_𝐿𝑆𝐸 we obtain 𝑟2 = 0, and the fit represents no improvement over a simple average. A value of 𝑟2 between 0 and 1 represents the extent of

improvement. So, 𝑟2 = 0.8 indicates that 80% of the original uncertainty has been explained by the linear model. Using the above expressions for 𝐸_𝐿𝑆𝐸, 𝐸_𝑀, 𝑎₀∗ and 𝑎₁∗ one may derive the following formula for the correlation coefficient, 𝑟, (your turn: Perform the algebra)

𝑟 = √𝐸𝑀 − 𝐸𝐿𝑆𝐸

𝐸_𝑀 =

𝑛 ∑(𝑥_𝑖𝑦_𝑖) − (∑ 𝑥𝑖)(∑ 𝑦𝑖)

√𝑛 ∑ 𝑥_𝑖2 − (∑ 𝑥_𝑖)2 _{√𝑛 ∑ 𝑦}

𝑖2 − (∑ 𝑦𝑖)2

Example. Fit a straight-line to the data provided in the following table. Find 𝑟2.

x 1 2 3 4 5 6 7

y 2.5 7 38 55 61 122 110

Solution. The following Matlab script computes the linear regression coefficients, 𝑎₀∗ and 𝑎₁∗, for a straight-line employing the LSE solution.

x=[1 2 3 4 5 6 7];

y=[2.5 7 38 55 61 122 110]; n=length(x);

a1=(n*sum(x.*y)-sum(x)*sum(y))/(n*sum(x.^2)-(sum(x)).^2) a0=sum(y)/n-a1*sum(x)/n

The solution is 𝑎₁∗ = 20.5536 and 𝑎₀∗ = −25.7143. The following plot displays the data and the regression model, 𝑦(𝑥) = 20.5536𝑥 − 25.7143.

The following script computes the correlation coefficient, 𝑟. x=[1 2 3 4 5 6 7];

y=[2.5 7 38 55 61 122 110]; n=length(x);

r=(n*sum(x.*y)-sum(x)*sum(y))/((sqrt(n*sum(x.^2)- ... (sum(x))^2))*(sqrt(n*sum(y.^2)-(sum(y))^2)))

The script returns 𝑟 = 0.9582 (so, 𝑟2 = 0.9181). These results indicate that about 92% of the variability in the data has been explained by the linear model.

A word of caution: Although the coefficient of determination provides a convenient measure of the quality of fit, you should be careful not to rely on it completely. For, it is possible to construct data sets that will similar 𝑟2 values, while the regression line is not well positioned for some sets. A good practice would be to visually inspect the plot of the data along with the regression curve. The following example illustrates these ideas.

Example. Anscombe's quartet comprises four datasets that have 𝑟2 ≅ 0.666, yet appear very different when graphed. Each dataset consists of eleven (𝑥_𝑖, 𝑦_𝑖) points. They were constructed in 1973 by the statistician Francis Anscombe to demonstrate both the importance of graphing data before analyzing it and the effect of outliers on statistical properties. Notice that if we are to ignore the outlier point in the third data set, then the regression line would be perfect, with 𝑟2 = 1.

Your turn: Employ linear regression to generate the above plots and determine 𝑟2 for each of the Anscombe’s data sets.

Linearization of Nonlinear Models

The straight-line regression model is not always suitable for curve fitting. The choice of regression model is often guided by the plot of the available data, or can be guided by the knowledge of the physical behavior of the system that generated the data. In general, polynomial or other nonlinear models are more suitable. A nonlinear regression technique (introduced later) is available to fit complicated

nonlinear equations to data. However, some basic nonlinear functions can be readily transformed into linear functions in their regression coefficients (we will refer to such functions as transformable or linearizable). Here, we can take advantage of the LSE regression formulas, which we have just derived, to fit the transformed equations to the data.

One example of a linearizable linear model is the exponential model, 𝑦(𝑥) = 𝛼𝑒𝛽𝑥, where 𝛼 and 𝛽 are constants. This equation is very common in engineering (e.g., capacitor transient voltage) and science (e.g., population growth or radioactive decay). We can linearize this equation by simply taking its natural logarithm to yield: ln(𝑦) = ln(𝛼) + 𝛽𝑥. Thus, if we transform the 𝑦_𝑖 values in our data, by taking their natural logarithms, and define 𝑎₀ = ln(𝛼) and 𝑎₁ = 𝛽 we arrive at the equation of a straight-line (of the form 𝑌 = 𝑎₀ + 𝑎₁𝑥). Then, we can readily use the formulas for the LSE solution, (𝑎₀∗, 𝑎₁∗), derived earlier. The final step would be to set 𝛼 = 𝑒𝑎0∗ and 𝛽 = 𝑎

1∗ and arrive at the regression solution,

𝑦(𝑥) = 𝛼𝑒𝛽𝑥 = (𝑒𝑎0∗)𝑒𝑎1∗𝑥

A second common linearizable nonlinear regression model is the power model, 𝑦(𝑥) = 𝛼𝑥𝛽. We can linearize this equation by simply taking its natural logarithm to yield: ln(𝑦) = ln(𝛼) + 𝛽ln (𝑥) (which is a linear model of the form 𝑌 = 𝑎₀ + 𝑎₁𝑋). In this case, we need to first transform the 𝑦_𝑖 and 𝑥_𝑖 values into ln(𝑦_𝑖) and ln(𝑥_𝑖), respectively, and then apply the LSE solution to the transformed data.

Other useful linearizable models include: 𝑦(𝑥) = 𝛼 ln(𝑥) + 𝛽 (logarithmic function), 𝑦(𝑥) = 1

𝛼𝑥+𝛽 (reciprocal function), and 𝑦(𝑥) = 𝛼𝑥

𝛽+𝑥

linearizable and provide their corresponding linearized form and their change of variable formulas, respectively.

Example. Fit the exponential model and the power model to the data in the following table. Compare the fit quality to that of the straight-line model.

𝑥 1 2 3 4 5 6 7 8

𝑦 2.5 7 38 55 61 122 83 143

Solution. Matlab script (linear.m) for the linear model, 𝑦 = 𝑎₀ + 𝑎₁𝑥: x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; n=length(x); a1=(n*sum(x.*y)-sum(x)*sum(y))/(n*sum(x.^2)-(sum(x)).^2); a0=sum(y)/n-a1*sum(x)/n; r=(n*sum(x.*y)-sum(x)*sum(y))/((sqrt(n*sum(x.^2)-(sum(x))^2))*(... sqrt(n*sum(y.^2)-(sum(y))^2))); a1, a0, r^2

Result 1. Linear model solution: 𝑦 = 19.3036𝑥 − 22.9286, 𝑟2 = 0.8811. Matlab script (exponential.m) for the exponential model, 𝑦 = 𝛼𝑒𝛽𝑥:

x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; ye=log(y); n=length(x); a1=(n*sum(x.*ye)-sum(x)*sum(ye))/(n*sum(x.^2)-(sum(x)).^2); a0=sum(ye)/n-a1*sum(x)/n; r=(n*sum(x.*ye)-sum(x)*sum(ye))/((sqrt(n*sum(x.^2)-(sum(x))^2))... *(sqrt(n*sum(ye.^2)-(sum(ye))^2))); alpha=exp(a0), beta=a1, r^2

Result 2. Exponential model solution: 𝑦 = 3.4130𝑒0.5273𝑥, 𝑟2 = 0.8141. Matlab script (power_eq.m) for the power model, 𝑦 = 𝛼𝑥𝛽:

x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; xe=log(x); ye=log(y); n=length(x); a1=(n*sum(xe.*ye)-sum(xe)*sum(ye))/(n*sum(xe.^2)-(sum(xe)).^2); a0=sum(ye)/n-a1*sum(xe)/n; r=(n*sum(xe.*ye)-sum(xe)*sum(ye))/((sqrt(n*sum(xe.^2)-

…

Result 3. Power model solution: 𝑦 = 2.6493𝑥1.9812, 𝑟2 = 0.9477

From the results for 𝑟2, the power model has the best fit. The following graph compares the three models. By visually inspecting the plot we see that, indeed, the power model (red; 𝑟2 = 0.9477) is a better fit compared to the linear model (blue; 𝑟2 = 0.8811) and to the exponential model (green; 𝑟2 = 0.8141). Also, note that the straight-line fits the data better than the exponential model.

Your turn: Repeat the above regression problem employing: (a) The logarithmic function, 𝑦 = 𝛼 ln(𝑥) + 𝛽; (b) The reciprocal function, 𝑦 = 1

𝛼𝑥+𝛽 ; (c) The

saturation-growth-rate function, 𝑦(𝑥) = 𝛼𝑥

General Linear LSE Regression and the Polynomial Model

For some data sets, the underlying model cannot be captured accurately with a straight-line, exponential, logarithmic or power models. A model with a higher degree of nonlinearity (i.e., with added flexibility) is required. There are a number of higher order functions that can be used as regression models. One important regression model would be a polynomial. A general LSE formulation is presented next. It extends the earlier linear regression analysis to a wider class of nonlinear functions, including polynomials. (Note: when we say linear regression, we are referring to a model that is linear in its regression parameters, 𝑎_𝑖, not 𝑥.)

Consider the general function in z,

𝑦 = 𝑎_𝑚𝑧_𝑚 + 𝑎_𝑚−1𝑧_𝑚−1 + ⋯ + 𝑎₁𝑧₁ + 𝑎₀ (1) where the 𝑧_𝑖 represents a basis function in 𝑥. It can be easily shown that if the basis functions are chosen as 𝑧_𝑖 = 𝑥𝑖, then the above model is that of an 𝑚-degree

polynomial,

𝑦 = 𝑎_𝑚𝑥𝑚 + 𝑎_𝑚−1𝑥𝑚−1 + ⋯ + 𝑎₁𝑥 + 𝑎₀

There are many classes of functions that can be described by the above general function in Eqn. (1). Examples include:

𝑦 = 𝑎₀ + 𝑎₁𝑥, 𝑦 = 𝑎₀ + 𝑎₁cos(𝑥) + 𝑎₂sin (2𝑥) and 𝑦 = 𝑎₀ + 𝑎₁𝑥 + 𝑎₂𝑒−𝑥2

One example of a function that can’t be represented by the above general function is the radial-basis-function (RBF)

𝑦 = 𝑎₀ + 𝑎₁𝑒𝑎2(𝑥−𝑎3)2

In other words, this later function is not transformable into a linear regression model, as was the case (say) for the exponential function, 𝑦 = 𝛼𝑒𝛽𝑥. The

regression with such non-transformable functions is known as nonlinear regression and is considered later in this lecture.

In the following formulation of the LSE regression problem we restrict the model of the regression function to the polynomial

𝑦 = 𝑎_𝑚𝑥𝑚 + 𝑎_𝑚−1𝑥𝑚−1 + ⋯ + 𝑎₁𝑥 + 𝑎₀

Given 𝑛 data points {(𝑥₁, 𝑦₁), (𝑥₂, 𝑦₂), … , (𝑥_𝑛, 𝑦_𝑛)} we want to determine the

regression coefficients by solving the system of 𝑛 equations with 𝑚 + 1 unknowns: 𝑎_𝑚𝑥₁𝑚 + 𝑎_𝑚−1𝑥₁𝑚−1 + ⋯ + 𝑎₁𝑥₁ + 𝑎₀ = 𝑦₁

𝑎_𝑚𝑥₂𝑚 + 𝑎_𝑚−1𝑥₂𝑚−1 + ⋯ + 𝑎₁𝑥₂ + 𝑎₀ = 𝑦₂ .

𝑎_𝑚𝑥_𝑛𝑚 + 𝑎_𝑚−1𝑥_𝑛𝑚−1 + ⋯ + 𝑎₁𝑥_𝑛 + 𝑎₀ = 𝑦_𝑛

The above system can be written in matrix form as Za = y,

[ 𝑥₁𝑚 𝑥₁𝑚−1 ⋯ 𝑥₁ 1 𝑥₂𝑚 𝑥₂𝑚−1 ⋯ 𝑥₂ 1 ⋮ ⋮ ⋱ ⋮ ⋮ 𝑥_𝑛−1𝑚 𝑥_𝑛−1𝑚−1 ⋯ 𝑥_𝑛−1 1 𝑥_𝑛𝑚 𝑥_𝑛𝑚−1 ⋯ 𝑥_𝑛 1][ 𝑎_𝑚 𝑎_𝑚−1 ⋮ 𝑎₁ 𝑎₀ ] = [ 𝑦₁ 𝑦₂ ⋮ 𝑦_𝑛−1 𝑦_𝑛 ]

where Z is an 𝑛x(𝑚 + 1) rectangular matrix formulated from the 𝑥_𝑖 data values with 𝑧_𝑖𝑗 = 𝑥_𝑖𝑚−𝑗+1 (𝑖 = 1, 2, … , 𝑛; 𝑗 = 1, 2, … , 𝑚 + 1), a is an (𝑚 + 1) column vector of unknown regression coefficients and y is an 𝑛 column vector of 𝑦_𝑖 data values. For regression problems, the above system is over-determined (𝑛 > 𝑚 + 1) and, therefore, there is generally no solution ‘a’ that satisfies Za = y. So, we seek the LSE solution, a*; the solution which minimizes the sum-of-squared-error (SSE) criterion

E(a) = ||y - Za||2 = ∑𝑛𝑖=1(𝑦𝑖 − ∑𝑚+1𝑗=1 𝑧𝑖𝑗 𝑎𝑚−𝑗+1) 2

where ||.|| denotes the vector norm. As we did earlier in deriving the straight-line regression coefficients 𝑎₀ and 𝑎₁, we set all partial derivatives 𝜕

𝜕𝑎_𝑖 𝐸(a) to zero and

solve the resulting system of (𝑚 + 1) equations:

𝜕 𝜕𝑎₀𝐸(𝑎0, 𝑎1, … , 𝑎𝑚) = −2 ∑ (𝑦𝑖 − ∑ 𝑥𝑖 𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) 𝑥_𝑖0 𝑛 𝑖=1 = 0 𝜕 𝜕𝑎₁𝐸(𝑎0, 𝑎1, … , 𝑎𝑚) = −2 ∑ (𝑦𝑖 − ∑ 𝑥𝑖 𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) 𝑛 𝑖=1 𝑥_𝑖1 = 0 𝜕 𝜕𝑎₂𝐸(𝑎0, 𝑎1, … , 𝑎𝑚) = −2 ∑ (𝑦𝑖 − ∑ 𝑥𝑖 𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) 𝑛 𝑖=1 𝑥_𝑖2 = 0 . . 𝜕 𝜕𝑎_𝑚𝐸(𝑎0, 𝑎1, … , 𝑎𝑚) = −2 ∑ (𝑦𝑖 − ∑ 𝑥𝑖 𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) 𝑛 𝑖=1 𝑥_𝑖𝑚 = 0

This system can be rearranged as (note: 𝑥_𝑖0 = 1)

∑ (𝑦_𝑖 − ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 0 𝑛 𝑖=1 ∑ (𝑥_𝑖𝑦_𝑖 − 𝑥_𝑖 ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 0 𝑛 𝑖=1 ∑ (𝑥_𝑖2𝑦_𝑖 − 𝑥_𝑖2 ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 0 𝑛 𝑖=1 . . ∑ (𝑥_𝑖𝑚𝑦_𝑖 − 𝑥_𝑖𝑚 ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑘+1}) = 0 𝑛 𝑖=1

or, ∑ ( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑦_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = ∑ 𝑥_𝑖𝑦_𝑖 𝑛 𝑖=1 𝑛 𝑖=1 ∑ 𝑥_𝑖2( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑥_𝑖2𝑦_𝑖 𝑛 𝑖=1 . . ∑ 𝑥_𝑖𝑚( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑥_𝑖𝑚𝑦_𝑖 𝑛 𝑖=1

which by setting 𝑧_𝑖𝑗 = 𝑥_𝑖𝑚−𝑗+1 (𝑖 = 1, 2, … , 𝑛; 𝑗 = 1, 2, … , 𝑚 + 1) can then be expressed in matrix form as (Your turn: derive it)

ZT(Za) = ZTy or (ZTZ)a= ZTy

(Refer to the Appendix for an explicit representation of the above equation for the case of a quadratic regression polynomial.)

The matrix ZTZ is a (𝑚 + 1)x(𝑚 + 1) square matrix (recall that 𝑚 is the degree of

the polynomial model being used). Generally speaking, the inverse of ZTZ does exist for the above regression formulation. Multiplying both sides of the equation by (ZTZ)-1 leads to the LSE solution for the regression coefficient vector a,

Ia* = a* = [(ZTZ)-1 ZT]y

Where I is the identity matrix. Matlab offers two ways for solving the above system of linear equations: (1) using the left-division operator a = (Z′*Z)\(Z′*y), where ‘′’ is the Matlab transpose operator, or (2) using a = pinv(Z)*y, where pinv is the built-in

The coefficient of determination, 𝑟2, for the above polynomial regression formulation is given by (for 𝑛 ≫ 𝑚)

𝑟2 = 1 −∑ (𝑦𝑖 − 𝑦̂𝑖) 2 𝑛 𝑖=1 ∑𝑛 (𝑦_𝑖 − 𝑦̅)2 𝑖=1

where 𝑦̂_𝑖 is the 𝑖th component of the prediction vector Za*, and 𝑦̅ is the mean of the 𝑦_𝑖 values. Matlab can conveniently compute 𝑟2 as,

1-sum((y-Z*a).^2)/sum((y-mean(y)).^2)

Example. Employ the polynomial LSE regression formulation to solve for a cubic curve fit for the following data set. Also, compute 𝑟2.

𝑥 1 2 3 4 5 6 7 8

𝑦 2.5 7 38 55 61 122 83 145

Solution. A cubic function is a third-order polynomial (𝑚 = 3) with the four coefficients 𝑎₀, 𝑎₁, 𝑎₂ and 𝑎₃. The number of data points is 8, therefore,

𝑛 = 8 and the Z matrix is 𝑛x(𝑚 + 1) = 8x4. The matrix formulation (Za = y) for this linear regression problem is

[ 1 1 1 1 8 4 2 1 27 9 3 1 64 16 4 1 125 25 5 1 216 36 6 1 343 49 7 1 512 64 8 1] [ 𝑎₃ 𝑎₂ 𝑎₁ 𝑎₀ ] = [ 2.5 7 38 55 61 122 83 145]

Solving using the pinv function, a = pinv(Z)*y, (y must be a column vector)

Alternatively, we may use the left-division operator and obtain the same result:

Therefore, the cubic fit solution is 𝑦 = 0.029𝑥3 − 0.02𝑥2 + 17.6176𝑥 − 19.2857 whose plot is shown below. Note that the contribution of the cubic and quadratic terms is very small compared to the linear part of the solution, for 0 ≤ 𝑥 ≤ 8. That is why the plot of the cubic fit model is close to linear.

which indicates that the cubic regression model explains 88% of the variability in the data. This result has a similar quality to that of the straight-line regression model (computed in an earlier example).

The following is a snapshot of a session with the “Basic Fitting tool” (introduced in the previous lecture) applied to data in the above example. It computes and

compares a cubic fit to a 5th-degree and a 6th-degree polynomial fit.

Your turn. Employ the linear LSE regression formulation to fit the following data set employing the model: 𝑦(𝑥) = 𝑎₀ + 𝑎₁cos(𝑥) + 𝑎₂sin (2𝑥). Also, determine 𝑟2 and plot the data along with 𝑦(𝑥). Hint: First determine the 10x3 matrix Z required for the Za = y formulation.

𝑥 1 2 3 4 5 6 7 8 9 10

Polynomial Regression with Matlab:

Polyfit

The Matlab polyfit function was introduced in the previous lecture for solving polynomial interpolation problems (𝑚 + 1 = 𝑛, same number of equations as unknowns). This function can also be used for solving 𝑚-degree polynomial regression given 𝑛 data points (𝑚 + 1 < 𝑛, more equations than unknowns). The syntax of polyfit call is p=polyfit(x,y,m), where x and y are the vectors of the

independent and dependent variables, respectively, and m is the order of the regression polynomial. The function returns a row vector, p, that contains the polynomial coefficients.

Example. Here is a solution to the straight-line regression problem (first example encountered in this lecture):

Example. Use polyfit to solve for the cubic regression model encountered in the example from the previous section.

Solution:

Note that this solution is identical to the one obtained using the pseudo-inverse-based solution.

Non-Linear LSE Regression

In some engineering applications, nonlinear models are required to be used to fit a given data set. The above general linear regression formulation can handle such regression problems as long as the nonlinear model is transformable into an

equivalent linear function in the unknown coefficients. However, in some cases, the models are not transformable. In this case, we have to come up with an appropriate set of equations whose solution leads to the LSE solution.

As an example, consider the nonlinear model 𝑦(𝑥) = 𝑎₀(1 − 𝑒𝑎1𝑥). This equation

can’t be manipulated into a linear regression formulation in the 𝑎₀ and 𝑎₁

coefficients. The LSE formulation (for this model with 𝑛 data points) takes the form

𝐸(𝑎₀, 𝑎₁) = ∑(𝑦_𝑖 − 𝑦(𝑥_𝑖))2 𝑛 𝑖=1 = ∑(𝑦_𝑖 − 𝑎₀(1 − 𝑒𝑎1𝑥𝑖))2 𝑛 𝑖=1 𝜕 𝜕𝑎₀𝐸(𝑎0, 𝑎1) = 2 ∑(𝑦𝑖 − 𝑎0(1 − 𝑒 𝑎₁𝑥_𝑖)) 𝑛 𝑖=1 (−1 + 𝑒𝑎1𝑥𝑖_{) = 0} 𝜕 𝜕𝑎₁𝐸(𝑎0, 𝑎1) = 2 ∑(𝑦𝑖 − 𝑎0(1 − 𝑒 𝑎₁𝑥_𝑖)) 𝑛 𝑖=1 (𝑎₀𝑥_𝑖𝑒𝑎1𝑥𝑖_{) = 0}

This set of two nonlinear equations need to be solved for the two coefficients, 𝑎₀ and 𝑎₁. Numerical algorithms such as Newton’s iterative method for solving a set of two nonlinear equations, or Matlab’s built-in fsolve and solve functions can be used to solve this system of equations, as shown in the next example.

Example. Employ the regression function 𝑦(𝑥) = 𝑎₀(1 − 𝑒𝑎1𝑥_{) to fit the following}

data.

x −2 0 2 4

Here, we have 𝑛 = 4, and the system of nonlinear equations to be solved is given by ∑(𝑦_𝑖 − 𝑎₀(1 − 𝑒𝑎1𝑥𝑖)) 4 𝑖=1 (−1 + 𝑒𝑎1𝑥𝑖) = 0 ∑(𝑦_𝑖 − 𝑎₀(1 − 𝑒𝑎₁𝑥_𝑖)) 4 𝑖=1 (𝑎₀𝑥_𝑖𝑒𝑎1𝑥𝑖) = 0

Substituting the data point values in the above equations leads to

(1 − 𝑎0(1 − 𝑒−2𝑎1))(−1 + 𝑒−2𝑎1) + (−4 − 𝑎0(1 − 𝑒2𝑎1))(−1 + 𝑒2𝑎1) + (−12 − 𝑎0(1 − 𝑒4𝑎1))(−1 + 𝑒4𝑎1) = 0

(1 − 𝑎0(1 − 𝑒−2𝑎1))(−2𝑎0𝑒−2𝑎1) + (−4 − 𝑎0(1 − 𝑒2𝑎1))(2𝑎0𝑒2𝑎1) + (−12 − 𝑎0(1 − 𝑒4𝑎1))(4𝑎0𝑒4𝑎1) = 0

After expansion and combining terms, we get 15 + (𝑒−2𝑎1 _{− 4𝑒}2𝑎1 _{− 12𝑒}4𝑎1) + 𝑎

0(3 + 𝑒−4𝑎1 − 2𝑒−2𝑎1 − 2𝑒2𝑎1 − 𝑒4𝑎1 + 𝑒8𝑎1) = 0

2𝑎₀[−𝑎₀𝑒−4𝑎1 _{+ (𝑎}

0− 1)𝑒−2𝑎1 − (𝑎0+ 4)𝑒2𝑎1 − (𝑎0 + 24)𝑒4𝑎1 + 2𝑎0𝑒8𝑎1] = 0

Matlab solution using function solve: syms a0 a1

f1=15+(exp(-2*a1)-4*exp(2*a1)-12*exp(4*a1))+a0*(3+exp(-4*a1)... -2*exp(-2*a1)-2*exp(2*a1)-exp(4*a1)+exp(8*a1));

f2=2*a0*(-a0*exp(-4*a1)+(a0-1)*exp(-2*a1)-(a0+4)*exp(2*a1)... -(a0+24)*exp(4*a1)+2*a0*exp(8*a1));

Matlab returns a set of four solutions to the above minimization problem. The first thing we notice is that for nonlinear regression, minimizing LSE may lead to multiple solutions (multiple minima). The solutions for this particular problem are:

1. 𝑎₀ = 𝑎₁ = 0, which leads to: 𝑦 = 0(1 − 𝑒0) = 0, or 𝑦 = 0 (the 𝑥-axis). 2. 𝑎0 = 0 and 𝑎1 ≅ −1.3610 + 1.5708𝑖, which leads to: 𝑦 = 0

3. 𝑎₀ = 0 and 𝑎₁ ≅ 0.1186 + 1.5708𝑖, which leads to: 𝑦 = 0.

4. 𝑎₀ ≅ 2.4979 and 𝑎₁ ≅ 0.4410, which leads to 𝑦(𝑥) = 2.4979(1 − 𝑒0.441𝑥).

The solutions 𝑦(𝑥) = 0 and 𝑦(𝑥) = 2.4979(1 − 𝑒0.441𝑥) are plotted below. It is obvious that the optimal solution (in the LSE sense) is 𝑦(𝑥) = 2.4979(1 − 𝑒0.441𝑥).

Your turn: Solve the above system of two nonlinear equations employing Matlab’s

fsolve.

Your turn: Fit the exponential model, 𝑦 = 𝛼𝑒𝛽𝑥, to the data in the following table employing nonlinear least squares regression. Then, linearize the model and

determine the model coefficients by employing linear least squares regression (i.e., use the formulas derived in the first section or polyfit). Plot the solutions.

𝑥 0 1 2 3 4

𝑦 1.5 2.5 3.5 5.0 7.5

Ans. Nonlinear least squares fit:

Numerical Solution of the Non-Linear LSE Optimization

Problem: Gradient Search and Matlab’s

fminsearch

Function

In the above example, we were lucky in the sense that the (symbolic-based) solve function returned the optimal solution for the optimization problem at hand. In more general non-linear LSE regression problems the models employed are complex and normally have more than two unknown coefficients. Here, solving (symbolically) for the partial derivatives of the error function becomes tedious and impractical.

Therefore, one would use numerically-based multi-variable optimization algorithms to minimize 𝐸(𝒂) = 𝐸(𝑎₀, 𝑎₁, 𝑎₂, … ), which are extensions of the ones considered in Lectures 13 and 14.

One method would be to extend the gradient-search minimization function

grad_optm2 to handle a function with two variables. Recall that this version of the function approximates the gradients numerically, therefore there is no need to

determine the analytical expressions for the derivatives. For the case of two variables 𝑎₀ and 𝑎₁, the gradient-descent equations are

𝑎₀(𝑘 + 1) = 𝑎₀(𝑘) + 𝑟 𝜕

𝜕𝑎₀𝐸(𝑎0, 𝑎1)

𝑎₁(𝑘 + 1) = 𝑎₁(𝑘) + 𝑟 𝜕

𝜕𝑎₁𝐸(𝑎0, 𝑎1)

where −1 < 𝑟 < 0. Upon using the simple backward finite-difference approximation for the derivatives, we obtain

𝑎₀(𝑘 + 1) = 𝑎₀(𝑘) + 𝑟𝐸[𝑎0(𝑘), 𝑎1(𝑘)] − 𝐸[𝑎0(𝑘 − 1), 𝑎1(𝑘)] 𝑎₀(𝑘) − 𝑎₀(𝑘 − 1)

𝑎₁(𝑘 + 1) = 𝑎₁(𝑘) + 𝑟𝐸[𝑎0(𝑘), 𝑎1(𝑘)] − 𝐸[𝑎0(𝑘), 𝑎1(𝑘 − 1)] 𝑎₁(𝑘) − 𝑎₁(𝑘 − 1)

The following is a Matlab implementation (function grad_optm2d) of these iterative formulas.

The above function [with 𝑎₀(0) = 2, 𝑎₁(0) = 0.5 and 𝑟 = −10−4_{] returns the}

same solution that was obtained above with solve [with, a minimum error value of 𝐸(2.4976, 0.4410) = 0.4364]:

Matlab has an important built-in function for numerical minimization of nonlinear multivariable functions. The function name is fminsearch. The (basic) function call syntax is [a,fa] = fminsearch(f,a0), where f is an anonymous function, a0 is a vector of initial values. The function returns a solution vector ‘a’ and the value of the

function at that solution, fa. Here is an application of fminsearch to solve the above non-linear regression problem [note how the unknown coefficients are represented as the elements of the vector, 𝑎₀ = a(1), 𝑎₁ = a(2), and are initialized at [0 0] for this problem].

A more proper way to select the initial search vector a = [𝑎0 𝑎1] for the above

optimization problem is to solve a set of 𝑘 nonlinear equations that is obtained from forcing the model to go through 𝑘 points (selected randomly from the data set). Here, 𝑘 is the number of unknown model parameters. For example, for the above problem, we solve the set of two nonlinear equations

𝑦_𝑖 − 𝑎₀(1 − 𝑒𝑎1𝑥𝑖) = 0

𝑦_𝑗 − 𝑎₀(1 − 𝑒𝑎1𝑥𝑗) = 0

where (𝑥_𝑖, 𝑦_𝑖) and (𝑥_𝑗, 𝑦_𝑗) are two distinct points selected randomly from the set of points being fitted. A numerical nonlinear equation solver can be used, say Matlab’s fsolve, as shown below [here, the end points (−2,1) and (4, −12) were selected].

Your turn: The height of a person at different ages is reported in the following table.

x (age) 0 5 8 12 16 18

y (in) 20 36.2 52 60 69.2 70

Determine the parameters 𝑎, 𝑏 and 𝑐 so that the following regression model is optimal in the LSE sense.

𝑦(𝑥) = 𝑎

1 + 𝑏𝑒−𝑐𝑥

Ans. 𝑦(𝑥) = 74.321

1+2.823𝑒−0.217𝑥

Your turn: Employ nonlinear LSE regression to fit the function

𝑦(𝑥) = 𝐾

√𝑥4 _{+ (𝑎}2 _{− 2𝑏)𝑥}2 _{+ 𝑏}2

to the data

𝑥 0 0.5 1 2 3

𝑦 0.95 1.139 0.94 0.298 0.087

Plot the data points and your solution for 𝑥 ∈ [0 6].

Ans. 𝑦(𝑥) = 0.888

As mentioned earlier, different initial conditions may lead to different local minima of the nonlinear function being minimized. For example, consider the function of two variables that exhibits multiple minima (refer to the plot):

𝑓(𝑥, 𝑦) = −0.02 sin(𝑥 + 4𝑦) − 0.2 cos(2𝑥 + 3𝑦) − 0.3 sin(2𝑥 − 𝑦) + 0.4cos (𝑥 − 2𝑦)

A contour plot can be generated as follows (the local minima are located at the center of the blue contour lines):

The following are the local minima discovered by function grad_optm2d for the indicated initial conditions:

The same local minima are discovered by fminsearch when starting from the same initial conditions:

Note that for this limited set of searches, the solution with the smallest SSE value is (𝑥∗_{, 𝑦}∗_{) = (0.0441, −1.7618), which represents a more optimal solution.}

Your turn (Email your solution to your instructor one day before Test 3). Fit the following data

𝑥 1 2 3 4 5 6 7 8 9 10

𝑦 1.17 0.93 −0.71 −1.31 2.01 3.42 1.53 1.02 −0.08 −1.51 employing the model

𝑦(𝑥) = 𝑎₀ + 𝑎₁cos(𝑥 + 𝑏₁) + 𝑎₂cos (2𝑥 + 𝑏₂)

This problem can be solved employing nonlinear regression (think solution via fminsearch), or it can be linearized which allows you to use linear regression (think solution via pseudo-inverse). Hint: cos(𝑥 + 𝑏) = cos(𝑥) cos(𝑏) − sin(𝑥) sin(𝑏). Plot the data points and 𝑦(𝑥) on the same graph.

In practice, a nonlinear regression model can have hundreds or thousands of coefficients. Examples of such models are neural networks and radial-basis-function models that often involve fitting multidimensional data sets, where each 𝑦 value depends on many variables,

𝑦(𝑥₁, 𝑥₂, 𝑥₃…). Numerical methods such as gradient-based optimization methods are often used to solve for the regression coefficients associated with those high-dimensional models. For a reference, check Chapters 5 and 6 in the following textbook:

Solution of Differential Equations Based on LSE Minimization

Consider the second-order time-varying coefficient differential equation

𝑦̈(𝑥) + 1

5𝑦̇(𝑥) + 9𝑥

2_{𝑦(𝑥) = 0 with 𝑦(0) = 1 and 𝑦̇(0) = 2}

defined over the interval 𝑥 ∈ [0 1].

We seek a polynomial 𝑦̃(𝑥) = 𝑎₀ + 𝑎₁𝑥 + 𝑎₂𝑥2 + 𝑎₃𝑥3 + 𝑎₄𝑥4 that approximates the solution, 𝑦(𝑥). In general, 𝑦̃(𝑥) does not have to be a polynomial. By applying the initial conditions to 𝑦̃(𝑥) we can solve for 𝑎₀ and 𝑎₁

𝑦̃(0) = 𝑎₀ = 𝑦(0) = 1 and 𝑑𝑦̃(0)

𝑑𝑥 = 𝑎1 = 𝑦̇(0) = 2

Now, we are left with the problem of estimating the remaining polynomial coefficients 𝑎₂, 𝑎₃, 𝑎₄ such that the residual

𝑓(𝑥, 𝑎₂, 𝑎₃, 𝑎₄) = 𝑑 2_𝑦̃(𝑥) 𝑑𝑥2 + 1 5 𝑑𝑦̃(𝑥) 𝑑𝑥 + 9𝑥 2_𝑦̃(𝑥)

is as close to zero as possible for all 𝑥 ∈ [0 1]. We will choose to minimize the integral of the squared residual,

𝐼(𝑎₂, 𝑎₃, 𝑎₄) = ∫ [𝑓(𝑥, 𝑎₂, 𝑎₃, 𝑎₄)]2𝑑𝑥

1 0

First, we compute the derivatives, 𝑑𝑦̃(𝑥)

𝑑𝑥 and 𝑑2𝑦̃(𝑥) 𝑑𝑥2 , 𝑑𝑦̃(𝑥) 𝑑𝑥 = 2 + 2𝑎2𝑥 + 3𝑎3𝑥 2 _{+ 4𝑎} 4𝑥3 𝑑2𝑦̃(𝑥) 𝑑𝑥2 = 2𝑎2 + 6𝑎3𝑥 + 12𝑎4𝑥 2

𝑓(𝑥, 𝑎₂, 𝑎₃, 𝑎₄) = 𝑑 2_𝑦̃(𝑥) 𝑑𝑥2 + 1 5 𝑑𝑦̃(𝑥) 𝑑𝑥 + 9𝑥 2_𝑦̃(𝑥) = 2𝑎₂ + 6𝑎₃𝑥 + 12𝑎₄𝑥2 +1 5(2 + 2𝑎2𝑥 + 3𝑎3𝑥 2 _{+ 4𝑎} 4𝑥3) + 9𝑥2(1 + 2𝑥 + 𝑎₂𝑥2 + 𝑎₃𝑥3 + 𝑎₄𝑥4) = 2 5 + 9𝑥 2 _{+ 18𝑥}3 _{+ 𝑎} 2(2 + 2 5𝑥 + 9𝑥 4_{) + 𝑎} 3(6𝑥 + 3 5𝑥 2 _{+ 9𝑥}5₎ + 𝑎₄(12𝑥2 +4 5𝑥 3 _{+ 9𝑥}6₎ or, 𝑓(𝑥, 𝑎₂, 𝑎₃, 𝑎₄) = 2 5 + 9𝑥 2 _{+ 18𝑥}3 _{+ 𝑎} 2(2 + 2 5𝑥 + 9𝑥 4_{) +} 𝑎₃(6𝑥 +3 5𝑥 2 _{+ 9𝑥}5_{) + 𝑎} 4(12𝑥2 + 4 5𝑥 3 _{+ 9𝑥}6₎

The following Matlab session shows the results of using fminsearch to solve for the coefficients 𝑎₂, 𝑎₃, 𝑎₄ that minimize the error function

𝐼(𝑎₂, 𝑎₃, 𝑎₄) = ∫ [𝑓(𝑥, 𝑎₂, 𝑎₃, 𝑎₄)]2𝑑𝑥

1 0

Note: the first component a(1) in the solution vector ’a’ is redundant; it is not used in function 𝐼.

Therefore, the optimal solution is

The following plot compares the “direct” numerical solution (red trace) to the minimum residual solution (blue trace). We will study the very important topic of numerical solution of differential equations in Lecture 22 (e.g., employing ode45).

Your turn: Consider the first-order, nonlinear, homogeneous differential equation with varying coefficient 𝑦̇(𝑥) + (2𝑥 − 1)𝑦2(𝑥) = 0, with 𝑦(0) = 1 and 𝑥 ∈ [0 1].

Employ the method of minimizing the squared residual to solve for the approximate solution

𝑦̃(𝑥) = 𝑎₀ + 𝑎₁𝑥 + 𝑎₂𝑥2+ 𝑎₃𝑥3 + 𝑎₄𝑥4 over the interval 𝑥 ∈ [0 1]. Plot 𝑦̃(𝑥) and the exact solution 𝑦(𝑥) given by

𝑦(𝑥) = 1

𝑥2_{− 𝑥 + 1} Ans. 𝑦̃(𝑥) = 1 + 0.964𝑥 + 0.487𝑥2− 2.903𝑥3+ 1.452𝑥4

Your turn: Determine the parabola 𝑦(𝑥) = 𝑎𝑥2 + 𝑏𝑥 + 𝑐 that approximates the cubic 𝑔(𝑥) = 2𝑥3− 𝑥2+ 𝑥 + 1 (over the interval 𝑥 ∈ [0 2]) in the LSE sense. In other words, determine the coefficients 𝑎, 𝑏 and 𝑐 such that the following error function is minimized,

𝐸(𝑎, 𝑏, 𝑐) = ∫ [𝑔(𝑥) − 𝑦(𝑥)]2𝑑𝑥

2 0

Solve the problem in two ways: (1) analytically; and (2) Employing fminsearch after evaluating the integral. Plot 𝑔(𝑥) and 𝑦(𝑥) on the same set of axis.

Appendix: Explicit Matrix Formulation for the Quadratic Regression Problem

Earlier in this lecture we have derived the 𝑚-degree polynomial LSE regression formulation as follows, ∑ ( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑦_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖 ( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = ∑ 𝑥_𝑖𝑦_𝑖 𝑛 𝑖=1 𝑛 𝑖=1 ∑ 𝑥_𝑖2( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑥_𝑖2𝑦_𝑖 𝑛 𝑖=1 . . ∑ 𝑥_𝑖𝑚( ∑ 𝑥_𝑖𝑚−𝑗+1 𝑚+1 𝑗=1 𝑎_{𝑚−𝑗+1}) = 𝑛 𝑖=1 ∑ 𝑥_𝑖𝑚𝑦_𝑖 𝑛 𝑖=1

Now, setting 𝑚 = 2 (designating a quadratic regression model) leads to three equations: ∑ (∑ 𝑥_𝑖3−𝑗 3 𝑗=1 𝑎_3−𝑗) = 𝑛 𝑖=1 ∑ 𝑦_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖(∑ 𝑥_𝑖3−𝑗 3 𝑗=1 𝑎_3−𝑗) = ∑ 𝑥_𝑖𝑦_𝑖 𝑛 𝑖=1 𝑛 𝑖=1 ∑ 𝑥_𝑖2(∑ 𝑥_𝑖3−𝑗 3 𝑗=1 𝑎_3−𝑗) = 𝑛 𝑖=1 ∑ 𝑥_𝑖2𝑦_𝑖 𝑛 𝑖=1

It can be shown that the above equations (Your Turn) can be cast in matrix form as, [ 𝑛 ∑ 𝑥_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖2 𝑛 𝑖=1 ∑ 𝑥_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖2 𝑛 𝑖=1 ∑ 𝑥_𝑖3 𝑛 𝑖=1 ∑ 𝑥_𝑖2 𝑛 𝑖=1 ∑ 𝑥_𝑖3 𝑛 𝑖=1 ∑ 𝑥_𝑖4 𝑛 𝑖=1 ] [ 𝑎₀ 𝑎₁ 𝑎₃] = [ ∑ 𝑦_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖𝑦_𝑖 𝑛 𝑖=1 ∑ 𝑥_𝑖2𝑦_𝑖 𝑛 𝑖=1 ]

This is a 3x3 linear system that can be solved using the methods of Lectures 15 & 16. With this type of formulation, care must be taken as to employ numerical solution methods that can handle ill-condition coefficient matrices; note the dominance of the (all positive) coefficients in the last row of the matrix.

The following two-part video (part1, part2) derives the above result (directly) from basic principles. Here is an example of quadratic regression: Part 1 Part 2.

Example. Employ the above formulation to fit a parabola to the following data.

𝑥 0 5 8 12 16 18

𝑦 20 36.2 52 60 69.2 70

The code that generated the above result is shown below.

Your turn: Verify the above solution employing Polyfit. Repeat employing the pseudo-inverse solution a = pinv(Z)*y applied to the formulation,

𝐙𝐚 = [ 𝑥₁𝑚 𝑥₁𝑚−1 ⋯ 𝑥₁ 1 𝑥₂𝑚 𝑥₂𝑚−1 ⋯ 𝑥₂ 1 ⋮ ⋮ ⋱ ⋮ ⋮ 𝑥_𝑛−1𝑚 𝑥_𝑛−1𝑚−1 ⋯ 𝑥_𝑛−1 1 𝑥_𝑛𝑚 𝑥_𝑛𝑚−1 ⋯ 𝑥_𝑛 1][ 𝑎_𝑚 𝑎_𝑚−1 ⋮ 𝑎₁ 𝑎₀ ] = [ 𝑦₁ 𝑦₂ ⋮ 𝑦_𝑛−1 𝑦_𝑛 ]

Your turn: Extend the explicit matrix formulation of this appendix to a cubic function 𝑓(𝑥) = 𝑎₀ + 𝑎₁𝑥 + 𝑎₂𝑥2 + 𝑎₃𝑥3 and use it to determine the polynomial coefficients for the data set from the last example. Compare (by plotting) your solution to the following solution that was obtained using nonlinear regression,

𝑦(𝑥) = 74.321

1+2.823𝑒−0.217𝑥.

Verify your solution employing Polyfit.