Gaussian Elimination

Certainly one of the most scientific computations is the solution of systems of simultaneous equations. The basic algorithm for solving systems of equations, Gaussian elimination, is relatively simple and has changed little in the 150 years since it was invented. This algorithm has come to be well understood, especially in the past twenty years, so that it can be used with some confidence that it will efficiently produce accurate results. This is an example of an algorithm that will surely be available in most computer installations; indeed, it is a primitive in several computer languages, notably APL and Basic. However, the basic algorithm is easy to understand and implement, and special situations do arise where it might be desirable to implement a modified version of the algorithm rather than work with a standard subroutine. Also, the method deserves to be learned as one of the most important numeric methods in use today.

As with the other mathematical material that we have studied so far, our treatment of the method will highlight only the basic principles and will be self-contained. Familiarity with linear algebra is not required to understand the basic method. We’ll develop a simple Pascal implementation that might be easier to use than a library subroutine for simple applications. However, we’ll also see examples of problems which could arise. Certainly for a large or important application, the use of an expertly tuned implementation is called for, as well as some familiarity with the underlying mathematics.

A Simple Example

Suppose that we have three variables and and the following three equations:

x + = 8,

5 8 CHAPTER 5

Our goal is to compute the values of the variables which simultaneously satisfy the equations. Depending on the particular equations there may not always be a solution to this problem (for example, if two of the equations are contradictory, such as + = 1, + = 2) or there may be many solutions (for example, if two equations are the same, or there are more variables than equations). We’ll assume that the number of equations and variables is the same, and we’ll look at an algorithm that will find a unique solution if one exists.

To make it easier to extend the formulas to cover more than just three points, we’ll begin by renaming the variables, using subscripts:

= -1.

To avoid writing down variables repeatedly, it is convenient to use matrix notation to express the simultaneous equations. The above equations are exactly equivalent to the matrix equation

There are several operations which can be performed on such equations which will not alter the solution:

Interchange equations: Clearly, the order in which the equations are written down doesn’t affect the solution. In the matrix representation, this operation corresponds to interchanging rows in the matrix (and the vector on the right hand side).

Rename variables: This corresponds to interchanging columns in the

matrix representation. (If columns and are switched, then variables and must also be considered switched.)

Multiply equations by a constant: Again, in the matrix representation,

this corresponds to multiplying a row in the matrix (and the cor- responding element in the vector on the side) by a constant.

Add two equations and replace one of them by the sum. (It takes a

little thought to convince oneself that this will not affect the solution.) For example, we can get a system of equations equivalent to the one above by replacing the second equation by the difference between the first two:

GAUSSIAN ELIMINATION 59

Notice that this eliminates from the second equation. In a similar manner, we can eliminate xi from the third equation by replacing the third equation by the sum of the first and third:

Now the variable is eliminated from all but the first equation. By sys- tematically proceeding in this way, we can transform the original system of equations into a system with the same solution that is much easier to solve. For the example, this requires only one more step which combines two of the operations above: replacing the third equation by the difference between the second and twice the third. This makes all of the elements below the main diagonal 0: systems of equations of this form are particularly easy to solve. The simultaneous equations which result in our example are:

= 6, = -8.

Now the third equation can be solved immediately: = 2. If we substitute this value into the second equation, we can compute the value of

4 6, 5.

Similarly, substituting these two values in the first equation allows the value of xi to be computed:

+ 15 = 8, = 1, which completes the solution of the equations.

This example illustrates the two basic phases of Gaussian elimination. The first is the forward elimination phase, where the original system is trans- formed, by systematically eliminating variables from equations, into a system with all zeros below the diagonal. This process is sometimes called triangula- tion. The second phase is the backward substitution phase, where the values of the variables are computed using the matrix produced by the first phase.

Outline of the Method

In general, we want to solve a system of equations in N unknowns: + + --- =

+ + + =

60 5

In matrix form, these equations are written as a single matrix equation:

or simply = b, where A represents the matrix, represents the variables, and b represents the sides of the equations. Since the rows of A are manipulated along with the elements of b, it is convenient to regard b as the (N column of A and use an N-by-(N + 1) array to hold both.

Now the forward elimination phase can be summarized as follows: first eliminate the first variable in all but the first equation by adding the ap- propriate multiple of the first equation to each of the other equations, then eliminate the second variable in all but the first two equations by adding the appropriate multiple of the second equation to each of the third through the Nth equations, then eliminate the third variable in all but the first three equations, etc. To eliminate the ith variable in the jth equation (for be- tween i 1 and N) we multiply the ith equation by and subtract it from the jth equation. This process is perhaps more succinctly described by the following program, which reads in N followed by an N-by-( N + 1) matrix, performs the forward elimination, and writes out the triangulated result. In the input, and in the output the ith line contains the ith row of the matrix followed by program output); var a: of real; i, k, integer; (N) for to N do

begin for to do k]); end; for to N do

for to N do

for i do

for to N do

begin for to do k]); end; end.

GAUSSIAN 61

(As we found with polynomials, if we wtint to have a program that takes N as input, it is necessary in Pascal to first decide how large a value of N will be “legal,” and declare the array suitably.) Note that the code consists of three nested loops, so that the total running time is essentially proportional to The third loop goes backwards so as to avoid destroying i] before it is needed to adjust the values of other #elements in the same row.

The program in the above paragraph is too simple to be quite right: i] might be zero, so division by zero could ‘occur. This is easily fixed, because we can exchange any row (from to N) with the ith row to make i] non-zero in the outer loop. If no such row can be found, then the matrix is singular: there is no unique solution.

In fact, it is advisable to do slightly more than just find a row with a non-zero entry in the ith column. It’s best to use the row (from to N) whose entry in the ith column is the largest in absolute value. The reason for this is that severe computational errors can arise if the i] value which is used to scale a row is very small. If i] is very small, then the scaling factor

i] which is used to eliminate the ith variable from the jth equation (for j from to will be very large. In fact, it could get so large as to dwarf the actual coefficients k], to the point where the value gets distorted by “round-off error.”

Put simply, numbers which differ greatly in magnitude can’t be accurately added or subtracted in the floating point number system commonly used to represent real numbers, but using a small i] value greatly increases the likelihood that such operations will have to be performed. Using the largest value in the ith column from rows to N will ensure that the scaling factor is always less than 1, and will prevent the occurrence of this type of error. One might contemplate looking beyond the ith column to find a large element, but it has been shown that accurate answers can be obtained without resorting to this extra complication.

The following code for the forward elimination phase of Gaussian elimina- tion is a straightforward implementation of this process. For each i from to we scan down the ith column to find the largest element (in rows past the ith). The row containing this element is exchanged with the ith , then the ith variable is eliminated in the equations to N exactly as before:

62 5

procedure eliminate; var i, j, k, max: integer;

t: real;