One of the most useful skills you’ll learn in linear algebra is the ability to solve systems of linear equations. Many real-world problems are expressed as linear equations in multiple unknown quantities. You can solve for n unknowns simultaneously if you have a set of n linear equations that relate the unknowns. To solve this system of equations, you can use basic techniques such as substitution, subtraction, and elimination by equating to eliminate the variables one by one (see Section 1.15), but the procedure will be slow and tedious for many unknowns. If the system of equations is linear, it can be expressed as an augmented matrix built from the coefficients in the equations. You can then use the Gauss–Jordan elimination algorithm to solve for the nunknowns (Section 4.1). The key benefit of the augmented matrix approach is that it allows you to focus on the coefficients without worrying about the variable names. This saves time when you must solve for many unknowns. Another approach for solving n linear equations in n unknowns is to express the system of equations as a matrix equation (Section 4.2) and then solve the matrix equation by computing the matrix inverse (Section 4.5).
In Section 7.6 you’ll learn how to decompose a matrix into a prod-uct of simpler matrices. Matrix decompositions are often performed for computational reasons: certain problems are easier to solve on a computer when the matrix is expressed in terms of its simpler con-stituents. Other decompositions, like the decomposition of a matrix into its eigenvalues and eigenvectors, give you valuable information about the properties of the matrix. Google’s original PageRank algo-rithm for ranking webpages by “importance” can be explained as the search for an eigenvector of a matrix. The matrix in question contains information about all hyperlinks that exist between webpages. The eigenvector we’re looking for corresponds to a vector that describes the relative importance of each page. So when I tell you eigenvec-tors are valuable information, I am not kidding: a 350-billion dollar company started as an eigenvector idea.
The techniques of linear algebra find applications in many areas of science and technology. We’ll discuss applications such as modelling multidimensional real-world problems, finding approximate solutions to equations (curve fitting), solving constrained optimization
prob-3.6 INTRODUCTORY PROBLEMS 135
lems using linear programming, and many other in Chapter 8. As a special bonus for readers interested in physics, a short introduction to quantum mechanics can be found in Chapter 10; if you have a good grasp of linear algebra, you can understand matrix quantum mechanics at no additional mental cost.
Our journey of all things linear begins with the computational aspects of linear algebra. In Chapter 4 we’ll learn how to efficiently solve large systems of linear equations, practice computing matrix products, discuss matrix determinants, and compute matrix inverses.
3.6 Introductory problems
Before we continue with the new material in linear algebra, we want to make sure you got the definitions down straight.
linearity, vector operations, matrix operations, etc.
P3.1 Find the sum of the vectors (1, 0, 1) and the vector (0, 2, 2).
Hint: Vector addition is performed element-wise.
P3.2 You friend is taking a physics class and needs some help with a vector question. Can you help your friend answer this question: Let |ai = 1|0i + 3|1i and |bi = 4|0i − 1|1i. Find |ai + |bi.
Hint: The weird angle-bracket notation denotes basis vectors: |xi ≡ ~ex. P3.3 Given ~v = (2, −1, 3) and ~w = (1, 0, 1), compute the following vector products: a) ~v · ~w; b) ~v × ~w; c) ~v × ~v; d) ~w × ~w.
P3.4 Given unit vectors ˆı = (1, 0, 0), ˆ = (0, 1, 0) and ˆk = (0, 0, 1). Find the following cross products: a) ˆı × ˆı; b) ˆı × ˆ; c) (−ˆı) × ˆk + ˆ × ˆı;
d) ˆk × ˆ+ ˆı× ˆı+ ˆ× ˆk + ˆ× ˆı.
Chapter 4
Computational linear algebra
This chapter covers the computational aspects of performing matrix calculations. Understanding matrix computations is important be-cause all later chapters depend on them. Suppose we’re given a huge matrix A ∈ Rn×nwith n = 1000. Hidden behind the innocent-looking mathematical notation of the matrix inverse A−1, the matrix prod-uct AA, and the matrix determinant |A|, lie monster computations involving all the 1000 × 1000 = 1 million entries of the matrix A.
Millions of arithmetic operations must be performed... so I hope you have at least a thousand pencils ready!
Okay, calm down. I won’t actually make you calculate millions of arithmetic operations. In fact, to learn linear algebra, it is sufficient to know how to carry out calculations with 3×3 and 4×4 matrices. Even for such moderately sized matrices, computing products, inverses, and determinants by hand are serious computational tasks. If you’re ever required to take a linear algebra final exam, you need to make sure you can do these calculations quickly. Even if no exam looms in your imminent future, it’s important to practice matrix operations by hand to get a feel for them.
This chapter will introduce you to the following computational tasks involving matrices:
Gauss–Jordan elimination Suppose we’re trying to solve two equa-tions in two unknowns x and y:
ax + by = c, dx + ey = f.
137
If we add α-times the first equation to the second equation, we obtain an equivalent system of equations:
ax + by = c
(d + αa)x + (e + αb)y = f + αc.
This is called a row operation: we added α-times the first row to the second row. Row operations change the coefficients of the system of equations, but leave the solution unchanged. Gauss–Jordan elimina-tion is a systematic procedure for solving systems of linear equaelimina-tions using row operations.
Matrix product The product AB between matrices A ∈ Rm×`
and B ∈ R`×n is the matrix C ∈ Rm×n whose coefficients cij are defined by the formula cij =P`
k=1aikbkj for all i ∈ [1, . . . , m] and j∈ [1, . . . , n]. In Section 4.3 we’ll unpack this formula and learn about its intuitive interpretation: that computing C = AB is computing all the dot products between the rows of A and the columns of B.
Determinant The determinant of a matrix A, denoted |A|, is an operation that gives us useful information about the linear indepen-dence of the rows of the matrix. The determinant is connected to many notions of linear algebra: linear independence, geometry of vec-tors, solving systems of equations, and matrix invertibility. We’ll discuss these aspects of determinants in Section 4.4.
Matrix inverse In Section 4.5 we’ll build upon our knowledge of Gauss–Jordan elimination, matrix products, and determinants to de-rive three different procedures for computing the matrix inverse A−1.
4.1 Reduced row echelon form
In this section we’ll learn to solve systems of linear equations using the Gauss–Jordan elimination procedure. A system of equations can be represented as a matrix of coefficients. The Gauss–Jordan elimina-tion procedure converts any matrix into its reduced row echelon form (RREF). We can easily find the solution (or solutions) of the system of equations from the RREF.
Listen up: the material covered in this section requires your full-on, caffeinated attentifull-on, as the procedures you’ll learn are somewhat tedious. Gauss–Jordan elimination involves many repetitive mathe-matical manipulations of arrays of numbers. It’s important you hang in there and follow through the step-by-step manipulations, as well
4.1 REDUCED ROW ECHELON FORM 139
as verify each step I present on your own with pen and paper. Don’t just take my word for it—always verify the steps!
Solving equations
Suppose you’re asked to solve the following system of equations:
1x1+ 2x2 = 5 3x1+ 9x2 = 21.
The standard approach is to use one of the equation-solving tricks we learned in Section 1.15 to combine the equations and find the values of the two unknowns x1and x2.
Observe that the names of the two unknowns are irrelevant to the solution of the system of equations. Indeed, the solution (x1, x2)to the above system of equations is the same as the solution (s, t) to the system of equations
1s + 2t = 5 3s + 9t = 21.
The important parts of a system of linear equations are the coefficients in front of the variables and the constants on the right-hand side of each equation.