R by specifying the mode argument as ‘complete’, although this is not required for most applications. The example below defines a 3 × 2 matrix, calculates the QR decomposition, then reconstructs the original matrix from the decomposed elements.
# QR decomposition
Listing 14.3: Example of calculating an QR decomposition.
Running the example first prints the defined 3 × 2 matrix, then the Q and R elements, then finally the reconstructed matrix that matches what we started with.
[[1 2]
Listing 14.4: Sample output from calculating an QR decomposition.
14.5 Cholesky Decomposition
The Cholesky decomposition is for square symmetric matrices where all values are greater than zero, so-called positive definite matrices. For our interests in machine learning, we will focus on the Cholesky decomposition for real-valued matrices and ignore the cases when working with complex numbers. The decomposition is defined as follows:
A = L · LT (14.6)
14.5. Cholesky Decomposition 113 Or without the dot notation:
A = LLT (14.7)
Where A is the matrix being decomposed, L is the lower triangular matrix and LT is the transpose of L. The decompose can also be written as the product of the upper triangular matrix, for example:
A = UT · U (14.8)
Where U is the upper triangular matrix. The Cholesky decomposition is used for solving linear least squares for linear regression, as well as simulation and optimization methods. When decomposing symmetric matrices, the Cholesky decomposition is nearly twice as efficient as the LU decomposition and should be preferred in these cases.
While symmetric, positive definite matrices are rather special, they occur quite frequently in some applications, so their special factorization, called Cholesky de-composition, is good to know about. When you can use it, Cholesky decomposition is about a factor of two faster than alternative methods for solving linear equations.
— Page 100, Numerical Recipes: The Art of Scientific Computing, Third Edition, 2007.
The Cholesky decomposition can be implemented in NumPy by calling the cholesky() function. The function only returns L as we can easily access the L transpose as needed. The example below defines a 3 × 3 symmetric and positive definite matrix and calculates the Cholesky decomposition, then the original matrix is reconstructed.
# Cholesky decomposition
Listing 14.5: Example of calculating an Cholesky decomposition.
Running the example first prints the symmetric matrix, then the lower triangular matrix from the decomposition followed by the reconstructed matrix.
[[2 1 1]
[1 2 1]
[1 1 2]]
[[ 1.41421356 0. 0. ]
[ 0.70710678 1.22474487 0. ]
14.6. Extensions 114
[ 0.70710678 0.40824829 1.15470054]]
[[ 2. 1. 1.]
[ 1. 2. 1.]
[ 1. 1. 2.]]
Listing 14.6: Sample output from calculating an Cholesky decomposition.
14.6 Extensions
This section lists some ideas for extending the tutorial that you may wish to explore.
Write a summary of matrix decomposition to explain the principle to other students.
Create one example using each operation with your own small array data.
Search machine learning papers and find 1 example of each operation being used.
If you explore any of these extensions, I’d love to know.
14.7 Further Reading
This section provides more resources on the topic if you are looking to go deeper.
14.7.1 Books
Section 6.6 Matrix decompositions. No Bullshit Guide To Linear Algebra, 2017.
http://amzn.to/2k76D4
Lecture 7 QR Factorization, Numerical Linear Algebra, 1997.
http://amzn.to/2BI9kRH
Section 2.3 LU Decomposition and Its Applications, Numerical Recipes: The Art of Scientific Computing, Third Edition, 2007.
http://amzn.to/2BezVEE
Section 2.10 QR Decomposition, Numerical Recipes: The Art of Scientific Computing, Third Edition, 2007.
http://amzn.to/2BezVEE
Section 2.9 Cholesky Decomposition, Numerical Recipes: The Art of Scientific Computing, Third Edition, 2007.
http://amzn.to/2BezVEE
Lecture 23, Cholesky Decomposition, Numerical Linear Algebra, 1997.
http://amzn.to/2BI9kRH
14.8. Summary 115
14.7.2 API
scipy.linalg.lu() API.
https://docs.scipy.org/doc/scipy-1.0.0/reference/generated/scipy.linalg.lu.
html
numpy.linalg.qr() API.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.linalg.qr.
html
numpy.linalg.cholesky() API.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.linalg.cholesky.
html
14.7.3 Articles
Matrix decomposition on Wikipedia.
https://en.wikipedia.org/wiki/Matrix_decomposition
LU decomposition on Wikipedia.
https://en.wikipedia.org/wiki/LU_decomposition
QR Decomposition on Wikipedia.
https://en.wikipedia.org/wiki/QR_decomposition
Cholesky decomposition on Wikipedia.
https://en.wikipedia.org/wiki/Cholesky_decomposition
14.8 Summary
In this tutorial, you discovered matrix decompositions and how to calculate them in Python.
Specifically, you learned:
What a matrix decomposition is and why these types of operations are important.
How to calculate an LU and QR matrix decompositions in Python.
How to calculate a Cholesky matrix decomposition in Python.
14.8.1 Next
In the next chapter you will discover the eigendecomposition, eigenvalues, and eigenvectors.
Chapter 15
Eigendecomposition
Matrix decompositions are a useful tool for reducing a matrix to their constituent parts in order to simplify a range of more complex operations. Perhaps the most used type of matrix decomposition is the eigendecomposition that decomposes a matrix into eigenvectors and eigenvalues. This decomposition also plays a role in methods used in machine learning, such as in the Principal Component Analysis method or PCA. In this tutorial, you will discover the eigendecomposition, eigenvectors, and eigenvalues in linear algebra. After completing this tutorial, you will know:
What an eigendecomposition is and the role of eigenvectors and eigenvalues.
How to calculate an eigendecomposition in Python with NumPy.
How to confirm a vector is an eigenvector and how to reconstruct a matrix from eigenvectors and eigenvalues.
Let’s get started.
15.1 Tutorial Overview
This tutorial is divided into 5 parts; they are:
1. Eigendecomposition of a Matrix 2. Eigenvectors and Eigenvalues 3. Calculation of Eigendecomposition 4. Confirm an Eigenvector and Eigenvalue 5. Reconstruct Matrix
116