This intermediate-level tutorial illustrates how to combine several simple building blocks to construct an advanced procedure called Principal Components Analysis, a powerful statistical technique used in pattern recognition (hereafter denoted "PCA"). It is assumed that you are already familiar with creating and running Eigen4AutoIt scripts, and are ready for the big time!
This simplified case-study is based on this resource by L.I. Smith.; you may also find these two pdfs helpful. Alternatively, you could
google "PCA tutorial" or similar terms for additional explanation. The argument as presented here goes like this:
Okay, let's dance! Here we have ten points in 2D space; these could be coordinates X and Y, or
simultaneous observations of two physical properties, for example. If you've been a diligent observer, they may look strangely familiar...
#include "Eigen4AutoIt.au3"; get ready to rumble!
_Eigen_Startup () ; you could use _Eigen_Startup ( "double" ) for more accurate results _Eigen_SetListMarker (); to enable a cleanup afterwards
$rows=10 $cols=2
Local $arrayA[ $rows][ $cols ] = [ _ [ 2.5, 2.4 ], _ [ 0.5, 0.7 ], _ [ 2.2, 2.9 ], _ [ 1.9, 2.2 ], _ [ 3.1, 3.0 ], _ [ 2.3, 2.7 ], _ [ 2.0, 1.6 ], _ [ 1.0, 1.1 ], _ [ 1.5, 1.6 ], _ [ 1.1, 0.9 ] ]
$matA=_Eigen_CreateMatrix_FromArray($arrayA) ; data matrix A MatrixDisplay($matA, "original data")
When we plot these data, the origin is the intersection of the two zero axes at bottom left. The dots are not centered around this origin, so first we'll translate them back there. To do this, we take the mean per column, and subtract those means from all individual values within that column. Since we'll need the means again later, we'll store those too.
"Matrixspecs* functions. Here we use _Eigen_MatrixSpecs_Colwise_Single(); "Colwise" = repeat for each column, "Single" = retrieve a single property of the target matrix. We end up with a Rowvector M (a matrix with one row and multiple columns) in which each cell contains the mean for its column in matrix A:
$specID = 3 ; we wish to retrieve the mean (average)
$matM = _Eigen_MatrixSpecs_Colwise_Single ( $matA, $specID ) ; get this spec per column _MatrixDisplay ( $matM, "column means" ) ; Rowvector M
Next, we have to subtract this vector of column means from the cells in each row of A. We also want to keep our original data, so we first clone matrix A. Some explanation of terms again: "Cwise" = cell-wise, or coefficient-wise = act on each cell individually; "Rowwise" = perform the action on each row in turn; "BinaryOp" = Binary Operator = use an operator with the data from two different matrices, storing the result in the first input matrix; "InPlace" = store the result in the input matrix itself.
$matB = _Eigen_CloneMatrix ( $matA ) ; create a new matrix (B), duplicating matrix A
_Eigen_CwiseBinaryOp_Rowwise_InPlace ( $matB, "-", $matM ) ; NB alters content of first matrix parsed!
$matZ = _Eigen_Center_Colwise ( $matA ) ; subtract column mean from each cell in that column, for each column in turn
_MatrixDisplay ( $matZ, "ditto direct" )
MsgBox ( 0, "IsCwiseEqual?", "Are these two results the same?" & @CR & _
( ( _Eigen_IsCwiseEqual_AB ( $matB, $matZ, 1.0e-3 ) ) ? "Yeah, Baby!" : "No way, Bro!" ) ) ; AutoIt's ternary operator
PCA is all about quantifying the dimensionality of data variability. For starters, we need to know how broad the variability is for each variable, and how changes in variable X affect variable Y. This is called covariance. The square covariance matrix contains the data variances on the diagonal (column = variable), and the co-variance of XY pairs in the off-diagonal cells (values mirror across the diagonal, because pair XY = pair YX).
$matC = _Eigen_Covariance_Colwise ( $matB ) _MatrixDisplay ( $matC, "covariance (Colwise)" )
Now we bring out the big guns: Jacobi Singular Value Decomposition (or "SVD," for Singular Value Decomposition; see the Jacobi Test script for info on what else it can produce). Before summoning this fearless number cruncher, we need to tell it which outputs to produce (matrices S and U in this case, to be explained below):
Next, we adjust the operational specifics of this decomposition with various flags, call the decomposition, collect the matrix ID of the returned eigenvalues (matrix S) and stick the latter in new vector D:
$enableSpecs = False ; various scalar values describing the results; not interesting here. $fullPivoting = False ; Pivoting is irrelevant here, because it deals with non-square inputs, $colPivoting =False ; and a covariance matrix is square by definition
$computeThin =False ; likewise irrelevant, for the same reason ; Houston, we are GO for launch!
_Eigen_Decomp_JacobiSVD ( $matC, $enableSpecs, $fullPivoting, $colPivoting, $computeThin ) If @error Then Exit ; if @error = 0: we're good, people!
$matS = _Eigen_GetActiveMatrix ( "S" ) ; Get the newly created matrix ID
_MatrixDisplay ( $matS, "SVD matrix S" ) ; Display the eigenVALUES = the length of each PC axis $matD = _Eigen_CreateMatrix ( 1, $cols ) ; create a Rowvector of length $cols for it
_Eigen_Copy_Adiag_ToBvector ( $matS, $matD ); store the eigenvalues here
The eigenvalues are size-ordered, and quantify how much of total data variability is mapped by each PC axis. So the first axis explains most overall variability, followed by number two, then three (if our data had three columns), and so forth. Note that the tail end of a highly-dimensional system will likely contain axes with an eigenvalue of zero (or close to zero). These map the system's null space, that is, they carry no additional information, and create a nasty singularity that will totally destroy subsequent results if given half a chance. Thus, if you're planning on using your principal components in follow-up procedures, it's an excellent idea to get rid of all bad apples with (near-)zero eigenvalues first.
An easy way to evaluate the relative size of the eigenvalues is in terms of proportions of one (1), or as percentages:
$specID = 1 ; define the "sum" of all cell values
$total = _Eigen_MatrixSpecs_Single ( $matD, $specID ) ; store sum of all matrix cells in $total _Eigen_CwiseScalarOp_InPlace ( $matD, "/", $total ) ; divide each cell by our sum
These results tell us that the second PC dimension contributes only a small fraction (less than 4%) to total data variability. This will become important later on. For now, let's move on to the second major output of Jacobi SVD: the eigenvectors = left singular vectors = orthogonal unit vectors. Rows represent the relative contributions of each original variable to each principal component (variable's contribution = row, PC = column). The values per column are sometimes called the PC "coefficients" or "loadings". Their generic output container is matrix U.
$matU = _Eigen_GetActiveMatrix ( "U" ) ; Get the newly created matrix ID _MatrixDisplay ( $matU, "SVD matrix U" ) ; show the eigenvectors
If you have large multivariate data sets, specific principal components may (if you are lucky) comprise major contributions from only one or a few original variables (that may themselves be related in some meaningful way). Since matrix U's columns are (like the eigenvalues) ordered by size, this means that the most influential variables, i.e., the ones explaining most observed variability overall, are the ones with the largest coefficients in the left-most columns. Some scientific discoveries are made this way. In this case we're not so lucky, as all coefficient contributions are approximately equal per component.
Question: why do we need the eigenvectors (that is, matrix U)? Answer: left-multiplying SVD matrix U with the original data constitutes an orthogonal transformation into size-ordered principal component-space, in which the first axis represents the axis of largest variance, and consecutive axes map ever smaller residual data variability. The resulting coordinates are sometimes called the "PC scores" (don't worry if this still sounds cryptic).
We can use the mean-centered data for this...
$matQ = _Eigen_Multiply_ABt ( $matU, $matB ) ; NB produces a transposed result! _MatrixDisplay ( $matQ, "PC scores: based upon centered data (transposed)" )
...or we can use the original data:
$matP = _Eigen_Multiply_ABt ( $matU, $matA ) ; produces a transposed result! _MatrixDisplay ( $matP, "PC scores: based upon original data (transposed)" )
These two results produce the same spatial point cloud, except that matrix P = matrix Q translated (shifted) by [the column means multiplied by the eigenvectors], as is shown here (and check out the double-nested call in the snippet):
$matT = _Eigen_CloneMatrix ( $matQ ) ; create a test container
_Eigen_CwiseBinaryOp_Colwise_InPlace ( $matT, "+", _Eigen_Transpose ( _Eigen_Multiply_AB ( $matM, $matU ) ) ) ; nested calls
_MatrixDisplay ( $matT, "matQ translated by M colwise" )
MsgBox ( 0, "IsCwiseEqual?", "Are these two results identical?" & @CR & _
( ( _Eigen_IsCwiseEqual_AB ( $matP, $matT, 1.0e-3 ) ) ? "Yeah, Baby!" : "No way, Bro!" ) ) ; AutoIt's ternary operator
Note the difference in PC axis scales when we plot matrix P; the vertical range (PC2, range ca. 0.8) is now a small fraction of the horizontal range (PC1, range ca. 4), unlike in the original data plot, where they were about equal in size.
Congratulations! You have transformed your data into principal component space. Now what? What does this even mean?
Well, remember how the original input variables seemed mutually correlated? The new axes are all orthogonal (at right angles) to one another, and the points' coordinates along any particular axis are now uncorrelated(!) with their counterparts along all other axes. This means that we've effectively split the compound profile of the original, interdependent data into separate, independent contributing factors (= axes). Let that sink in for a moment.
PCA is a reversible operation, meaning we can recover our original data from the PCs if we so desire. Because U is an orthogonal matrix, its inverse equals its transpose by definition: orthogonal U-1 = UT, so R
= PT * UT. Instead of creating yet another matrix container just to store the transposed version of U, we'll
call a dedicated Multiply function in the following code snippet that reads in the original matrices
transposed directly, without first moving or copying the data inside them. To retrieve our original data, we multiply our PC-transformed points with the transpose of (eigenvector) matrix U, and, if using centered data, add our earlier-stored column means per column to each cell to de-center the coordinates again:
$matR = _Eigen_Multiply_AtBt ( $matP, $matU ) ; both matrices are multiplied in transposed form _MatrixDisplay ( $matR, "P-recovered data" )
$matR = _Eigen_CwiseBinaryOp_Rowwise ( _Eigen_Multiply_AtBt ( $matQ, $matU ), "+", $matM ) ; nested calls
_MatrixDisplay ( $matR, "Q-recovered data" )
Function _Eigen_CwiseBinaryOp_Rowwise() we've seen before when we subtracted the mean with the minus operator ("-"). This time, we're adding the mean back with the plus operator ("+"). It really is that simple! It's actually more complicated to do this in Eigen/C++ itself!
Incidentally, the nested call works like this:
1. _Eigen_Multiply_AtBt() returns the ID of a new, so far unnamed matrix;
2. _Eigen_CwiseBinaryOp_Rowwise() alters that matrix with the contents of matrix M and assigns the former's ID (with altered contents) to variable $matR (so we can handle matrix R thereafter).
Still with me? Because now comes THE IMPORTANT PART! (fanfare, dancing elephants, fireworks...) Recall that we earlier established that the second principal component (PC2) contributed very little to overall data variability. That means that if we discard it, we won't lose much information. This is often called "dimensional reduction." It is used in data compression and facial recognition, for example. Common practice is to discard all PC's with eigenvalues below unity (the value 1), or to plot eigenvalue versus principal component rank and look for a steep drop-off. We can also take the cumulative percentage of total variability explained by adding all cell values in matrix D (a vector) up to cell (PC) 2, 3, 4, and so on, until we reach our target of X% of total variance mapped. The target may be >50%, or 90%, or 95%, for example. The choice of cut-off depends on the data context and your own aims and judgement. Seek thee Wisdom!
We can discard principal components by zeroing out their column in matrix U. This is even easier than it sounds. In our simplified example, we'll discard PC2, keeping only PC1. Generally speaking, please be careful when destroying data like this; make certain you won't be needing the discarded columns of matrix U in future, or else make a backup copy of U beforehand (nag nag nag).
The data reconstruction derived from the single remaining eigenvector is now neatly aligned, while retaining over 96% of the original variability. The resulting graph of our new dataset shows that the back- projected point cloud has lost the secondary scatter at right angles to its main axis, while retaining the full spread along its slope. This is called "lossy compression," used, for example, to turn .wav sound files (as stored on music CDs) into .mp3 files (which, on average, require about one tenth the space of a wav file; now you know why).
Obviously, the full power of PCA only becomes apparent once you are dealing with large data sets with dozens, thousands, perhaps even millions of variables (= dimensions). For example, in facial recognition, each same-sized B/W image can be stored as a single row vector of pixel grayscale values (in which the original pixel rows of the image are simply appended in sequence). Suppose we then compare 50 images of different faces, and each image is 1000 x 1500 pixels. Then our input matrix has 50 rows and 1.5 million
columns (variables) = 75 million cells (note: special ma-tricks exist to reduce this huge multi-dimensional space prior to computation).
PCA may then establish "the common face" defined by, say, the first few hundred principal components, and then express the original images in terms of those PCs only (lossy compression), or attempt to identify a new image by finding the nearest match in terms of the PC profile of the faces in the database (facial recognition). Google the term "Eigenface" to find out more. Issues concerning differences in scale, image quality, lighting, angle, and facial expression complicate such efforts no end, so I would suggest you start exploring PCA with a more modest problem.
Now that you've worked through the steps and examined the various outputs, PCA hopefully no longer seems a blackbox algorithm anymore. You are now fully prepared to start using the two in-built functions _Eigen_PCA() and _Eigen_PCA_ReDim() directly. The latter has parameter $PCs_to_Keep as input and spits out matrix R; If $PCs_to_Keep equals the number of original variables or is left at zero (the default), you will retrieve your original data set (useful as a test only). Enabling P and/or Q suffices here; the rest is activated as needed (flags-dependent). In the following snippet (the start of a new script), we omit duplicating the first few lines of the Tutorial script, where we initialise the work environment and load matrix A with the original data (see first snippet).
_Eigen_SetActiveMatrix ( "PQ", True ) ; True = reset all flags first _Eigen_PCA ( $matA ) ; rather shorter than before, no?
If @error Then Exit ; Can I go now?
Alternatively, you could have used the provided flags $returnPscores and $returnQscores, like this: _Eigen_ResetActiveMatrix() ; we're not using these switches now; this function disables them all $center = True ; set some operational booleans
$normalise = False $computeCovar = True
$returnPscores = True ; set this True if you want matrix P returned $returnQscores = True ; set this True if you want matrix Q returned
_Eigen_PCA ( $matA, $center, $normalise, $computeCovar, $returnPscores, $returnQscores ) If @error Then Exit
$matB = _Eigen_GetActiveMatrix ( "B" ) ; You can store these IDs anywhere you like $matC = _Eigen_GetActiveMatrix ( "C" ) ; (they are just indices to array $MatrixList), $matD = _Eigen_GetActiveMatrix ( "D" ); but here we'll use sensible variable names. $matM = _Eigen_GetActiveMatrix ( "M" ); ...
$matP = _Eigen_GetActiveMatrix ( "P" ) ; "I like New York in June...."
$matQ = _Eigen_GetActiveMatrix ( "Q" ); How about you?" (ta dum dum dum), $matS = _Eigen_GetActiveMatrix ( "S" ) ; "I like a Gershwin tune...."
_MatrixDisplay ( $matS, "eigenvalues" ) _MatrixDisplay ( $matU, "eigenvectors" )
After calling _Eigen_PCA() once, we can try different PC transforms as often as needed, without calling _Eigen_PCA() over and over again. We can use either P, or Q and M (both!) as inputs, but not all at the same time. Here is the matrix P-based data recovery test:
$PCs_toKeep = 0 ; use all PCs (recovery testing)
$matR = _Eigen_PCA_ReDim ( $matU, $PCs_toKeep, 0, $matP ) ; 0 = $matM is not used here If @error Then Exit
Local $arrayR ; let's capture the results *directly* in an array this time _Eigen_CreateArray_FromMatrix ( _Eigen_GetActiveMatrix ( "R" ), $arrayR ) _ArrayDisplay ( $arrayR, "recovered via P" )
And here's the QM-based recovery test (only the second and last line are different): $PCs_toKeep = 0 ; use all PCs (recovery testing)
$matR = _Eigen_PCA_ReDim ( $matU, $PCs_toKeep, $matM, 0, $matQ ) ; 0 = $matP is not used here If @error Then Exit
Local $arrayR ; a bit redundant perhaps (see above)
_Eigen_CreateArray_FromMatrix ( _Eigen_GetActiveMatrix ( "R" ), $arrayR ) _ArrayDisplay ( $arrayR, "recovered via QM" )
Actual dimensional reduction is achieved by setting $PCs_toKeep to some integer value larger than zero but smaller than the number of columns of the input matrix. In our simplified example, we started out with two axes, so we can only discard PC2 here, keeping one axis.
$PCs_toKeep = 1 ; At last, dimensional reduction!
$matR = _Eigen_PCA_ReDim ( $matU, $PCs_toKeep, $matM, 0, $matQ ) ; 0 = $matP is not used here If @error Then Exit
_MatrixDisplay ( _Eigen_GetActiveMatrix ( "R" ), "lossy compression" ) ; matrix ID not stored
Finally, we release all matrices created during this session, and close the dll handle. Check out the reference links at the top of this Tutorial if you wish to learn more. Good night, and good luck!
_Eigen_Cleanup ()
MsgBox ( 0, "Eigen4AutoIt", "This concludes this Tutorial." )
Pooled Multi-Processing