Common factor analysis

(1)

(2)

Common factor analysis

y This is what people generally mean when they say "factor analysis"

y This family of techniques uses an estimate of common variance among the original variables to generate the factor solution

y It is based on the fundamental assumption that some underlying factors, which are smaller in number than the observed variables, are

responsible for the covariation among them

(3)

Questions to answer

y

How many different factors are needed to explain the pattern of relationships among these variables?

y

What is the nature of those factors?

y

How well do the hypothesized factors explain the observed data?

y

How much purely random or unique variance does each observed variable include?

y

As with previous techniques we have discussed, the goal is

dimension reduction, i.e. to describe a number of variables in a

simpler form

(4)

Issues

y Factor solutions in the exploratory endeavor will be different depending on the data, algorithm and other researcher choices

y The goal with exploratory factor analysis is to discover structure, not determine it

y There are also differences in reporting such that one

may see quite different results from sample to sample,

study to study

(5)

Factor Analysis

y There are four basic steps:

y

data collection and generation of the correlation matrix

y

extraction of initial factor solution

y

rotation and interpretation (also validation)

y

construction of scales or factor scores to use in further analyses

y A good factor:

y

Makes sense

y

Will be easy to interpret

y

Possesses simple structure

y Items have low cross‐loadings

(6)

Factor Analysis

y Factor analysis can be seen as a family of techniques, of which both PCA and EFA are members

¹

y Factor analysis is a statistical approach that can be used to analyze interrelationships among a large number of

variables and to explain these variables in terms of their common underlying dimensions (factors)

y It involves finding a way of condensing the information contained in a number of original variables into a smaller set of dimensions (factors) with a minimum loss of

information

(7)

Principal Components Analysis

y Principle components analysis (PCA) is a statistical technique applied to a single set of variables to discover which variables in the set form coherent subsets that are independent of one another

y Provides a unique solution, so that the original data, the covariance or correlation matrix, can be reconstructed from the results

y Looks at the total variance among the variables, so the solution generated will include as many factors/components as there are variables, although it is unlikely that they will all meet the criteria for retention

y Variables that are correlated with one another which are also largely independent of other subsets of variables are combined into factors y Factors are generated which are thought to be representative of the

underlying processes that have created the correlations among variables

y The underlying notion of PCA is that the observed variables can be transformed into linear combinations of an underlying set of

hypothesized or unobserved components (factors) y PCA is typically exploratory in nature

(8)

Common factor model

y PCA and common factor analysis may utilize a similar method and are conducted with similar goals in mind

y The difference between PCA and common FA involves the underlying model

y The common factor model for factor analysis

y PCA assumes that all variance is common, which is akin to assuming they are perfectly reliable¹

y All ‘unique factors’, i.e. sources of variability not attributable to a factor, set equal to zero

y The common factor model on the other hand holds that the observed variance in each measure is attributable to relatively small set of

common factors (latent characteristics common to two or more variables), and a single specific factor unrelated to any other underlying factor in the model

(9)

Comparison of underlying models ¹

y PCA

y

Extraction is the process of forming PCs as linear combinations of the measured variables as we have done with our other techniques

y PC₁ = b₁₁X₁ + b₂₁X₂ + … + b_k1X_k

y PC₂ = b₁₂X₁ + b₂₂X₂ + … + b_k2X_k

y PC_f = b_1fX₁ + b_2fX₂ + … + b_kfX_k

y Common factor analysis

y

Each measure X has two contributing sources of variation: the common factor ξ and the specific or unique factor δ:

²

y X₁ = λ₁ξ + δ₁

y X₂ = λ₂ξ + δ₂

y X_f = λ_fξ + δ_f

(10)

Example

y Consider the following example from Holzinger and Swineford 1939

y Ol’ skool yeah!

y Variables

y Paragraph comprehension

y Sentence completion

y Word meaning

y Addition

y Counting dots

y Each person’s score is a reflection of the weighted combination of the common factor (latent variable) and measurement error (uniqueness, unreliability)

y In these equations, λ represents the

extent to which each measure reflects the underlying common factor

(11)

Factor Analysis

y When standardized the variance can be decomposed as follows:

y λi, in unsquared form, is now interpretable as a

correlation coefficient and its square the proportion of the variation in X accounted for by the common factor, i.e. the communality

y The remaining is that which is accounted for by the specific factor or uniqueness (individual differences, measurement error, some other known factor e.g.

intelligence)

var( X

_i

) = var( λ ξ δ

_i

+

_i

) = λ

2

+ var( ) δ

_i

= 1

2

var( ) 1

ii

ii communality δ θ

θ

=

− =

(12)

Measurement error

y The principal axis extraction method of EFA can be distinguished from PCA in terms of simply having communalities on the diagonal of the correlation matrix instead of 1

^s

y What does this mean?

y Having 1

^s

assumes each item/variable is perfectly reliable, i.e. there is no

measurement error in its ability to distinguish among cases/individuals

L = Eigenvalue matrix V = Eigenvector matrix R = Correlation matrix

A = Loading matrix L = V'RV

R = AA'

(13)

Measurement error

y

The fact that our estimates in psych are not perfectly reliable suggests that we use methods that take this into account

y

This is a reason to use EFA over PCA in cases involving

measurement scales or items, as the communalities can be seen as the lower bound (i.e. conservative) estimate of the variables’

reliability

y

Note however, that low communalities are not interpreted as evidence of poor fit so much as evidence that the variables

analyzed have little in common with one another, and thus are

not ‘reliable’ measures of a proposed factor solution

(14)

Two‐factor solution

y From our example before, we could have selected a two factor solution

y Quantitative vs. Verbal reasoning

y Now we have three sources of variation observed in test scores

y Two common factors and one unique factor

y As before, each λ reflects the extent to which each common factor

contributes to the variance of each test score

y The communality for each variable

is now λ

²1

+ λ

²2

(15)

Analysis: the Correlation Matrices

y Observed Correlation Matrix

y

Note that the manner in which missing values are dealt with will determine the observed correlation matrix

y

In typical FA settings one may have quite a few, and so casewise deletion would not be appropriate

y

Best would be to produce a correlation matrix resulting from some missing values analysis (e.g. EM algorithm)

y Reproduced Correlation Matrix

y

That which is produced by the factor solution

y Recall that in PCA it is identical to the observed

y

It is the product of the matrix of loadings and the transpose of the pattern matrix (partial loadings)*

y Residual Correlation Matrix

y

The difference between the two

res

=

R = AA'

R R - R

(16)

Analysis: is the data worth reducing?

y The Kaiser‐Meyer‐Olkin Measure of Sampling Adequacy

y A statistic that indicates the proportion of variance in your variables that might be caused by common underlying factors

y An index for comparing the magnitudes of the observed

correlation coefficients to the magnitudes of the partial correlation coefficients

y If two variables share a common factor with other variables, their partial correlation will be small once the factor is taken into account y High values (close to 1.0) generally indicate that a factor analysis

may be useful with your data.

y If the value is less than 0.50, the results of the factor analysis probably won't be very useful.

y Bartlett's test of sphericity

y Tests the hypothesis that your correlation matrix is an identity

matrix (1s on the diagonal, 0s off‐diagonals), which would indicate that your variables are unrelated and therefore unsuitable for

structure detection

(17)

Analysis: Extraction Methods

y Principal (Axis) Factors

y Estimates of communalities (SMC) are in the diagonal; used as starting values for the communality estimation (iterative)

y Removes unique and error variance

y Solution depends on quality of the initial communality estimates y Maximum Likelihood

y Computationally intensive method for estimating loadings that maximize the likelihood of sampling the observed correlation matrix from a population y Unweighted least squares

y Minimize off diagonal residuals between reproduced and original R matrix y Generalized (weighted) least squares

y Also minimizes the off diagonal residuals

y Variables with larger communalities are given more weight in the analysis y Alpha factoring

y Maximizes the reliability of the factors y Image factoring

y Minimizes ‘unique’ factors consisting of essentially one measured variable

(18)

Analysis: Rotation Methods

y After extraction, initial interpretation may be difficult y Rotation is used to improve interpretability and utility

y Refer back to the PCA mechanics handout for the geometric interpretation of rotation¹

y By placing a variable in the n‐dimensional space specified by the factors involved, factor loadings are the cosine of the angle formed by a vector from the origin to that coordinate and the factor axis

y Note how PCA would be distinguished from multiple regression

y PCA minimizes the squared distances to the axis, with each point mapping on to the axis forming a right angle (as opposed to ‘dropping straight down’

in MR)

y MR is inclined to account for variance in the DV, where PCA will tilt more to whichever variable exhibits the most variance

(19)

Analysis: Rotation Methods

y

So factors are the axes

y Orthogonal Factors are at right angles

y Oblique rotation allows for other angles

y Often achieve simpler structure, though at the cost that you must also consider the factor inter‐correlations when interpreting

results

y

Repositioning the axes changes the loadings on the factor but keeps the relative positioning of the points the same

y

Length of the line from the origin to the variable coordinates is

equal to the communality for that variable

(20)

Example: Rotation

Note that the variance of the two factors for the original and rotated solutions sum to the same amount

(21)

Analysis: Rotation Methods

y Orthogonal rotation keeps factors uncorrelated while increasing the meaning of the factors

y Varimax – most popular

y ‘Cleans up the factors’

y Makes large loadings larger and small loadings smaller

y Quartimax

y ‘Cleans up the variables’

y Each variable loads mainly on one factor

y Varimax works on the columns of the loading matrix; Quartimax works on the rows

y Not used as often; simplifying variables is not usually a goal

y Equamax

y Hybrid of the two that tries to simultaneously simplify factors and variables

y Not that popular either

(22)

Analysis: Rotation Methods

y Oblique Rotation Techniques

y

Direct Oblimin

y

Begins with an unrotated solution

y

Has a parameter

¹

that allows the user to define the amount of correlation acceptable; gamma values near ‐4 Æ orthogonal, 0 leads to mild correlations (also direct quartimin) and close to 1 highly correlated

y

Promax

²

y

Solution is orthogonally rotated initially (varimax)

y

This is followed by oblique rotation

y

Orthogonal loadings are raised to powers in order to drive

down small to moderate loadings

(23)

Analysis: Orthogonal vs. Oblique output

y Orthogonal Rotation

y Factor matrix

y Correlation between observed variable and factor for the unrotated solution y Pattern vs. structure matrix

y Structure matrix

y Loadings, i.e. structure coefficients

y Correlation between observed variable and factor

y Pattern matrix

y Standardized, partialled coefficients (weights, loadings)

y These structure and pattern matrices are the same in orthogonal solutions and so will not be distinguished

y Factor Score Coefficient matrix

y Coefficients used to calculate factor scores (like regression coefficients) from original variables (standardized)

(24)

Analysis: Orthogonal vs. Oblique output

y Oblique Rotation

y Factor matrix

y Correlation between observed variable and factor for the unrotated solution y Structure Matrix

y Simple correlation between factors and variables

y Factor loading matrix y Pattern Matrix

y Unique relationship between each factor and variable that takes into account the correlation between the factors

y The standardized regression coefficient from the common factor model

y The more factors, the lower the pattern coefficients as a rule since there will be more common contributions to variance explained

y For oblique rotation, the researcher looks at both the structure and pattern coefficients when attributing a label to a factor

y Factor Score Coefficient matrix

y Again used to derive factors scores from the original variables y Factor Correlation Matrix

y correlation between the factors

(25)

Analysis: Factor scores

y Factor scores can be derived in a variety of ways, some of which are presented here¹

y Regression

y Regression factor scores have a mean of 0 and variance equal to the squared multiple correlation between the estimated factor scores and the true factor values. They can be correlated even when factors are assumed to be

orthogonal. The sum of squared residuals between true and estimated factors over individuals is minimized.

y Least squares

y Minimizes squared residuals of scores and true factor scores

y Same approach as above but uses the reproduced R matrix instead of the original

y Bartlett

y Minimizes the effect of unique factors (consisting of single variables)

y Anderson‐Rubin

y Same as Bartlett’s but produces orthogonal factor scores

y Once obtained one can use factor scores in other analyses

y Recall PC regression

(26)

Other stuff: An iterative process

y To get an initial estimate of the communalities, we can simply start with the squared multiple correlation coefficient (R²) for each item regressed on the other remaining items

y There are other approaches, but with this one we can see that if a measure was completely unreliable its R² value would be zero

y We run the EFA and come to a solution, however now we can estimate the communalities as the sum of the squared loadings for an item across the factors

y

We now use these new estimates as communalities and rerun

y

This is done until successive iterations are essentially identical

y Convergence is achieved

y

If convergence is not obtained, another method of factor

extraction, e.g. PCA, must be utilized, or if possible, sample size

increased

(27)

Measurement error vs. sampling error

y

Measurement error is the variance not attributable to the factor an observed variable purportedly represents (1 ‐ Reliability)

y

Sampling error is the variability in estimates seen as we move from one sample to the next

y

Just like every person is different, every sample taken would be

y

Note that with successive iterations, we increase the likelihood

that we are capitalizing on the unique sampling error associated with a given dataset, thus making our results less generalizable

y

As one might expect, with larger samples (and fewer factors to

consider) we have less to worry about regarding sampling error, and so might allow for more iterations

y

Unfortunately there are no hard and fast rules regarding the

limitation of iterations, however you should be aware of the

trade off

(28)

Other stuff: More on sample size

y How big?

y Assume you’ll need lots

y From some simulation studies

y

1. 4 or more variables per factor with large structure coefficients (e.g. greater than .6) may work with even small samples

y

2. 10 variables or more per factor, loadings .4 Æ N > 150

y

3. sample size > 300

y The larger the communalites (i.e. the more reliable), the

better off you are

(29)

Other stuff: How many factors?

y Refer back to PCA notes, there are many ways to determine this

y But recall that we are doing exploratory factor analysis y As such, just as we suggested with cluster analysis, go

with the solution that makes the most sense y Also how you interpret them is, as it was with

previous analyses, entirely subjective

(30)

Other stuff: Exploratory vs. Confirmatory

y Exploratory FA

y Summarizing data by grouping correlated variables

y Investigating sets of measured variables related to theoretical constructs

y Usually done near the onset of research

y The type of FA and PCA we are talking about y Confirmatory FA

y More advanced technique

y When factor structure is known or at least theorized

y Testing generalization of factor structure to new data, etc.

y This is tested through SEM methods

(31)

Concrete example

y Beer data we did with PCA

y See appendix for code if you want to go along

y What influences a consumer’s choice behavior when shopping for beer?

y 200 consumers are asked to rate on a scale of 0‐100 how important they

consider each of seven qualities when deciding whether or not to buy the six pack:

y COST of the six pack,

y SIZE of the bottle (volume)

y Percentage of ALCOHOL in the beer

y REPUTATion of the brand

y COLOR of the beer

y AROMA of the beer

y TASTE of the beer

y First perform a PCA with varimax rotation

y In descriptives check correlation coefficients and KMO test of sphericity y Make sure to select that the number of factors to be extracted equals the

number variables (7)

y For easier reading you may want to suppress loadings less than .3 in the options dialog box

(32)

First we’ll run a PCA and compare the results

y As always first get to know your correlation matrix

y You should be aware of the simple relationships quite well, and one can already guess the factor structure that may be found

y

The first and last 3 correlate well within their group,

reputation, correlates

moderately negatively with all the others

y Our tests here indicate we’ll be okay for further analysis

Correlation Matrix

1.000 .832 .767 -.406 .018 -.046 -.064

.832 1.000 .904 -.392 .179 .098 .026

.767 .904 1.000 -.463 .072 .044 .012

-.406 -.392 -.463 1.000 -.372 -.443 -.443

.018 .179 .072 -.372 1.000 .909 .903

-.046 .098 .044 -.443 .909 1.000 .870

-.064 .026 .012 -.443 .903 .870 1.000

cost size alcohol reputat color aroma taste Correlation

cost size alcohol reputat color aroma taste

KMO and Bartlett's Test

.665

1637.869 21 .000 Kaiser-Meyer-Olkin Measure of Sampling

Adequacy.

Approx. Chi-Square df

Sig.

Bartlett's Test of Sphericity

(33)

PCA

y Recall with PCA we extract all the variance y At this point it looks

like we’ll stick with two components/factors

which account for almost 85% of the variance

Communalities

1.000 1.000

Initial Extraction

Extraction Method: Principal Component Analysis.

(34)

PCA

y We’ve got some

strong loadings here, but it’s not easily

interpretable

y Perhaps a rotation is in order

y We’ll do varimax and see what we come up with

Component Matrix^a

.550 .734 .064 .384 .011 .075 -.022

.667 .675 .235 -.085 -.078 -.137 .105

.632 .699 .066 -.283 .101 .106 -.072

-.735 -.071 .670 .009 .037 .060 .005

.760 -.576 .233 .042 -.021 -.142 -.115

.736 -.614 .080 -.037 -.218 .158 .032

.710 -.646 .032 .038 .262 .027 .077

1 2 3 4 5 6 7

Component

7 components extracted.

a.

(35)

PCA

y Just going by eigenvalues > 1, it looks like now there maybe a third factor worth considering

y Here, loadings < .3 have been suppressed

y Ah, much nicer, and perhaps we’ll go with a 3 factor interpretation y One factor related to practical

concerns (how cheaply can I get drunk?)

y Another to aesthetic concerns (is it a good beer?)

y One factor is simply reputation (will I look cool drinking it?)

Rotated Component Matrix^a

.809 .556

.972 .952

-.912 .977

.937 .939 cost

size alcohol reputat color aroma taste

1 2 3 4 5 6 7

Component

Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 7 iterations.

a.

(36)

Exploratory Factor Analysis

y Now we’ll try the EFA

y

Principal Axis factoring

y

Varimax rotation

y We’ll now be taking into

account measurement error, so the communalities will be different

y If we just take the eigenvalues greater than 1 approach, we have 2 factors accounting for 80% of the total variance

Communalities

.738 .745

.912 .914

.866 .866

.499 .385

.922 .892

.857 .896

.881 .902

Initial Extraction

Extraction Method: Principal Axis Factoring.

Total Variance Explained

3.313 47.327 47.327 3.123 44.620 44.620

2.616 37.369 84.696 2.478 35.396 80.016

.575 8.209 92.905

.240 3.427 96.332

.134 1.921 98.252

.085 1.221 99.473

.037 .527 100.000

Factor 1 2 3 4 5 6 7

Total % of Variance Cumulative % Total % of Variance Cumulative % Initial Eigenvalues Extraction Sums of Squared Loadings

(37)

EFA

y Here are our initial structure coefficients before rotation

y Similar to before, not so well interpreted y How about the rotated solution?

Factor Matrix^a

.494 .708

.644 .706

.595 .715

-.614 -.088

.785 -.526

.759 -.565

.735 -.601

1 2

Factor

2 factors extracted. 7 iterations required.

a.

(38)

EFA

y Much better once again

y But note now we have the reputation variable loading on both, and

negatively

y As this might be difficult to

incorporate into our interpretation we may just stick to those that are

loading highly

¹

y However, this is a good example of how you may end up with different results whether you do PCA or EFA

Rotated Factor Matrix^a

.862 .953 .930

-.431 -.447

.942 .946 .950 cost

size alcohol reputat color aroma taste

1 2

Factor

Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 3 iterations.

a.

(39)

EFA vs. PCA

y Again, the reason to use other methods of EFA rather than PCA is to take into account measurement error

y In psych, this would the route one would typically want to take

y Because of the lack of measurement error, physical sciences typically do PCA

y However, in many cases the interpretation will not change for the most part, more so as more variables are involved

y The communalities make up less of the total values in the correlation matrix as we add variables

y Ex. 5 variables 10 correlations, 5 communalities

y 10 variables 45 corr 10 communalities

y Some of the other EFA methods will not be viable with some datasets y Gist: in many situations you’ll be fine with either, but perhaps you

should have an initial preference for other methods besides PCA

Common factor analysis