Assess Validity and Reliability

Principles of Assessing Validity and Reliability

Churchill (1979) calls for the final data set to be checked once more for reliability, to ensure confidence in the developed and refined measures, which form the basis for hypothesis testing.

These measures must be reliable and valid for researchers to have any confidence in the outputs of hypothesis testing. Churchill suggests checking Cronbach’s alpha again, arguing against the use test-retest reliability in addition to the internal consistency check. Test-retest was used to ensure forces external to the measures are not likely to have a substantial impact on supposedly

187

stable constructs (like traits or attitudes, for example), but is problematic because it relies upon respondents’ memories and has subsequently fallen from favour.

Another procedure which is essential, before hypotheses are tested, is to check the construct to determine the number of dimensions which it contains. Churchill (1979) recommends using factor analysis to identify the number and composition of dimensions in terms of items. He highlights the importance of doing this after the metric purification stage, as ‘garbage items’

which don’t share a common core tend to produce more dimensions than are identifiable conceptually. As such, factor analysis has been saved for the time when there is sufficient sample size, and the measure has been purified and checked initially for reliability. In the following sections for each study, first principal component analyses will be presented for each construct, then these scales (with key items identified) be checked for reliability.

Principal Component Analysis

Factor analysis is an overarching term used for a family of analytical techniques for identifying ways to reduce data into fewer components. Factor analysis tools are described by Churchill (1979), MacKenzie et al (2011) and many others (Pallant 2010) as useful in the development and appraisal of scale measures. Using factor analysis, it is possible to take multiple items, and find a way to group them according to common patterns emerging from data. In this way, fewer distinct dimensions are identified, which enables the simplification of subsequent hypothesis testing. Rather than testing many potentially unreliable and invalid items in hypothesis tests, fewer robust and statistically valid variables can be tested. This lends further credibility to the validity of hypothesis testing, i.e. that these tests will contain variables which are worthy of testing, and therefore findings will have greater validity also.

Factor analysis is most generally broken down into two approaches- exploratory and

confirmatory, and within each of these there are different types. Confirmatory factor analysis is a more complex set of techniques used to confirm the structure of constructs using structural equation modelling. Exploratory factor analysis techniques are used more readily to examine how items are interrelated, giving indication of suitable ways to group a set of items into fewer

‘latent’ variables, which represent the underlying dimensions of a construct. Literature is sometimes conflicting in describing factor analysis and principal component analysis (PCA).

Principal component analysis takes the original observed items, and uses the variance between

188

all of the items to transform them into fewer aggregate ‘latent’ variables, so at face value, appears to be the same as factor analysis. While some describe PCA as just one of a family of technique of the overarching factor analysis (Pallant 2010), others describe factor analysis as quite different from PCA, with factors estimated from mathematical models with only shared variance considered (Tabachnick, Fidell et al. 2001). Principal component analysis is argued to be most useful if the end goal is the reduction of data into its component parts, providing ‘an empirical summary of the data set’ (Stevens 1996) p363, and has the benefit of avoiding

potentially problematic ‘factor indeterminacy’ (Tabachnick & Fidell et al 2001 p61), and so shall be used to identify dimensions and dimensional composition.

A set of procedures for PCA outlined by Pallant (2010) will be followed to first explore the data, before determining which items comprise which dimensions. While it would be ideal to identify dimensions commensurate with theoretical dimensions, this may not always be possible.

Pallant (2010) suggests approaching PCA (considered in this investigation in as one of the techniques included in the factor analysis family) in three steps. These steps were followed to factor analyse each of the constructs considered across the two studies.

The first step in PCA entails assessing the data to determine its suitability for factor analysis. A key determinant of PCA suitability is the sample size. Small data sets do not yield factors that generalise well (Pallant 2010 p183). Larger samples are usually more desirable. Recommended sample size for factor analysis varies considerably. Malhotra & Birks (2007) concede that in marketing research, it is often the case that sample sizes may be reasonably small. Along with others, they suggest that it is not sample size in general that is of concern, rather, that the sample must have at least 5 respondents per item. So a 20 item construct would require a sample of at least 100. The largest construct in study 1 is 35 items in length, suggesting a sample of 175 is required for factor analysis. The study 1 sample of 292 is therefore adequate for PCA. For study 2, the largest construct is 20 items in length, therefore requiring a sample of 100. The final sample of 177 for study 2 is therefore adequate for PCA.

The other key determinant of suitability of factor analysis is the strength of item

multicollinearity. One of the first things PCA provides is a correlation matrix of items within the construct of interest. It is suggested that if few correlations between items are over .3, that factor analysis is not relevant (Tabachnick & Fidell 2001). This is essentially because principal

189

component analysis is looking to determine ‘latent’ variables with the greatest intra-item correlations, with different latent variables showing little correlation. Other tests can be run to determine whether data should be considered as suitable for PCA- these include Bartlett’s test of sphericity and the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (Pallant 2010), with a significant Bartlett’s result and a KMO value of above .6 recommended (Kaiser 1970).

Before PCA is carried out for constructs across the two studies, correlation matrices, Bartlett’s test of sphericity and the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy will be first examined and summarised to confirm suitability of PCA.

The second step in factor analysis is to determine, for a construct, the most appropriate number of factors (latent variables) within that construct, and can be done using a number of techniques, including an initial PCA. At its heart, PCA is trying to achieve a balance between explaining as much variance in the data as it can, and reduce to number of factors to the fewest possible. A certain number of dimensions for a particular construct are often expected based on preceding theoretical discussion of a construct. Indeed, for these studies, items were generated to ensure multiple dimensions for a particular construct would be fully captured in the data collected, with multiple items for each dimension. However, it is not always the case when new measures are developed, in different contexts from previous studies, that the theoretically defined dimensions will also be captured in new studies. Factor analysis is an important tool to examine patterns in the data to determine whether the number and composition of dimensions (factors) in the data collected match up with the number of dimensions identified in theories developed from

previous studies. As one of the goals of PCA is to reduce the data to the most simple solution of fewest factors, it is often likely that the dimensions suggested from PCA are fewer in number and potentially broader in scope than those suggested in the theory. A combination of

techniques within PCA will be considered to identify the optimal number of components for a construct. Pallant (2010) suggests considering Eigenvalue rule (Kaiser’s criterion), examining the scree plot and parallel analysis. The Eigenvalue rule suggests that only components with an eigenvalue (which relates to the extent of variance explained) of 1 or above be kept. The scree test is a visual aid to interpreting the variance of each component explained, by plotting the eigenvalues for each factor. Parallel analysis takes key information relating to the data collected (number of variables and subjects) and creates random data from these numbers, and using multiple replications, extracts Eigenvalues. The Eigenvalues from the real data can then be compared with those constructed from the parallel analysis. When the real Eigenvalue exceeds the ‘criterion value’ from parallel analysis, it is accepted that this construct is acceptable. These,

190

as well as visual aids of multicollinearity shall be considered in determining the optimal number of components.

The final step, once the appropriate number of factors has been determined, is to run the final PCA and interpret it. This is described by Pallant (2010) as ‘Factor rotation and interpretation’.

(p184). Factor rotation will aid the interpretation of the PCA components. Data will first be examined using factor analysis, before checking the output factor scales for reliability. Checking reliability of the theoretically defined scales before factor analysis may be a fruitless exercise if factor analysis identifies different or fewer scales than anticipated.

Assessing Validity and Reliability for Study 1

In SPSS, the situational construct for study 1, with items built around based on Belk’s (1975) taxonomy, was factor analysed. Using principal components analysis, the 35 items were scrutinised to determine whether they factor loaded on the five dimensions suggested by Belk (1975). Data was first assessed to determine whether factor analysis was suitable (see appendix I). Barlett’s Test of Sphericity (Bartlett 1954) reached statistical significance (p <.001), and Kaiser-Meyer-Olkin (KMO) reached an acceptable score of .806 (Pallant 2010) supporting the factorability suggested by the correlation table, which revealed multicollinearity.

Initial principal components analysis suggested that 11 factors were present with eigenvalues above 1, with factors explaining between 2.9% and 17.476% of variance (cumulatively 58.67%

of variance). 11 items are far more than anticipated, since the items were developed based on Belk’s five situational dimensions. Also, statistical methodologists suggest that using procedures such as the ‘greater-than-one rule’ are flawed (O’Connor 2000), and that tests like parallel analysis and Velicer’s minimum average partial (MAP) test are more robust and more widely recommended. Inspection of the scree plot (in appendix I) suggested a possible break after 4 factors. Catell’s (1966) scree test suggests that 4 components be retained for further testing.

Parallel Analysis, conducted using Monte Carlo PCA for Parallel Analysis, suggested that 4 factors may be more appropriate (please see table 3.9).

191

Table 3.9: Comparison of Eigenvalues from principal components analysis (PCA) & the corresponding criterion values obtained from parallel analysis for Situational construct Component Number Actual Eigenvalue

from PCA

Criterion value from parallel analysis

Decision

1 5.172 1.726 Accept

2 2.251 1.629 Accept

3 1.828 1.562 Accept

4 1.524 1.503 Accept

Only four components had eigenvalues greater than the corresponding values in the randomly generated data matrix (based on 35 variables, and a sample of 292 respondents). Therefore, principal component analysis will be conducted with four components, with varimax rotation also conducted to aid interpretation (see table 3.10), in line with earlier studies (Hackett and Foxall 1999; Leek, Maddock et al. 2000). For the table of unrotated loadings, please see appendix I.

192

Table 3.10 Four factor Varimax Rotated Component Matrix^a

Component

1 Social Temporal Physical

Do you find shopping exciting? .721

Do you enjoy shopping for its own sake, not just for the items you purchase? .698

Do you go shopping to escape ordinary life? .587

Compared to other activities, does your time spend shopping feel truly enjoyable? .618 Do you find you shop mainly because you want to, and not because you have to? .506

Do you find your mood improves with each purchase you make? .398 .335

Have you ever gone shopping when sad or depressed, to cheer yourself up? .671

Do you ever shop to put yourself in a better mood? .649 .354

Do you tend to make a lot more purchase decisions immediately after payday? .320

Is it important to you that your friends like the products you buy? .590

Would you be put off buying a product you really liked if your friends did not like it? .583

Do you try to keep up with current fashions and trends? .519

Have you ever found certain products more desirable when someone you admired used/endorsed it? .544 Do your tastes (e.g. in clothes, movies, music, etc.) change to match those around you? .571

Do you ever watch others to keep up with changes in fashion? .525

Do you try to buy products that are similar to those your friends buy? .601 Do you believe a crowded shopping centre must be a good shopping centre? .363 Do you ever get excited when seasonal decorations appear in shopping centres? .304

Would you make an unnecessary purchase, just to cheer yourself up? .402 .564

Do you tend to make more rash purchase decisions when excited? .566

Do you tend to make more rash purchase decisions when bored? .587

Do you find you start to make more bad decisions when you are hungry? .389

Do you find a bit of time pressure can push you to make important purchase decisions? .305 .598 Do you find yourself buying more food when grocery shopping, if you have not yet eaten? .399 Do you find you make more split-second purchase decisions when pushed for time? .660 Do you find yourself spending more on yourself in the run-up to a seasonal event? .333 .340

Do you ever find yourself browsing, even when you have no intention to buy? .579

Do you find the type of things you buy yourself change depending on time of year? .464

Do you enjoy getting into the spirit of holidays? .473

Do you still browse through shops even when you do not have money? .446

Have you ever left a store after noticing a bad smell? .372

Have you ever stayed a long time in a store that plays good background music? .377

Have you ever left a store that displays items in a haphazard or disorganised way?

Have you ever gone inside a store to warm up on a cold day?

Have you ever left a store because you felt the music was too loud?

Percentage of Variance Explained Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization.

a. Rotation converged in 7 iterations.

193

Interpretation of the components suggest that component 1(8 items) reads somewhat like a mix of task affect and antecedent states, component 2 (7 items) describes social surroundings, component 3 (8 items) relates to temporal perspectives and component 4 (6 items) physical surroundings. In light of a four factor solution, which is comparable with Foxall’s four aspects of the situation, component 1 may in fact be interpreted as regulatory forces. Though not originally conceived of to measure regulatory forces, several items relate to forces imposed on consumers in the situation by personal, social, and situational regulations, such as the influence of personal feelings towards shopping, ability to engage in shopping due to monetary

constraints, etc. In some definitions, regulatory forces are the rules concerning shopping (Oliveira-Castro, Foxall et al. 2005), self or other rules that specify contingencies (Foxall and Yani-de-Soriano 2005).

Disappointingly, when the situational scales are tested for reliability with the key contained items displayed in table 3.10 above, not all scales pass reliability analysis at the anticipated level. The regulatory forces scale is acceptable as having good reliability, while social surroundings and temporal perspective generally round up to provide acceptable reliability. Physical surroundings however, only achieve a Cronbach’s alpha of .549, which George & Mallery (2003) suggest is poor reliability. Later discussion of ‘latent’ variable calculation will discuss how this may be overcome.

Assessing Reliability and Validity for Study 2

Because study 2 considers several different constructs, most of which were developed from scratch, factor analysis was conducted for each of these. The following sections shall consider the factor analysis for each of these constructs in turn. First, the ‘Situational’ construct will be considered, then the ‘Reinforcement construct’, the ‘Approach-avoidance construct’, the

‘Learning History’ construct, and finally, ‘Emotional Response’ construct. Factor analysis should determine whether the dimensions identified in the extant theory are recognisable in the final data collected for study 2, or whether constructs applied in the context of shopping centre choice have different dimensions. Full figures for reliability tests for study 2 can be found in appendix J.

194

Situational Construct Principal Components Analysis (PCA)

Firstly, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity were tested to help determine whether factor analysis is appropriate for situational variables. Kaiser-Meyer-Olkin returned a value of .887, which is greater than the recommended value of .6 (Kaiser 1970) and Bartlett’s test of sphericity was significant, so both tests suggest the situational items are suitable for factor analysis.

Next, the correlations of situational items were examined to ensure sufficient cross correlations exist. Figure 3.4 below shows a visual representation of how all of the situational items correlate with each other, with light areas denoting the largest correlation coefficients, and dark areas the smallest correlations. Looking at the correlation table itself, many items had reasonable

correlations of over .3 (Pallant 2010).

Initial PCA revealed four components with eigenvalues over 1, explaining 34.9%, 22.1%, 7.2%

and 5.6% respectively, to a total of 69.3% variance. However, with further examination, the scree plot (Cattell 1966) suggested a clear break after just two components. Interpretation of parallel analysis suggested that either two or three components should be considered. In the parallel analysis summarised in table 3.11, it is suggested that three components be retained for PCA, with the eigenvalues of three components exceeding the criterion values from the

randomly generated data with comparable parameters (20 variables x 177 respondents). Further repeats of parallel analysis sometimes provide similar results, while others show eigenvalues for only two components exceeding parallel criterion values. Given the potentially conflicting results from parallel and scree plots, the (most likely) two factor and also three factor solutions shall be explored.

Table 3.11: Comparison of Eigenvalues from principal components analysis (PCA) and the corresponding criterion values obtained from parallel analysis for Situational construct Component Number Actual Eigenvalue

from PCA

Criterion value from parallel analysis

Decision

1 6.878 1.6611 Accept

2 4.431 1.5314 Accept

3 1.433 1.4318 Accept

4 1.113 1.3557 Reject

As initial exploratory factor analysis for the situational construct compared with parallel analysis suggests, only two, at most three dimensions can be shown within the data collected, rather than

195

four factors for situational scales derived from theory. Rather than having distinct scale measures for physical surroundings, social surroundings, temporal perspective and regulatory forces, two new constructs are found for the data. It may be that the scales look completely different from those identified in the theory, or it may be that items across multiple theoretically identified dimensions in fact comprise just one aggregate dimension.

Rather interestingly, it is the examination of the normalised covariance matrix that offers the most definitive insight into whether there are two or three correlations. Figure 3.4 below shows the groupings of strong correlation coefficients. It is clear from figure 3.43 that there are two distinct groupings of variables (though again, it does highlight some potential for a third component), with variables within each group sharing strong correlations with each other, and very weak correlations with variables in the other grouping.

Figure 3.4 Covariance for Behaviour Setting

This figure provides initial insight into the composition of the components likely to be

uncovered in the PCA, though at this stage does not necessarily give clear indication of loadings.

It appears that physical surroundings and social surroundings variables correlate strongly with each other, suggesting that component 1 is instead an overarching measure of the surroundings.

A temporal and a regulatory variable also appear to relate quite strongly here. The regulatory item (Regulatory 2) asks about the visibility of security personnel, so it is not surprising that is correlates strongly with the strongly social and sensory characteristics of variables measuring

A temporal and a regulatory variable also appear to relate quite strongly here. The regulatory item (Regulatory 2) asks about the visibility of security personnel, so it is not surprising that is correlates strongly with the strongly social and sensory characteristics of variables measuring

In document Shopping Centre Choice: A Behavioural Perspective (Page 187-200)