4. Methods
4.6. Pilot
I tested the utility of the core outcomes when used in a self-report scale by conducting a large scale pilot. I tested the 156 item self-report tool on over 400 healthcare professionals to assess the usefulness of each item. If an items produced a large ceiling effect, for
example all participants strongly agreed with it, then it does not demonstrate variability. Or if an item had no relation to other items then it would also not be a useful measure.
The pilot served two purposes firstly to generate data to allow me to remove items that did not have optimal psychometric properties. This was the final stage in the reduction of outcomes: beginning with the thousands of the outcomes from the systematic review that were slowly funnelled into a 40-item self-assessment tool, as depicted in figure 13. The secondary purpose was to generate some preliminary data on how the tool works, for example the scores that participants got, the variability between different groups, the effect and interactions of any variables.
4.6.1. Rationale for using a statistical data reduction technique
Principal Component Analysis (PCA) is a dimension-reduction tool that can be used to reduce a large set of items to a small set that still contains most of the information in the large set (247). Principal component analysis (PCA) is a mathematical procedure that transforms a large number of (possibly) correlated items into a (smaller) number of uncorrelated variables called principal components (247). In relation to item response theory (IRT) (described in Chapter 3) it looks for items that seem to represent a latent variable and groups them together, so rather than having 100 items that measure lots of things, it creates a smaller groups items, that measure a few different things. The first principal component accounts for as much of the variability in the data as possible, and103
each succeeding component accounts for as much of the remaining variability as possible (248).
Principal components analysis is similar to another multivariate procedure called Factor Analysis (FA) (247). Both are data reduction techniques, they capture the variance in variables within a smaller set of items. Using an oversimplified example, if I wanted to measures a participant’s ability to solve calculations I could present 50 calculation questions and use total score as a measure of calculation ability. However, 50 questions would take considerable time and effort on behalf of the participants, PCA aims to reduce this. It uses computer modelling to search for the questions that convey the most about each individual’s ability: the items with the best psychometric properties. For example, if almost everybody answers 20 questions correctly and 20 incorrectly, these 40 questions convey little about each individual’s ability compared to the group (because everybody answered the same). However, in the 10 remaining questions the responses are spread: a few people answer correctly, a few incorrectly and a few show correct planning but arrive at the wrong answer. Presenting only these 10 questions to a new group of people would convey similar information about the individual’s ability (as the original 50). It would allow me to make greater comparisons between individuals/groups, but reduce the
cognitive load from 50 to 10. If I administered all 50 of the original questions there might a small amount of extra information about each individual, however because the responses were largely the same, it’s a lot of extra work for very little additional information.
Secondly, both PCA and FA are methods aligned to item response theory. Both are usually run in statistical software using the same procedure, and both methods produce similar outputs. Both methods involve similar steps, namely- extraction, interpretation, rotation, selecting the number of factors or components. Yet, despite the multiple similarities, there is a fundamental difference between them: PCA is a linear combination of variables; Factor Analysis is a measurement model of a latent variable. What this means is that PCA’s approach to data reduction is to create one or more index variables from a larger set of measured variables. It does this using a linear combination (essentially a weighted average) of a set of variables. The created index variables are called components. The main rationale for PCA is to understand how to do this in an optimal way: the optimal number of components, the optimal choice of measured variables for each component, and the optimal weights. Factor Analysis approaches data reduction in a fundamentally
104
cannot be directly measured with a single variable (for example try to measure intelligence or social anxiety with just one question). Instead, it is seen through the relationships it causes in a set of variables. Comparing the two methods visually, one main conceptual difference is the way in which the arrows point. Factor analysis, see figure 14, is a model with the underlying assumption that a latent trait causes a person to answer in a particular way. An answer to any given questionnaire item is a sum of to the influence of the latent trait and the variance that is unexplained by the latent trait. Whereas principle component analysis, whilst still related to latent trait theory aims to find the items that best represent a component (which is a similar theoretical entity to a latent trait). In both models the answers are weighted (depicted by the ‘w’ in the figure) so some items are more representative than others. Put into perspective of my research using another
oversimplified example, factor analysis would look to model how much a latent trait such as clinical confidence affects individual’s answers to a set of questions. Whereas principle component analysis would look to find a model that interprets which questions best measures clinical confidence.
105
Figure 15: A visual depiction of Principle Component Analysis
4.6.2. Rationale for choosing principal component analysis
Advocates report that it is frequently possible to reduce the number of itemsconsiderably while still retaining much of the information in the original data set, when using PCA. PCA is probably the best known and most widely used psychometric reduction technique (247). I wanted to reduce the number of items assessed, whilst ensuring I retained the majority of the information that described health professional learning on international placements. I chose PCA over FA as it lends itself to Computer Adaptive Testing; which was a future direction I thought the self-assessment output could go (249). It provided me with more opportunities about where the output could go beyond PhD and provided more opportunities for the research to have impact. It would mean that individuals could answer questions on a computer and the next question answered would be generated based on the answer to the previous question, taking into account the weightings of each question (249).
The principle component analysis was performed by a colleague with expert statistical knowledge and training (see acknowledgments). Hence, this thesis does not describe the
106
statistical and mathematical basis of principle component analysis, but rather the conceptual reasons for choosing this method. For a detailed description of the mathematical principles that underpin PCA please see Richardson (248).
4.6.3. Multidimensional item response theory model
A multidimensional item response theory (MIRT) model was created based on the results of the best iteration of the principal component analysis. This is a model that shows how the items in the self-assessment relate to the latent traits and the correlational relationships between the traits and items. The multidimensional model shows which items are used to assess which latent traits, figure 16 is an example model, the actual model can be seen in chapter 7.
Figure 16: A visual depiction of Multidimensional Item Response Theory