Streiner and Norman (2008) provide a clear chronological guide for devising a reliable and valid measure that was adhered to. This ensured that sound methodology was used to develop the measurement tool. For the construction of a new measure several crucial stages were undertaken; generate an item pool, determination of the measure format, experts to review the measure, administration of the items and evaluate, amend based on this and pilot again (DeVellis, 2003).
The measure was for a new topic area therefore the generation of items derived from a number of sources, including a comprehensive literature review, focus groups with the key informants and relevant experts in the field (Streiner and Norman, 2008; Onwveglouzie, Bustamante and Nelson, 2010). The initial stage reviewed the current evidence-base to support the development of the instrument (see chapter three). It was evident there were a number of gaps in the literature, therefore focus groups were used to inform the construction of the scale items. The use of focus groups provided a dynamic social interaction which allowed the voices of the key informants to be heard (Onwveglouzie, Bustamante and Nelson, 2010). Although this is typically overlooked in scale development it wasa useful strategy that informed the phrasing of the items increasing the validity of the questionnaire. Key informants were involved throughout the initial construction of the measure from their words informing
109 the phrasing of items to providing feedback on the items to ensure clarity, relevance and length to enhance the validity of the measure (Onwveglouzie, Bustamante and Nelson, 2010).It was therefore important the voices of the target audience were included (Onwveglouzie, Bustamante and Nelson, 2010). A careful planned approach to generate items was important in the initial stage of the scale development as it improved the internal validity of the measure.
The items generated were checked with a group of intended respondents in addition to experts in the field to establish context validity. This stage in the generation of the item pool checked the items thus far were not complex, ambiguous, double- barrelled, leading or contain jargon and that any ambiguous items were either eliminated or rewritten (Streiner and Norman, 2010). The rule of thumb was that the reading age of a questionnaire should be that of a 12 year old child (Streiner and Norman, 2010).
The measure that was to be constructed needed to be able to compare between groups of participants and across different outreach activities with ease. Therefore a self-report mode for the questionnaire was most suitable. In using this approach respondents can complete items without assistance and enable practitioners to make numerical comparisons easily. To easily compute numerical data for comparison the response scale used was a Likert scale, in which each item score contributes to computing an overall score of intention, so a higher score equated to a higher level of intention to progress to HE.
To answer an item involves respondents using a number of cognitive processes, which includes recalling the relevant behaviour, attitude or belief, map the answer onto the response scale and edit their answer (Streiner and Norman, 2010). Therefore questions should not be over demanding. To identify any areas of difficulty when answering questions, cognitive interviewing is recommended to explore the mental processes respondents use to answer the items, thus to develop this new measure cognitive interviewing was conducted (Streiner and Norman, 2010). The approach of cognitive interviewing overcomes the limitation of the nomothetic approach typically used in scale development, a method that does not include the intended respondents to test the assumption that items developed are consistent with the intended audiences’ interpretation of the items.
110 A psychometric approach was used to select the items for the scale; a unique approach to the development of a scale to use in the evaluation of outreach activities. Factor analysis is a statistical method that organises items into factors based on their relationship with each other. The use of this approach ensures the scale is valid, as all the items are measuring the same construct. The technique ensuredthe scale was homogenous. To determine the final item set for the scale required human intuition as well as statistical analysis, which is the common approach when using a psychometric approach.
Pre-testing of a measure before the inclusion in the main evaluation is crucial for a new measure. The measure was administrated as it would of been for the intended respondents to highlight any potential pitfalls or misinterpretations. The length of the scale was also considered during the process. The time taken to respond to the questionnaire was noted to ensure the measure was no longer than ten to fifiteen minutes as this can be problematic due to reducing motivation levels (Worthington and Whittaker, 2006). The scale constructed was quick and easy to administer, it was a short scale to reduce burden and to motivate respondents (Streiner and Norman, 2008; DeVellis, 1991).
The approach taken to develop the measure was unique compared to practices used previously in the area of designing questionnaires for the evaluation of outreach activities. The use of the processes and approaches in this chapter ensured the measure was a reliable and valid for the evaluation of ThinkSmart.
8.4.1METHOD
8.4.1.1 Aim
The study developed and validated a self-report measure of intention to progress to HE to use in the evaluation of ThinkSmart (chapter eight).
8.4.1.2 Design
A three phase approach to develop the measure was based on that outlined by Streiner and Norman (2008) to ensure a sound methodology approach was used to establish a reliable and valid measure of intention to engage with HE. The first stage was concerned with item construction, which was the generation of the item pool, the refinement of the item pool and the use of cognitive interviewing to explore the response process. The second stage involved testing the psychometric properties of
111 the measure developed with the use of factor analysis.The reliability of the scale was explored by using Cronbach's alpha to measure internal consistency and test-retest reliability (external reliability). The final stage piloted the measure with the intended audience, a sample of participants who had engaged with ThinkSmart to determine the measures suitability.
8.4.1.3 Sample
At each stage of the scale construction an opportunistic sampling method was used, both males and females aged 11 to 18 years were recruited. The majority of the participants were of White British ethnicity due to the population characteristics of the two counties Herefordshire and Worcestershire. The sample size for each stage is to be discussed in each section of the scale construction.