Cognitive Interviews

In line with the old adage “communication isn’t about what you say, it’s about what they hear”, within the context of developing questionnaires it is important to not only determine the relevance of each item, but also to ensure that each item is

successfully interpreted in the manner in which it was originally intended (Conrad & Blair, 1996). If items are misinterpreted or cause confusion, there is an increased risk of non-response, non-completion or inaccurate response; all of which can negatively impact upon subsequent statistical analyses and generalisation of findings (Drennan, 2003). In order to minimise the likelihood of such occurrences, front end processes should not only address issues around item justification (as in the case of the expert panels), but also examine the respondents’ cognitive processes to ensure appropriate levels of comprehension.

As one such front-end process, cognitive interviewing is designed to uncover respondents’ thought processes when answering a survey question, in a bid to identify problems within the survey and generate potential solutions (Willis, 2005). Despite it being acknowledged that there is still an air of ambiguity around cognitive interview best practice (Drennan, 2003), researchers suggest using a combination of techniques to determine the cognitive processes involved and to identify potential response problems. Such techniques include think-aloud procedures, probes, behavioural observations, and paraphrasing questionnaire items (Beatty & Willis, 2007; Campanelli, 2008; Conrad & Blair, 1996; Willis, 2005; Willis, Schechter, & Whitaker, 1999), and have been utilised to good effect in recent talent development literature (e.g., MacNamara & Collins, 2011)

As part of a think-aloud procedure, participants are required to vocalise their thoughts and feelings when answering a survey question. These verbal reports are subsequently understood to demonstrate the participants’ cognitive processes, and can occur either concurrently (i.e., at the time of answering) or retrospectively, often as part of a debrief (Campanelli, 2008). Unlike retrospective think-aloud techniques,

concurrent think-aloud procedures are not susceptible to recall issues, and therefore may potentially offer greater validity. However, concurrent thinking aloud is not a typical everyday activity; respondents can frequently require neutral probes to encourage them to vocalise their thoughts effectively, potentially impacting upon the flow of the

questionnaire. Conversely, retrospective thinking aloud is recognised as less

burdensome for the respondent (Beatty & Willis, 2007; Campanelli, 2008; Mehrotra, 2007).

A potential issue with using think aloud protocols in the development of the PCDEQ2 lies within its target demographic; namely developing athletes. Young people are often recognised to have difficulties in articulating their thoughts and feelings, and may perceive words differently to adults (Drennan, 2003), thus making the think-aloud protocol problematic. In recognition of such issues, the deployment of retrospective verbal probing can be used. Campanelli (2008) suggest that in order to ascertain the cognitive processes behind the responses, probes should address issues around

comprehension, recall, judgement and response. Requesting respondents to paraphrase questions in their own words, asking them to define meanings of key words within questions, explain their responses and identifying areas that they found problematic are all ways of eliciting a respondent’s understanding of a question (Czaja, 1998; Drennan, 2003). As well as such pre-scripted probes, unscripted probes offer an opportunity to explore unexpected responses (Conrad & Blair, 1996). Behavioural observations can often act as a prompt for such probes, with behaviours such as skipping questions,

changes in facial expression, hesitation in answering, and changing answers all offering further lines of enquiry (Conrad & Blair, 1996; Drennan, 2003; Willis, 2005).

In analysing the data generated through cognitive interviewing, Conrad and Blair (1996) propose that response problems to questionnaire items can be categorised in to five different types: lexical, temporal, logical, computational, and

omission/inclusion issues; and any of these issues can occur at each stage of the response process, namely understanding, task performance and response formatting.

Lexical problems are those based around the participant not knowing the meaning of a

word or how to use it correctly. This extends to idioms (e.g., “elbow grease”) and unfamiliar word pairings (e.g., “monkey tennis”), and can occur if, for example, the respondent is unsure as to a particular meaning of a category label. This is particularly pertinent to this study, in ensuring that the items are not beyond the understanding or vocabulary of the target demographic (Dillman, 2007). As a special case of lexical problem, temporal problems involve respondents struggling to understand or operationalise terms relating to time. To illustrate such an issue, Conrad and Blair (1996) offer the example of the phrase “in the last year”, a phrase that could be interpreted as meaning the last twelve months or the last calendar year. Inclusion /

exclusion problems arise when it cannot be determined if certain concepts are to be

considered in relation to the word in question, and can often lead to respondents providing multiple or incorrect responses (Drennan, 2003). Logical problems arise in relation to presuppositions, contradictions, and connecting words such as “and” or “or”, and can lead to respondents having to answer two (or more) questions in one (Conrad & Blair, 1996; Drennan, 2003). Finally, computational problems often involve memory, mental arithmetic and language processing, but as all of the issues discussed above in one way or another, this category serves as a catch-all for all issues that don’t fit in to the previous four (Conrad & Blair, 1996).

7.2.3.1 Participants. Individual cognitive interviews were conducted with six

purposively sampled developing athletes from football and rugby union academies (n = 4 and n = 2 respectively), with two athletes aged 14-16 years, two aged 17-18 years, and two aged 19-20 years old; thus representing the intended demographic of the

questionnaire, and representative of the environments from which the items were developed. Although such limited numbers cannot guarantee the comprehensive identification of potential problems relating to questionnaire design (Beatty & Willis, 2007; Blair, Conrad, Ackermann, & Claxton, 2006), given the lack of consensus around appropriate sample size in cognitive interviewing (Conrad & Blair, 1996) and that the cognitive interview process is in essence a qualitative exercise and not a quantitative one (i.e., logical and structural problems persist independent of sample size; Willis, 2005), the use of a small, high quality sample that reflected the sub-populations of the final questionnaire’s target demographic was deemed appropriate.

7.2.3.2 Procedure. The randomised items were split across 16 sections and

administered to participants on a section-by-section basis, with breaks between sections in order to minimise both participant and investigator fatigue. In line with the

recommendations of Willis and colleagues (Beatty & Willis, 2007; Willis, 2005; Willis et al., 1999), a combination of proactive and reactive verbal probing was utilised, including think-aloud protocols, reinterpretations, and observations (see Appendix G). Following the completion of each section, respondents were also invited to comment upon their answers and underpinning rationales. Observations noted included

hesitations, changing answers, skipping questions and behavioural indicators (e.g., head scratching and fidgeting), and were recorded along with the probe responses.

7.2.3.4 Results. Following the completion of the cognitive interview process,

comments for each item were collated and categorised according to Conrad and Blair’s (1996) taxonomy. This process resulted in the amendment of 9 items (items 14, 26, 28,

38, 77, 91, 93, 117, 134) due to lexical problems (e.g., not knowing what was meant by “iron self-discipline”) and 1 item (item 76) due to temporal issues (“I am happy with how my body looks” failed to recognise the transformational nature of talent

development). No items were removed at this point in the process.