We recruited a convenience sample of 39 adults (22 from Arizona, 17 from Washington State) with chronic back pain through recommendations from CAM pro- viders, fliers posted in the community, word of mouth, and internet advertising. Eligibility criteria included: ages 20 to 64; back pain lasting at least 3 months; and no ex- perience with at least one of the four therapies of inter- est (acupuncture, massage, chiropractic, yoga classes). Participants were not required to be starting a new ther- apy, but only to express interest in trying one of the four target therapies and to be naïve to that therapy. These inclusion criteria were useful for finding participants who would resemble patients willing to enroll in a clin- ical trial of CAM. Both the Group Health and University of Arizona Institutional Review Boards determined that these cognitive interviews were “not Human Subjects Research” because the questions were non-sensitive, hypothetical and used for questionnaire development and therefore did not require that we provide formal in- formed consent. However, prospective participants who responded were provided with a brief description of the study and screened for eligibility. Eligible participants were provided with complete study description, an explanation
16 Read more
Cognitive interviewing is increasingly used in the devel- opment and evaluation of patient education materials . Commonly referred to as “think-aloud” interview- ing, cognitive interviews allow researchers to understand how patients perceive and interpret information and to identify any potential problems in the material [14,15]. The interviewee is encouraged to “think aloud” while reading through the material presented. The interviewer “probes” by asking what is confusing, or unclear, or ask- ing for comments on specific elements in the material. The purpose of the cognitive interviews was to get feed- back from stable CAD patients in order to make revi- sions to “Making Choices©” DA content, design, and language. The “think-aloud” approach was used by the interviewer to ask the participants what they were think- ing while reading each page of the decision aid. Scripted probes were included in the interview protocol to pro- vide content standardization across interviews .
The paper aims to demonstrate the value of cognitive interviewing (CI) as a survey pretesting method in comparative education research. Although rarely used by education researchers, CI has been successfully applied in different disciplines to evaluate and improve question performance. The method assumes that observing people’s thought processes when they answer survey questions can detect response problems and point to possible solutions. To illustrate the merits of CI, we present the findings from eight cognitive interviews, which informed the development of a bilingual English/Georgian online questionnaire. The main objectives of our CI study were to a) examine cognitive validity of survey questions, b) determine semantic equivalence of the source (English) and translated (Georgian) versions of the questionnaire, and c) establish conceptual equivalence of survey measures across two cultures. We conducted two rounds of cognitive interviews, one in each language, using a combination of think-aloud and verbal probing techniques. Our analysis suggests that CI can help to identify causes of response difficulties and develop more accurate and comparable survey measures for cross-cultural education research.
40 Read more
Cognitive interviews with Australian adults aged ≥65 years revealed problems with using the self-report past 7 days form of IPAQ. Data collected from participants’ “thinking aloud” as they answered the IPAQ questions and from their responses to probing questions about their answers uncovered problems older adults may encounter when completing IPAQ. These included difficulty in understanding the intent of the questions, in recalling the information requested, and in making the calculations required to perform the task. For most participants, errors resulted in over-reporting, although for a few participants errors resulted in under-reporting. Participants experienced more difficulties with the MPA, walking and sitting questions than with the VPA questions. The question about duration of sitting time required strategies different from those used to answer questions about the other activity domains. Our findings indicate that caution is
41 Read more
The interviewing protocol consisted of two sections: a BI component and an ESS component. The interview was semistructured, consisting of the test ques- tions followed by a few general pre-scripted probe questions, for example: ‘‘Why did you answer this way?’’ Interviewers were instructed to spend 30 minutes on each section (BI or ESS), regardless of whether or not that compo- nent was completed. Additionally, interviewers were instructed to begin half of their interviews with the BI component and the other half with the ESS ques- tions. The protocol was written in English, with the BI questions posed in U.S. English and the ESS questions posed in British English. Countries conducting interviews in languages other than English were responsible for producing a translated protocol. Countries were required to produce translations using the Translation Review Adjudication Pre-test and Documentation (TRAPD) committee approach (Harkness and Schoua-Glusberg 1998). The country- specific protocols, sample composition, and other project documents can be downloaded from www.europeansocialsurvey.org. The written probes were intended to serve only as a guide for interviewers to illicit how respondents understood the question as well as how they formulated their answer; they were not intended to be used verbatim. During the interview, respondents were asked each survey item and then probed to explain their answer.
19 Read more
Because the majority of the German population currently lacks experiences with electric mobility, constructs assessing electric mobility and charging options are mostly built through public media and comparison to vehicles with an internal combustion engine. Therefore, perceptions of charging op- tions are heavily influenced by regular refuelling procedures, with which car drivers are usually familiar. The assessment of charging options for battery electric vehicle, for example, is determined by subjective categories that are susceptible to interpretation . For two different individuals a particular wording, despite being objectively the same, can have two completely different meanings. Therefore, the individuality of construct systems requires a specific method to capture how a person perceives an item. Kelly developed the Reper- tory Grid Technique to grasp subjective construct systems. Even qualitative depth or semi-structured interviews have lim- itation when exploring subconscious routines . Therefore, instead of observing characteristics and explaining people using objective observations, he asked the respondents to ex- plain their world, as they perceive it .
14 Read more
Conditional probes are likely to be of more use than other probes as they are only asked when a respondent has given a certain answer or exhibited certain behaviour, and therefore are more likely to produce information than anticipated probes which are not tailored to the respondent’s individual situation and thus may not be of relevance to them. This is not, however to suggest that other forms of probing should be abandoned in favour of conditional probes, to do so would be to miss the richness and variety of information that comes from a varied probing strategy. It may however be wise, other than in any pilot interviews undertaken, to avoid doing too much in the way of spontaneous probes to maintain the focus of the interview and keep the respondent engaged in the topic in hand.
23 Read more
Patient-reported outcome (PRO) measures must provide evidence that their development followed a rigorous process for ensuring their content validity. To this end, the collection of data is performed through qualitative interviews that allow for the elicitation of in-depth spontaneous reports of the patients ’ experiences with their condition and/or its treatment. This paper provides a review of qualitative research applied to PRO measure development. A clear definition of what is a qualitative research interview is given as well as information about the form and content of qualitative interviews required for developing PRO measures. Particular attention is paid to the description of interviewing approaches (e.g., semi-structured and in-depth interviews, individual vs. focus group interviews). Information about how to get prepared for a qualitative interview is provided with the description of how to develop discussion guides for exploratory or cognitive interviews. Interviewing patients to obtain
10 Read more
The cognitive interviews were conducted face-to- face in the field. Participants were interviewed in their home or community, which is preferable to a cognitive lab because it emulates real survey adminis- tration practices that are likely to take place when the survey is pilot tested . Cognitive interviews that deal with sensitive topics are best conducted in the field to facilitate more comfort, accuracy, and encour- age more truthful responses . The cognitive inter- view process proved to be successful among SA immigrants who completed the Urdu and English lan- guage version of the survey. This supports findings from one review that most studies employing scripted verbal and emergent probing were effective across di- verse language and cultural groups .
15 Read more
Our study has several limitations. First, each item received a minimum of 5 cognitive interviews. Although we felt this was sufficient, some authors suggest that 10 – 15 interviews are better . Because of experience on previ- ous scale development projects [5,18,19] with very similar items we felt comfortable performing fewer overall inter- views on these items. Since a minimal number of children ages 8 or 9 were required to review the items, some impor- tant findings for this age group could be missed. Secondly, as with any qualitative study, the item development team had to make judgments as to the importance of an item problem and whether revisions were necessary. We tried to adhere to the operationalization of two negative com- ments leading to revision, but all such judgments are inherently qualitative. Our team, however, was interested in identifying the most clear and important items for inclusion and carefully responded to all of the feedback from the children. Lastly, the interview questions about content validity were phrased very broadly and did not add additional information to our previous studies utiliz- ing focus groups .
10 Read more
The primary objective of this research was to evaluate the face and content validity of a newly developed care- giver burden questionnaire through cognitive interviews, as advocated by regulators and experts in COA develop- ment [18,19]. Cognitive testing of version 1.0 of CBQ- HF and subsequent item reduction and modification in version 2.0 has led to a shortened version 3.0 of the CBQ-HF. Version 3.0 of the CBQ-HF includes 26 items, measured on a 5-point Likert severity scale, assessing 1) Physical Burdens (5 items); 2) Emotional/Psychological Burdens (15 items); 3) Social Burdens (2 items); and 4) Lifestyle Burdens (4 items). Eight items were deleted from version 1.0 following the first set of cognitive inter- views and a further seven items were deleted from ver- sion 2.0 following the second set of cognitive interviews. Definitions were modified, as was the wording of an additional six items in version 2.0. Decisions to delete and modify the questionnaire content were made with expert input throughout the process to ensure that the most important conceptual domains were retained and the item wording remained conceptually clear and easy to understand for caregivers. The inclusion of care- givers with a range of educational backgrounds both in the initial development of the conceptual model  and during this validation research has ensured that the CBQ-HF is understandable to caregivers with vary- ing education and health literacy levels by using caregiver-friendly language.
12 Read more
Adjustments to the design of the questionnaire were made to facilitate data collection but primarily to improve re- spondent satisfaction. The original questionnaire starts with instructions and a definition of moral distress. The definition was excluded because, before answering, the participants in the cognitive interviews tended to try to figure out whether the nature of the disturbance in the described situations was moral distress according to the definition. Furthermore, in the original version the intro- duction informed participants to “indicate how frequently you experience each item described and how disturbing the experience is for you. If you have never experienced a particular situation, select “0” (never) for frequency.” This led to participants going up and down between the items and the instruction and therefore, in the Swedish version this text was moved from the introduction to the headings of the two columns. Thus the instruction for each column was changed from “Level of disturbance” to “How would this situation affect you?” and from “Frequency” to “How often have you experienced this situation?” Consequently, the verb in the beginning of statements, for example “pro- vide”, was changed to show activity and since the Swedish language lacks gerunds the word “to” was added, for ex- ample “to provide”, synonymous with “providing” in Eng- lish. Changing the wording from “how disturbing the situation is” to “how disturbing the situation would be” also made it easier for participants who had never experi- enced the situation to answer.
Some limitations of this study must be acknowledged. While a comprehensive randomized search strategy was used to sample patients through primary care, patients who responded and took part in a cognitive interview were self-selecting. Furthermore, those with mental LTCs were less well represented in comparison to those with physical LTCs, as were those with experience of social care use. This was a similar experience to our previous study to inform the initial questionnaire items, which included data supplementation via a secondary analysis of previously conducted interviews with people who had schizophrenia, since none was recruited directly into this study. While this research may have benefited from further input by people with mental health conditions, the authors were reassured by the involvement of people with mental health LTCs in the previous stage of the study contributing to content, some representation in the cognitive interviews, and input from stakeholders who work in the mental health sector. The next stage of research will also test the questionnaire further in this population, as well as in people with physical health conditions.
11 Read more
We applied a sampling scheme that allowed each parti- cipant to be interviewed for approximately 1 h on approximately 30 to 40 items rather than all 185 items. By this method, the vast majority of the items in the bank were reviewed by at least 5 participants (97%) meeting the target demographic characteristics outlined above (see Recruitment and Participants Section). Care- givers with asthmatic children underwent the cognitive interview on the asthma-specific item set while the other participants were randomly assigned to receive another item set. During the cognitive interviews, parti- cipants were asked to provide verbal open-ended feed- back on each item regarding response categories, time frame, item interpretation and overall impression of domain content and coverage. These questions (see Proxy Cognitive Interview Questions section) were based on prior published work  and developed by the expert item development team.
13 Read more
Because the purpose of the cognitive debriefing interviews was to confirm the relevance and comprehensibility of the SF-36v2, rather than to explore content themes, a formal saturation coding process and analysis was not required. The narrative data were coded in an Excel database using a series of ratings to systematically summarize patient feed- back. The instructions, recall period, and each item and its response choices in the survey were coded to reflect whether a potential problem in understanding was reported spontaneously by the patient, noticed by the interviewer as possibly causing confusion (noted by pauses and facial ex- pressions during the think-aloud process), or reported by a patient after the interviewer probed, specifically asking whether there were any problems understanding that part of the survey. Any potential problem was then also assigned a code as to whether it was a confirmed problem (as op- posed to a pause by the patient that did not mean there was a problem in understanding), and whether or not a change could be made to the survey to resolve the problem. This process allowed for the most important elements of the patient reports to be summarized. If a problem was re- ported, the nature of that problem was also recorded. Prior to coding, a pre-specified threshold of 25% (i.e. three or more patients reporting a consistent problem) of interviews
10 Read more
Methodology 2: Cognitive debriefing interviews Cognitive debriefing aims to verify the comprehensiveness and patient understanding of a questionnaire, as well as its clinical applicability and acceptability. A series of individual interviews (n = 5) with allergic rhinitis sufferers, who had been prescribed a nasal spray, was conducted after the first draft of the experience module was developed. These interviews examined whether the draft questionnaire was appropriate to measure the target population (who were treat- ment candidates). The identical inclusion/exclusion criteria from methodology 1 were applied to the target population.
testing period, accommodations were made for participant fatigue, scheduling conflicts, and transportation difficulties. The sequence of testing was consistent for all participants; the cognitive battery of tests was completed first, then measures of depression and postinjury stress, followed by interpersonal relatedness measures and demographic information. Data were collected in the presence of the principal investigator. Each participant was paid $10. Both the participant and a family member were asked questions about severity of brain injury to ensure that retrospective data about the duration of unconsciousness and length of posttraumatic amnesia (PTA) were accurate. Data from survivors and family members were used only when they were consistent with chart data.
Many developed countries have been concentrating lately on the quality of preschool education. Generally, what is considered one of the most important aspects of quality preschool education is quality teaching performance, which has been focused on in many research studies. Research also investigates the level of reflection in preschool teachers, especially in relation to their professional development. However, none of the research studies have, so far, focused on observing the relationship between the two phenomena. We carried out a multiple case study to investigate the relationship between the quality of reflection and the quality of teachers´ performance. Eight pre- primary teachers were video-recorded in their work, and subsequently, in-depth inter- views were conducted. During the interviews, we used stimulated recall over selected video sequences, focusing specifically on selected aims of education. The findings indicate that teachers with a higher quality of reflection reached higher quality in the selected areas of teachers’ performance. The results also showed that the teachers with a university degree reached higher quality performance. The findings can contribute to the discussion on the professional development support of preschool teachers and can initiate a discussion on the need to examine the university preparation programmes for these teachers.
21 Read more
Diaries were content analysed by two reviewers independ- ently to identify activities reported as causing difficulties, their frequency and then agreeing analyses . These ac- tivities were cross-matched with items in Part 2 of the EDAQ. New activities identified as problematic were added to a draft English language EDAQ. From the cognitive debriefing interviews, medians (IQRs) for importance re- sponses for each item were calculated, with items scoring 3 or less (ie not at all to moderately important) considered for deletion. Importance ratings were reviewed by two re- searchers, and new activities and potential items for dele- tion or merging identified. The research team finalised the English EDAQ using these patient-generated decisions.
CASPAR: Classification criteria of Psoriatic Arthritis; cDAPSA: clinical disease activity index for psoriatic arthritis; CDIs: Cognitive debriefing interviews; CPDAI: Composite psoriatic disease activity index; FDA: US food and drug administration; GRAPPA: Group for research and assessment in psoriasis and psoriatic arthritis; HAQ: Health assessment questionnaire; MDA: Minimal disease activity; OMEARCT: Outcome measures in rheumatology; PASI: psoriasis area and severity index; PGA: Patient global disease activity; PhGA: Physical global assessment; PROs: patient-reported outcomes; PsA: Psoriatic arthritis; PsAQoL: Psoriatic arthritis quality of life scale; QoL: Quality of life; SF-36: Medical outcome short-form 36; UK: United Kingdom