Cognitive interviewing is a method of collecting detailed information regarding the clarity, coverage, and importance of survey items and overall flow of the survey prior to finalizing and administering the survey. In cognitive interviewing, interviewees are asked to “think out loud” while they complete the survey and a researcher records their reactions to survey items, difficulties they may have, and thought processes in coming up with responses to survey items. Below are two documents used for the Vermont Workforce Survey cognitive interviews. The first is the codebook used to systematically record findings from the interviews to ensure consistent interpretation of interview data. The second is the protocol used to guide researchers in conducting the interviews.
VT Early Childhood and Afterschool Workforce Survey Cognitive Interviews
Code sheet and Analysis Procedures March 2015
Cognitive Interview Analysis Procedures
Through qualitative analysis of cognitive interview data, the research team seeks to: (1) explore respondents’ overall perceptions of the survey, (2) identify and revise items that lack clarity, relevance, flow or coverage, and (3) limit nonresponse bias and nonsampling error in the large-scale
administrations of the final survey. This code sheet is intended to ensure the systematic and consistent coding of participants’ responses.
To code and then analyze the interview data, the team will use a coding hierarchy that identifies and classifies participants’ responses to standardized probes (i.e., anticipated and conditional probes) and follow-up questions. Data will be assigned codes (described more specifically below) based on the following coding hierarchy: (1) overall or item specific, (2) topical area of the feedback (e.g. relevance, clarity, etc.), and (3) classification as support, issue/question, or suggestion for improvement. Portions of interview data may be doubled-coded when necessary to ensure the proper capturing of depth and breadth of respondents’ feedback.
Data that a researcher is unsure of how to properly code will be highlighted for discussion between the two research team members. If research team members find portions of the interview data that do not seem to be covered by the pre-developed coding hierarchy, data will be marked as “other” and be reviewed by the research coding team. The team will decide which code fits the data or determine if another code should be created to cover the information provided. Researchers responsible for coding will meet throughout the process to ensure consistency, compare coded data, and discuss potential questions.
A-42
Coding by Hand with the Excel Workbook _________________________________________________ Coding will be done by hand with the assistance of this code sheet. Codes will be entered into an Excel Workbook that will be used to review results and suggested changes and discuss the results of the cognitive interviewing process with the Vermont team.
List of Pre-determined Codes _________________________________________________ To aid in completing the coding and filling out the Excel workbook, the following is a list of
predetermined codes. Following this list are detailed descriptions regarding the codes and examples of what these might look like in practice.
Overall Survey Codes
Survey > Relevance > Support Survey > Relevance > Issue/Question Survey > Relevance > Suggestion Survey > Length > Support Survey > Length > Issue/Question Survey > Length > Suggestion Survey > Flow > Support Survey > Flow > Issue/Question Survey > Flow > Suggestion Survey > Other > Support Survey > Other > Issue/Question Survey > Other > Suggestion
Specific Item Codes Q# > Relevance > Support Q# > Relevance > Issue/Question Q# > Relevance > Suggestion Q# > Clarity > Support Q# > Clarity > Issue/Question Q# > Clarity > Suggestion Q# > Coverage > Support Q# > Coverage > Issue/Question Q# > Coverage > Suggestion Q# > Other > Support Q# > Other > Issue/Question Q# > Other > Suggestion Introduction Codes
Introduction > Relevance > Support Introduction > Relevance > Issue/Question Introduction > Relevance > Suggestion Introduction > Clarity > Support Introduction > Clarity > Issue/Question Introduction > Clarity > Suggestion Introduction > Coverage > Support Introduction > Coverage > Issue/Question Introduction > Coverage > Suggestion Introduction > Other > Support Introduction > Other > Issue/Question Introduction > Other > Suggestion
A-43
Coding: Overall Perceptions of the Survey
Protocol questions targeting respondents overall perceptions can be found in both the “Overall Perception” section and “Wrap-up” section. Responses to these questions as well as information
provided in the “think aloud” exercise will be categorized into three topical areas including: 1) relevance, 2) length and 3) flow. Feedback that does not fit into one or more of the topical categories will be coded as “other.” Researchers responsible for coding interview data will discuss items in the other category to determine whether they should be recoded into one of the three main categories or remain as items to be considered outside of the three topic areas identified. To assist with analysis, items coded into one of the topical areas will then be coded into one of the following subcategories: 1) “support” for survey, 2) “issue/question,” or, 3) “suggestion” for improvement.
Relevance: The extent to which the survey and its items tap into appropriate policies and practices.
Subcategory Examples
Support This survey covers important aspects of the early childhood workforce. I think it is very
relevant.
Issue/Question It is sort of relevant for us, but I wasn’t sure about the part that asked about opportunities for
advancement. I am not sure that makes sense for my position.
Suggestion for Improvement
The survey questions were relevant for me, but I might add more options to some of the questions to make it work for other early childhood educators.
Length: The number of items and the time taken to complete the survey
Subcategory Examples
Support I think it went quickly. I had no problem with the length.
Issue/Question I am not sure if teachers will take the time to complete all of these questions with their busy schedules. Suggestion for
A-44
Flow: The survey format and/or the grouping and ordering of items
Subcategory Examples
Support The questions made sense to be and seemed to be in the right order.
Issue/Question I couldn’t get the survey to let me go to the next page several times. I couldn’t figure out how to make it work. Suggestion for
Improvement
Maybe you could put the questions about benefits offered by my employer before the question about benefits I get through a spouse.
Coding: Specific Survey Items__________________________ ____________ Data coded for a survey item will reflect both information from the “think aloud” portion of the
interview and the item-by-item follow-up section of the protocol. Protocol questions pertaining to specific survey items include both standardized and conditional probes. Responses will be categorized into three topical areas including: 1) clarity, 2) coverage, and 3) relevance. For the purpose of this project, these topics are defined as follows: