Sense-Making Data Collection

This section covers the data collection process, documenting data, issues associated with use of the timeline interview, and maintaining confidentiality for sensitive data.

3.9.1 Data Collection Process

Data collection was performed following the guidance provided in the interview

protocol. To avoid curtailing an interview while participants were still providing useful information, participants were notified during the interview scheduling process that complex topics or topics requiring substantial explanation of context could take longer than 30 minutes, and that opportunity would be provided after the data collection to discuss the dissertation research if desired. The researcher was careful not to schedule back-to-back interviews, and to check with participants about how much time they had available prior to beginning an interview. This worked well.

Prior to beginning the data collection interview, voluntary written informed consent was obtained from participants, including permission to audio record the interviews. Refer to Appendix G for the approved consent form. The purpose of the interview was then explained. Opportunity was provided for the participant to ask questions.

A lesson learned for future interview protocols is that it’s helpful to let participants know up front that if there is anything in the design experience that they REALLY want to share and we don’t cover it in the interview questions, they will have the opportunity to share it at the end of the interview. That reduces the potential for participants to focus on what they want to discuss versus responding to the questions they are asked.

Participants were asked to describe a recent instructional design experience using an open-ended interview format that permitted them to tell their own story of what happened (refer to section 3.8.1). The questionnaire obtained data in the participant’s actual words, reducing the potential for researcher misinterpretation of data. To ensure adequate detail, the questionnaire included prompts for clarity and completeness.

Demographics were optional, but were not refused by any participant. All but one participant requested a copy of the study results (that individual was about to retire at the time of the interview). One participant requested a copy of the final dissertation.

The next section discusses the means of documenting data.

3.9.2 Documenting Data

Data was documented using index cards, a digital clipboard (DigiMemo System), and audio recording. One issue noted in previous timeline interviews was that although individual index cards are easily sorted, arranged, and shared with participants, they

are small enough to easily get out of order during an interview and have limited writing space available. It was decided to try using fanfold form-feed 4” x 6” index cards in sheets of three to provide additional writing space and easier organization. That still permitted separation into individual index cards if the participant wanted to see them and manually organize them as a memory aid during the interview. Form- feed index cards can be pre-labeled using a dot matrix printer. This worked nicely.

The digital clipboard could upload data files directly to Word and Excel to minimize the amount of transcription required, and provided redundancy for written data. However, it’s highly dependent on the researchers’ ability to print neatly, maintain a linear data stream (due to a rigid digital pagination structure), and remember to advance pages when needed. The digital clipboard had a clip failure during the second interview, forcing awkward use of a binder clip from then on. The digital clipboard was good for data redundancy/peace of mind, but a digital pen would probably be more effective.

Audio recording was optional, but was not refused by any participants. Audio

recording was extremely helpful due to the large amount of information communicated by respondents during interviews. It’s difficult to maintain the interview as an engaged conversation while trying to write down every word. Audio recording was performed using a Philips digital voice recorder. No problems were experienced with the audio recording except once when the recorder was accidentally turned on while inside a book bag and ran down the batteries. This problem could be avoided by obtaining a hard case for the recorder.

3.9.3 The Timeline Interview as One Piece of the Puzzle

Participants were interviewed about an aspect of an instructional design situation that was new to them, generally resulting in discussion of one portion of a larger

instructional design effort. Interviewing designers who were currently in the middle of a large design project and/or very complex design situation tended to take more time than for a simpler, more straightforward design situation, sometimes considerably so. Questions asked during the timeline interview sometimes required the participant to think about his or her design process in new ways.

Thinking in terms of steps/questions/question context was a new approach for most people. Participants may initially be confused about describing their actions as steps. Use of index cards as a visual aid helped them to envision their ‘comic strip,’ and after a few steps they grasped the process. Similarly, explaining details of question context may initially cause some anxiety, particularly for helps and hurts. Participants become familiar with the process after the first few question loops.

Discussion of one part of a larger design experience may require explanation of past and current design context and anticipated future design direction for the researcher to fully grasp the difficulties and opportunities involved. Sensitive information sometimes came up, especially during discussion of question context. That can cause anxiety. The approach for dealing with sensitive information is described next.

3.9.4 Maintaining Confidentiality for Sensitive Information

The summertime data collection timeframe resulted in a small sample size of 18

work on large special projects. Design efforts were sometimes so specialized that extra caution was needed to ensure that confidentiality was maintained.

The small sample size required establishing an approach to maintain data

confidentiality and reassure participants. If possibly sensitive information arose, the policy followed was to explain to the participant that while that data might be

acceptable to use in a much larger study where it would be one experience out of many, because of the small sample size no sensitive or controversial data would be shared. All data files were carefully reviewed to ensure that possibly sensitive information was not transcribed or included in the final dataset.

The next section discusses transcription and data entry.