Mixed Methods Research Design

Visual model of mixed methods design procedures

QUAN data collection

QUAL data collection

Interview Exit Q Experiment no time limits QUAN analysis QUAL analysis Results integration

Study 1: Intervention mixed methods design in concurrent triangulation design (QUAN + QUAL)

Intervention / Experiment Decision Tasks x6 Pre Q Decision Dec Q Post Q Time limit: none

Topics: 6 + practice Topic order: 6x6

QUAN data collection

QUAL data collection

Interview Exit

Q

Study 2: Intervention mixed methods design in concurrent nested design (QUAN + qual)

Intervention / Experiment Decision Tasks x6 Pre Q Decision Dec Q Post Q Time limit: none

vs. 5 min. limit Topics: 6 + practice Order: 4x4 + 2x2 QUAN analysis QUAL analysis Results integration

This research project used a multiple phase, mixed methods research design as shown in Figure 4.1. In a mixed methods research design, both quantitative and qualitative data are collected and analyzed. The data and/or results are integrated and interpreted in order to take advantage of the strengths of each type of data collection and analysis approach.

The first phase of the research, Study 1, was used to inform the experimental manipulations and semi- structured interview protocol for the second phase, Study 2. Each study used a mixed methods experiment (intervention) design in which both quantitative and qualitative data are collected during and after the experiment in one research session (Creswell & Clark, 2017) . Data were collected using the same methods in each study. Participants in both studies were sampled from the same University student, staff, and faculty population; all participants were drawn from the volunteers from a single recruitment email. Participants were recruited for 90-minute study sessions, and they were given a $30 financial incentive for participating in the study. During the experiment, quantitative and qualitative data were collected using process tracing methods (e.g., search interaction logs, eye-tracking logs) and questionnaires with closed- and open-ended items. After the experiment, additional questionnaires were administered and a semi-structured interview probed participants’ perceptions of their experiences during the experiment. Concurrent think-aloud was not used during the experiment due to potential method reactivity in which think-aloud may reduce the speed of information processing and “disrupt task-related processing if the situation is time constrained.” (Maule & Hockey, 1993, p. 18)

Studies 1 and 2 differed along several dimensions. In the first study, the quantitative and qualitative data were analyzed separately and integrated in the interpretation of the results. The quantitative and qualitative results were given equal weight (i.e., priority) in the results. In the second study, the qualitative analysis included quantifying data to include in the quantitative analysis, and quantitative data was given priority.

There are two main differences in the design of the two studies. In Study 1, there were no time limits, and six topics were used and evaluated.3 In Study 2, there were two time limit conditions, and six topics were used and evaluated. One additional topic was used for a practice task in both studies. Second, the timing of the data integration differs. In Study 1, the quantitative and qualitative data was analyzed separately and integrated in the interpretation phase. In Study 2, the quantitative and a subset of the qualitative data were integrated in the analysis phase; the remaining qualitative data will be analyzed as future work.

4.1.1 Overview of experimental design. Both studies included a lab-based experiment in which participants completed up to six decision tasks. Participants were asked to complete “a series of decision tasks” embedded in a simulated everyday life scenario which asked them to make a recommendation for a friend on an assigned topic. They were explicitly not informed in advance of the number of tasks to complete. A search system was provided if they wanted to search for additional information to inform their decision. Participants were told that the purpose of the study was to investigate decision-making. Study 2 participants were not informed of the true purpose of the study: investigating information search in support of decision-making under time pressure. A waiver of elements of informed consent was received from the ethics review of the UNC Institutional Review Board, and a debriefing was held at the end of the study session to reveal the true purpose of the study.

To induce time pressure, a task time limit were used in Study 2. Task time limit was a between-subjects experimental factor with two conditions: a 5-minute task time limit, and no task time limit. The amount of time allowed in the time-constrained condition was set based on the mean task completion time in Study 1. In the preliminary study (Study 1), participants were not given any task time limits nor guidance for how long they should spend on a task.

Participants were not informed in advance of the number of tasks to complete to minimize time pressure induced by the presence of 1.5 hour session time limit and to prevent time allocation across multiple tasks. The experiment system showed the next action for the current task and did not show future tasks and/or questionnaires. As described in Section 4.5.2, participants were assigned an additional task if they had spent less than a certain amount of time on the experiment. This threshold was operationalized as the time elapsed since the start of the practice task and was set to 70 minutes for Study 1 and 55 minutes for Study 2. Participants were assigned up to six decision tasks on six different topics. To minimize order effects, the order of the topics was counterbalanced using a Graeco-Latin Square. Topics and topic assignment are described in more detail in Section 4.4.4.