5.4 Chapter Summary
6.1.3 Evaluation in the Wild
We conducted a field study to investigate the effects of the group mirror in a “real world” scenario. Our main questions were, what influence the group mirror has on performance, on the use of the different roles and on the acceptance of the feedback.
Method
As already described above, we conducted a field study, because there are only few studies that investigated group mirrors in the wild. Conducting a field study brings a number of advantages compared to laboratory studies, such as an increased external validity (Rogers, 2011). We compared two conditions, one with support of the group mirror, one without. The study was designed as a within-groups experiment.
Setup and Procedure
We conducted the study during a practical course at our university. The study took part in an intermediate step of this course. Beforehand, students had developed app ideas in groups of four. Now, these ideas should be discussed using the Disney Method. Therefore, each group presented their app idea. After each presentation, the app idea was discussed for ten minutes using the Disney Method. As 16 students took part in the course, four discussions were held in total. The four students that just had held the presentation did not take part in the discussions but listened and took notes. The remaining students were divided in two groups of six participants. One of these groups was supported with the group mirror. In each discussion the combination of participants was altered to reduce the effects of opinion leaders and group dynamics.
As in the studies described earlier, we used a Wizard of Oz approach (Kelley, 1983) to con- trol the group mirror. Two persons, each one sitting next to one group at a time, classified the contributions according their role. All sessions were audio and video recorded. We further- more handed out questionnaires after each session with 5-point Likert scales (1 = strongly disagree, 5 = strongly agree). Time and role of entered contributions were logged by the application.
6 Supporting the Disney Method 2,87 1,25 1,63 2,87 3,56 0,82 1,25 1,25 0,00 1,00 2,00 3,00 4,00 0 1 2 3 4 5 Standard Deviation
With Group Mirror Without Group Mirror Round 1 Round 2 Round 3 Round 4
43 24 22 15 30 17 37 20 0 10 20 30 40 50
With Group Mirror Without Group Mirror
Number of contributions
1. Round 2. Round 3. Round 4. Round Average
33
19
Figure 6.2: Results on the number of contributions and balanced use of the roles. Left: Absolute number and average of contributions with and without group mirror. Right: Standard deviation of the different Disney roles in the different rounds.
Participants
In the study, 16 participants (5 female; age range from 20 to 25 years) took part. All par- ticipants were media informatics students participating in a course on iOS development. All participants already knew each other as they at least had been working together on their app idea over the past three weeks. Only one of the participants was familiar with the Disney Method, all others did not know this creativity technique.
Results
The study was evaluated using the questionnaires, audio- and video-recordings and applica- tion loggings. A dependent t-test was used to evaluate quantitative information. A 5% level of significance was applied to the tests. Excel was used for calculating the t-test.
Performance With support of the group mirror, more contributions (independent from the role) were stated than without group mirror. We compared the amount of contributions that were voiced with group mirror (M = 33, SD= 9.055) with the amount of contributions stated without the support of the system (M= 19,SD= 3.916). A dependent t-test reveals a significant difference between these conditions (p< 0.05) (see Figure 6.2, left).
Balanced Use of Roles We could not find significant differences regarding the balanced use of the three roles. To calculate how balanced the roles were used, we compared the distribution of the number of contributions in each role from their mean (i.e., the standard deviation). As Figure 6.2 (right side) shows, the standard deviation was lower without group mirror in three rounds (without statistical significance). However, this can also be attributed to the fact that the number of contributions was higher with group mirror.
When looking at the use of the different roles, it became apparent that our initial assumption that people shy away especially from the critic role (that was based on observations of other groups using role-based creativity techniques) actually was true. Without group mirror, 39%
11 10 3 5 6 4 7 6 4 7 5 8 0 5 10 15 20
Dreamer Realist Critic
Number of contributions without group mirror
1. Round 2. Round 3. Round 4. Round
7.8 6.8 Average 4.8 18 14 11 6 9 7 8 12 10 12 16 9 0 5 10 15 20
Dreamer Realist Critic
Number of contributions with group mirror
1. Round 2. Round 3. Round 4. Round 9.5
11.8
Average 8.3
Figure 6.3: Results on absolute number and average of contributions separated between the Disney roles.Left: With group mirror. Right: Without group mirror.
were dreamer contributions, 36% contributions attributed to the role of the realist and 28% to the role of the critic. Furthermore, we could observe a tendency that especially the less used roles of the realist and the critic are used more frequently with support of the group mirror. With system, 33% were dreamer, 39% realist and 28% critic contributions. The absolute numbers are displayed in Figure 6.3.
Social Pressure and Stress As this type of visualization was a first attempt to focus more on content-related aspects instead of performance-related aspects, we were interested, if participants felt pressured through the system. Results from the questionnaires confirm that this group mirror visualization did not put much pressure on group members. Without group mirror, all participants disagreed to have felt stressed during the discussion. With group mirror, 13 disagreed, 2 were neutral, only one agreed that he or she had felt stressed. We furthermore asked, if participants felt observed. In both conditions, participants either disagreed or were neutral regarding this aspect (with group mirror: 11 disagreed, 5 neutral; baseline: 13 disagreed, 3 neutral).
Disruption The group mirror was not perceived as too disrupting. 15 participants dis- agreed, one was neutral regarding the aspect of disruption. Furthermore, 13 group members agreed that they could follow the discussion and the information on the display simultane- ously, 3 were neutral. This is also confirmed by the answers to the question if participants were able to focus on the discussion. In the group mirror condition, 15 participants agreed, 1 ticked neutral. In the baseline condition without feedback, 15 agreed that they could focus on the discussion, 1 participant did not agree.
Preferences In the questionnaires, we asked participants if they liked the condition with or without group mirror more. 14 stated that they liked being supported by the system more than without, one participant liked both conditions equally and one preferred the condition without feedback.
6 Supporting the Disney Method
Summary and Discussion
With the prototype and study described in this section, we investigated how a group mirror showing information about the usage of different roles in a role-based creativity technique influences collaboration. We did this by conducting a field study during a course at our university. We compared two conditions, a baseline and a condition in which the group mirror shows, how many ideas to the three different roles,dreamer, realist, andcriticwere stated.
Results show that the group mirror led to an increase of the overall amount of contributions. In a previous study (see Section 5.2), we compared a visualization that showed the overall amount of ideas (thecooperativevisualization) with a baseline, however, we could not find significant differences in the amount of ideas. In the study described in the current section, the visualization also shows the overall amount of ideas. Here, the study shows a significant difference in the amount of contributions compared to the baseline. One explanation for these different findings could be that the visualization showing the different roles of the Disney Method has encoded additional information (i.e., the different roles). Being reminded of the different aspects that group members can contribute could have led to the increase in performance, while in the study described in Section 5.2 no additional, possibly stimulating, information was visualized.
However, the study setup could have been another reason for these different findings. In the study described in this section, only two conditions were compared in a within-groups experiment. This implies that half of the participants took part in the baseline condition without knowing a group mirror system before. In the study described in Section 5.2, four conditions were compared in a within-groups experiment. Consequently, only one out of four participants took part in the baseline without knowing a group mirror. Having used a group mirror before could lead to an adapted behavior (i.e., being more sensitive about the aspects that the group mirror reflects) in the baseline condition, possibly leading to an increase in performance even when no group mirror is displayed.
Results show, furthermore, that the roles of the realist and the critic are used less often than the dreamer in the baseline. Group members already knew each other and their task was to discuss the ideas of their colleagues. Especially the role of the critic (and also the role of the realist) probably was less popular, as participants did not want to offend their colleagues by being critical about their ideas. With the group mirror, people stated more contributions matching the realist and the critic role. At the same time, disruption and stress did not increase, indicating that the more critic views were not accompanied with people feeling more stressed.