Design Quality

Similarly to previous work [80,81], I recruited three experts to rate the designs produced by each participant for quality. All were expert designers in academia with at least a decade of experience in practicing design and with grading student work. The designs were ranked by overall and protean design quality. Protean means “being able to change easily” or that a design is versatile. Essentially, this quality is used in design to classify how worthy something is to build upon. Ratings ranged from 0 to 5, with 5 being best. The results are summarized in Figure 4-12. Figure 4-13 depicts the averages of these results. The overall design quality of the outcomes with both systems was similar, which may be a consequence of the limited time that participants had to work on their designs. Yet, Interestingly, the results achieved with GEM-NI were ranked overall higher in terms of being protean, which I interpret that their creativity potential is higher.

Figure 4-12. Overall and protean quality ratings of participants’ designs by each of the experts (E1-3). Higher is better.

0 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 R an k Participant

Design Quality Ratings

E1 Overall NodeBox E1 Overall GEM-NI E1 Protean NodeBox E1 Protean GEM-NI E2 Overall NodeBox E2 Overall GEM-NI E2 Protean NodeBox E2 Protean GEM-NI E3 Overall NodeBox E3 Overall GEM-NI E3 Protean NodeBox E3 Protean GEM-NI

Figure 4-13. Average overall and protean design quality rankings by the three experts (higher is better). Error Bars: ±1 SE.

4.6 Overall Discussion

The results of the second user study shine more light on issues revealed in the first user study. With a larger sample size, GEM-NI was confirmed to support exploration better with the new features, as also supported by the results from the CSI questionnaire as well. As in the first user study, a number of participants created alternatives in the first task, where they did not know about GEM-NI’s features, without prompting. In the freeform feedback three participants expressed that they wished they had the features of GEM-NI available in the generative design tools they use daily. All this supports my initial hypothesis for the need for GEM-NI’s approach in current generative design tools. Some freeform feedback revealed that the learning curve for using a system that fully enables exploration through alternatives, such as GEM-NI, is a potential issue.

Interestingly, some features that participants identified as desirable in the interviews in

0 1 2

NodeBox GEM-NI NodeBox GEM-NI

Overall Overall Protean Protean

R

an

k

Ranking Category and System Average Design Quality Rankings

second user study. On the other hand, minimizing, one of the suggested features, was used at least by a part of the participants. This underlines that user interface designers should approach all requested features critically and consider them thoroughly. In contrast, the major features that I decided to implement within GEM-NI proved to be more useful, with the exception of global undo. The fact that global undo was not popular may potentially be explained by the fact that few people can simultaneously design multiple alternatives in parallel, due to the increased complexity of such a task. On the other hand, the ability to perform post-hoc merging and resurrection from history was confirmed as useful, which points out that considering multiple alternatives in linear or potentially non-linear fashion is much more realistic. A participant from the interview in the first user study expressed exactly this insight. In hindsight, I believe that if I had designed the user study and logging specifically to reveal instances when parallel editing occurred and contributed to the design, I would be able to state this more confidently. From the current logs it is difficult to say whether parallel editing or global undo was accidental or desired. Merging, a concept novel for generative design, but common in software development as part of version control (e.g., Git), received some interest. But as in the first user study only few used it in the second one. One reason could be that the participants I worked with in the second study were not familiar with the concept. In hindsight, I should have asked about experience with version control systems in the pre-questionnaire. That might have enabled me to better analyze this issue. It is also interesting to point out that beyond exploring alternatives, the mechanisms introduced in GEM-NI permit designers to use the system as a version control system. Like in source code version control systems, designers do not need to create multiple files and are not restricted by the limitations of copy and paste. As in source code systems, branching and borrowing of ideas is facilitated with the ability to resurrect from history. Through merging, designers can save time by re-using existing parts of the design. I also can point out that in GEM-NI alternatives and version control are effectively integrated into one user interface.

Based on these findings I speculate that the user interface of GEM-NI for managing alternatives is not specific to generic design. All the newly introduced techniques can be

applied to other design workflows. At least one of the participants already expressed the wish that the alternative management techniques should be available in their more familiar 3D parametric modeling workflow. Though, adopting a design gallery into a system like Processing, which is purely code-based, would arguably require a new interface.

I also believe that the techniques that were popular in GEM-NI in the evaluations will likely also be popular in domains outside of design that involve problem solving, such as visual analytics. Similarly, I expect the unpopular techniques (such as global undo) to be unpopular in other domains as well. Finally, and beside all the benefits of the new techniques, I expect that the obstacles that GEM-NI’s interface encountered to transfer as well. For example, parallel editing should not be the default, regardless of domain.

Both user studies may suffer from a hidden skill transfer effect, because NodeBox was presented and evaluated first and GEM-NI second. However, it is not easy to use counterbalancing with a between-subject study since the two tasks used here are not exactly the same. Moreover, if participants are alerted to the use of alternatives in the first phase through GEM-NI, this would likely bias the results for NodeBox as well.

4.7 Summary

In this chapter I presented GEM-NI—a new system for creating and managing alternatives in generative design. The system supports parallel editing via checkmarks and sandboxing, two new methods to control which alternatives are affected by a parallel edit. Also, I introduced a novel method for post-hoc merging of alternatives. Moreover, GEM-NI provides several methods to create alternatives, including a new method for resurrecting alternatives from a graphical history with previews, with full lineage preservation. Another way to create alternatives is with a new design gallery, which enables users to select which ranges of parameters and/or parts of the generative network model to use for exploration. Moreover, my design gallery supports a new method to explore products of generative networks.

I also conducted two user studies to explore usefulness of the system. These user studies were conducted as part of a workshop where participants first learned how to use the tool.

The feedback from participants in the first workshop and in-depth interviews suggest that GEM-NI, and more broadly the approach behind it, indeed enables designers to work more creatively. The results indicate the direct applicability of the presented techniques for the design process also via the CSI questionnaire. While the sample size of my user study is small, it identified the potential for better creativity support through alternatives in design tools. In the second user study, the tasks were refined and revisited with a larger sample of participants. The results resolved the issues raised by the first user study. GEM-NI was found to support exploration better due to the newly introduced features, which was also supported through the CSI questionnaire. In the freeform feedback three participants expressed that they wished they had the features of GEM-NI available in the generative design tools they use daily. Three expert designers in academia evaluated the quality of the participants’ designs. I interpret the rankings as suggesting that the creativity potential of the results is higher with the designs produced with GEM-NI.