Methodological remarks

The experimental work was conducted in the form of several standalone design sessions, each of them tackling a particular design problem. All design tasks presented to the participating designers were taken from the domain of designing controllers for large-scale and complex systems. A typical assignment included the design of a control strategy, a control loop with the controller, or both. Brief details of each design problem are listed in section 7.2 and more in Appendix A.

All problems involved technical or technological systems. That is, we did not test the RFD theory on tasks involving artistic design, such as house interior design or architecture. Architectural design was a favourite research topic of many researchers in the past decade. A lot of high quality research was done in this area and can be found e.g. in (Schön 1983; Gero 1990; Fischer 1992; Gomez de Silva Garza and Maher 1996; Maher, Poon et al. 1996; Watson and Perera 1997). Neither did we include the sociological or economic design problems, such as described by Schön (1983). Primary attention focused on technological systems, devices and controllers. The referential literature includes, for instance, (Schön 1983; Altshuller 1984; Bylander and Chandrasekaran 1985; Smithers, Conkie et al.

1990; Iwasaki, Fikes et al. 1993; Bhatta, Goel et al. 1994; Choueiry, McIlraith et al. 1998; Cook and Brown 1999).

From the methodological point, we note that we were focusing on the early – or better, conceptual phase of design. The most serious implication of such a focus for the designers was restricting them to the use of sketchy drawings on paper, conceptual descriptions and justifications of their decisions in words. They did not have any powerful modelling packages at hand, such as Matlab/Simulink™ and CAD tools. However, unlike similar conceptual design studies (Cross 1997; Cook and Brown 1999;

Cross 1999), our participants were given a computational tool to structure and record their reasoning on-the-fly. We admit that the presence of such a tool can be a double-edged sword; i.e. it may have both, positive and negative influences on a designer’s performance (Davies 1995). Since the experimental settings are crucial for the validity of the results, let us first attend to the usage of the customised computation tool in the design experiments, and compare it to more common observational methods.

The advantage of the naturalistic and ethnographic methods is that the experiment is conducted in the setting that is natural for the participant (Rose, Shneiderman et al. 1995; Jagodzinski, Reid et al.

2000). In other words, the investigator is involved in the everyday activities of the participants, and typically takes the role of an external observer or interviewer. Different versions of the ethnographic approach include video and tape recording, note taking, looking over the participant’s shoulder, or thinking aloud (Anderson and Crocca 1993; Kensing and Munk-Madsen 1993). All these methods could be applied on-line or off-line. In the former case, the experiment is made overtly, and the observers have an active role. They may ask questions immediately, when an unclear decision or an unexpected line of thought is observed. Alternatively, the observer works covertly, without disturbing the participants. In this case, the ambiguities are usually clarified at the end of the experiment, when the investigator goes through the notes, and interprets the observations with the domain experts. Very often, the direct observation is followed by an interview with a participant.

It is commonly accepted (ACM 1993) that the on-line methods are better, when it comes to the precision of the observation. However, because the observer directly interacts with the designer, and intervenes in the design, the validity of the observation may suffer (Davies 1995). The interventions of the observer might destruct the natural setting of the experiment, and in the extreme case, the recorded process may differ from the situation, when the designer was left alone. On the other hand, the off-line approach preserves the natural setting, and minimises the observer’s interventions. However, because the ambiguities are discussed and justified after the session, the participant may produce a totally different line of reasoning to justify a particular decision compared to the one that actually occurred earlier during the design.

Therefore, we decided to take a combined approach to capturing participant’s reasoning and justifications in the course of a design process. In practice it meant that we left the participant working on his or her own, without any direct intervention by the investigator. The designers chose their own pace, strategy, levels of granularity, etc. Simultaneously, they were asked to record their progress using a networked computational toolkit that enables the external observers to follow the explicit portion of their reasoning. Such a set-up also gave designers an opportunity to post a query or an interesting idea to the debating environment. From there, any ambiguous or otherwise interesting statement may be picked up by the observer, who can answer it, provide a comment on it, or leave it unnoticed.

This form of interaction was chosen after the initial interviews with the domain expert from the participating organisation. The leader of the group, from which participants were selected, claimed that they were used to interacting with their peers on an informal level, but preferred a more formal intervention from the senior staff, tutors or customers. When we later interviewed the participants and raised this point, they explained that formal interaction gave them the opportunity to both acquire the requested information and have a record of it for future reference.

We asked the participants to record their notes in a kind of on-line notebook. Each statement they entered was automatically organised into threads (Sumner and Buckingham Shum 1998), if it extended an existing (i.e. previously recorded) statement. A thread typically contained the reasoning step behind one design decision or a set of closely related decisions. It could be a decision concerned with a specification of the design task or a generation of the solution. New threads were typically started, when the designer’s focus moved to a different module or extension of an existing module beyond the scope of the explicit context, as defined at the particular moment.

A snapshot of the environment recording the justified threads of a designer’s reasoning is shown in Figure 7–2. The right-hand window shows details of one such justification. The left-hand window depicts sequences and threads of reasoning steps. A detailed transcript of this threaded record is available in Appendix D. Threading is a useful vehicle for tracing the development and extension of a particular reasoning chain. Figure 7–2 shows the situation when the designer discovered an unexpected flaw in his approach to smoothing a paper (details in section 8.4). The danger of damaging the paper was noted in the ‘active’ thread (see label n), and detailed in the small window (labelled as o). Next, a follow-up trying to fix the conflict appeared that was unacceptable (label p and ‘thumb-down’ icon).

Thus, another – acceptable modification was proposed (see q and ‘thumb-up’ icon).

Figure 7–2. Threaded justification of reasoning steps

n

o

p

q

Another tool that was available to the designers aimed to capture and structure the co-development of the problem specifications and solutions. In line with the RFD theory (see chapter 6), the tool depicted in Figure 7–3 defined the term ‘context’ (see label n) as a selection of explicit requirements or constraints (in the left pane, labelled o) articulated by the designer in respect to the designed artefact. The explicit commitments to a certain problem specification were usually accompanied by articulations of respective solutions addressing the particular design context or focus; see pane labelled p. ‘Active context’ was incremented automatically by the tool whenever the designer proposed a new (partial) solution. The explicit details of a commitment to a particular requirement or solution are displayed in the bottom right-hand pane (label q shows details of requirement 6.1).

Figure 7–3. Design context capture

The snapshot of the tool in Figure 7–3 shows design problem T21 annotated and analysed in section 8.4. It was made at the moment when the designer returned to a previously attended context II, after articulating several additional extensions. Thus, the designer was able to keep several lines of enquiry open at the same time – but grouped into separate ‘parallel’ contexts. As the design progressed, the lists of explicit requirements and solutions grew accordingly. New requirements could extend the ‘old’ ones (e.g. 6.1 is a refinement of 6 – see also pointer r), or address a new feature/issue (e.g. 7 or 8). A detailed transcript of records of such a co-evolution between specifications and solutions for design session T21 is presented in Appendix C and its annotated version in section 8.4.

Further details on the deployed capturing tools can be found in (Dzbor 2001; Dzbor and Zdrahal 2001a). Let us therefore mention only the main advantages of the applied methodology. The utility of these two tools was that they collected data about design reasoning in a structured form, and simultaneously provided the observer with a discrete opportunity for intervention. The intervention could be masked as a design comment or recommendation within a threaded sequence. It fitted well into the work style the designers were acquainted with. They were not distracted and did not have to stop their reasoning to answer a query from the observer. They could go on with their development, and answer the query, when more suitable. This would be impossible, if the observer’s interventions were formulated verbally and presented directly in person (as in the more usual approaches).

n o

p

q

r

Taking into account the ‘non-invasive’ collection of data, the automation of threading and context creation we would like to argue that the danger of imposing the investigator’s opinion onto designers was minimised. The recordings of the sequences of reasoning steps constituted the primary source of data for the analysis. However, to elicit participants’ subjective opinions regarding the toolkit intuitiveness or user friendliness, post-session ‘interviews’ were conducted. During these debriefing conclusions, each session was summarised, with the intention to highlight the main ‘milestones’ in the interpretation of the design problem.

‘Debriefing’ also played an important role in explaining the positions of various accompanying drawings and sketches in the explicitly recorded sequence of reasoning steps. This post-processing phase was needed because of the usage of multiple media for capturing the designer’s reasoning;

including paper notebook and pencil. ‘Debriefing’ thus merged data from the separate sources into a single design case. It included the location of exact spots of where the designer moved from words (computational tools) to drawings on paper, or vice-versa. The information about these ‘interruptions’

or shifts in reasoning was useful especially for the validation of the RFD model of design process.