The purpose of this study was to investigate designers’ behaviour in regard to people information. It was assumed that designers, like other information users, would have two types of information behaviour; ‘perceived’ and ‘actual’. The Design Bugs Out study mainly focused on investigating the ‘actual’ information behaviour but also aimed to address the perceived behaviour. This was in order to prohibit future limitations in the analysis of the collected information (Fidel and Green, 2004) and also to enable comparisons and reflections on any potential differences between the results of the ‘observed’ versus ‘reported’ behaviour of designers. Thus, for the Design Bugs Out study, observation was adopted as the primary data collection method and follow-up questionnaire as the complementary method in order to provide comprehensive information on both information behaviour aspects. The study was divided into two sections based on the methods used. In the first stage of the study, designers’ actual information behaviour was studied through observation. In the second stage, designers’
perceived information behaviour was studied through the use of self-reporting questionnaires.
5.3.1 Observation
Being mainly a descriptive study, direct observation was used as the primary method for capturing the actions of the designers (Robson, 2002). Jordan and Henderson (1995) argue that direct observation is a valuable method as it acts as a shared source and helps to overcome the gap between what people say they do and what they actually do. This was a major consideration for adopting observation as the key method, as the previous study had primarily focused on what ‘designers said they did’. Also, it is argued that the use of observational methods could reduce the amount of assumptions the researcher makes about the behaviour of real users (designers in this case) (Keates and Clarkson, 2003).
Zeisel (2006) lists the key qualities of observation method as being ‘empathetic’,
‘direct’, ‘dynamic’ and ‘variably intrusive’. In being intrusive, Zeisel argues that as a dynamic phenomenon, the observation method allows the researchers to vary their level of intrusiveness in the project. Based on this, he suggests there are various roles a researcher could adopt in an observational setting. Zeisel (2006) categorises these roles into four main types including ‘Secret outsider’,
‘Recognised outsider’, ‘Marginal participant’ and ‘Full participant’. These
121
categories were in line with Robson’s (2002) classification of observations, ranging from ‘Participant observer’ to ‘Participant-as-observer’.
Apart from the observer’s role, observational methods could be classified based on other factors such as their nature, purpose, level of pre-structure and the type of information collected. Robson (2002) describes two key types of observation structure-wise, including ‘formal’ and ‘informal’. Informal observation is less structured and prescriptive and gives the observer a higher level of freedom in terms of what information they collect and how they collect it. Formal observation, however, imposes certain level of structure and pre-defined direction in terms of what has to be observed and how. Robson argues the former method is complex and needs a more in-depth analysis and synthesis role from the observer while the latter is more reliable and valid but at the same time may be limited in terms of exploration and completeness. He classifies the former as mainly qualitative and the latter as quantitative. The approach to observation in this study was largely formal, yet at the same time informal; the observer was specifically looking for information regarding the six dimensions of information framework, at the same time the observation process was fully open to document any unidentified behaviour or attitude. Addressing different levels of observer participation, Robson (2002, p. 321) says “while the pure-observer typically uses an observation instrument, the participant observer is the instrument.” In the Design Bugs Out study, the ‘Marginal Participant’ role was adopted where the observer participated in the design project as an identified but unimportant ‘team member’. It was hoped that this would largely limit the
‘being observed’ effect on designers and would thus have the least interference with the natural flow of the real-world design project.
5.3.2 Follow-up questionnaire
Questionnaires provided designers with the opportunity to reflect on the process they had gone through in the Design Bugs Out project and report back on their people information behaviour in terms of various dimensions of the information framework. In order to complement findings from the observation of designers and their information behaviour throughout the Design Bugs Out project, follow-up questionnaires were conducted with the key designers involved in the project.
This was in order to collect designers’ own reflection and self-report on their information behaviour in the project and facilitate analysis and comparison of
122
findings from the self-report and observation. One designer from the design consultancy and one designer from the manufacturer company were asked to complete the questionnaire after the project had come to an end. The two designers were selected on the basis of their key role in Design Bugs Out project.
Bernard (2002) identifies the highest level of structure in ethnographic interviews as ‘structured interview’. He describes one type of structured interviews in ethnography as being questionnaires which may be self‑administered by the study participant where respondents are given the same choices of responses to select. The benefit of such structured technique is the reliability and validity of the results. Following the same logic and aiming at the most reliability and validity in designers’ self-report data, questionnaires were provided to designers with given choices. Fixed questions were combined with open ended questions resulting in close ended questions with multiple choices in which participants were also allowed to select an “Other, please specify” response. It gave the participant the chance to provide a response category (in this case, sub-dimension) not listed, which helped to increase rigour, inclusivity and precision and provided the researcher with a more robust response.
The questionnaire was designed in an electric format and was made available online. The online format was preferred to hard copy format or face to face interviews as it gave designers the highest level of flexibility to respond in their own time, was more convenient to send and collect and was also in line with the online platforms designers used on a daily basis. The questionnaire asked designers to reflect back on the Design Bugs Out project design process and report their information behaviour by answering to questions on each dimension of the information framework. The questionnaire was organised in a chronological order, based on stages of the design process. For this purpose, the Design Council Double Diamond model (Design Council, 2005) was adopted. In the questionnaire, it was made specifically explicit that the focus of all questions was on people information. A copy of the full questionnaire is provided in Appendix B1.
123
5.3.3 Observation procedure
The observer was introduced to the design team as the research partner, at the initiation stage of the project, at the same time with introduction of other team members and formation of the design team. This helped create a better acceptance of the observer as a team member, rather than an outsider. The team members were made aware of the role of the research partner in general and the nature of their involvement in the Design Bugs Out project. It was also explicitly mentioned that the whole process of the design project might be used for research purposes. However the team members were not specifically made aware that their information behaviour was being observed. This was in order to avoid making the designers self-conscious of their information behaviour and influencing it in any way.
As the core design team consisted of members from both the design consultancy and the manufacturer company based at different locations, keeping constantly in touch was a major consideration for the team. Thus, a detailed project plan was devised by the design team including milestones and deadlines and dates for weekly face-to-face meetings in order to report on progress made on each side, communicate findings, and develop and discuss the impact of these findings in a fuller picture. The weekly meetings were considered critical to the project progress and key decisions were made in them including major direction and specifications of the research, assigning roles and deciding the areas of focus for each team member until the next meeting. These meetings were usually held in the design consultancy office and the observer was expected to attend these meetings as a research partner to the design team. Also, email and phone conversations were intensively used as another channel of communication. Apart from meetings held between team members and phone and email correspondence, a number of visits, workshops and observational sessions were carried out by the team, where the observer also attended and observed as a team member. All these various types of team activities were treated as an observation session with an observational context.
124
5.3.4 Recording the observations
Zeisel (2006) suggests a list of observation recording devices including notation, preceded checklists, maps, photographs, and videotapes and movies.
Considering the role and level of involvement of the observer in the Design Bugs Out project (marginal participant) and also the length of observation period (throughout the project, lasting approximately six months) it was crucial to adopt a consistent, comprehensive yet manageable approach to data collection and choose appropriate methods and devices for recording the various types of observation sessions that were held. Having adopted a ‘participant observer’
approach, using obtrusive recording devices such as a video-camera and voice recorder needed careful consideration and was largely avoided in most cases.
Use of video and audio recording for observation is generally considered heavily time-consuming and labour-intensive in terms of collection and the analysis of raw data (Robson, 2002). This was particularly important considering the long observation period in the Design Bugs Out study. Thus, considering the nature of the study in all the above, notation and checklists were used as primary recording tools for the observation, accompanied by photographs and videos if and where appropriate. This was alongside but separate from all the material and documentations produced in the process of the Design Bugs Out project. These documents were produced by various team members, mainly for the purpose of internal communication among the design team or for external communications with the clients, experts or media. These included emails, meeting minutes, short or long reports, full versions or extracts from various secondary sources including expert reports and internet content, images and videos, and PowerPoint presentations. All the above were also recorded and documented for the purpose of further review, reflections and analysis of the designers’ information behaviour.
In recording informal observations, it is suggested to start with a descriptive observation, explaining the setting and then to analyse the well-described setting in order to come up with explanations and frameworks detailing the events and acts. Whitehead (2005) suggests a set of comprehensive categories for informing observation that is focused, descriptive and selective. Also, Spradley’s model for descriptive observation (1980) suggests nine dimensions including Space, Actors, Activities, Objects, Acts, Events, Time, Goals and Feelings for the descriptive
125
observation of a setting. However, such framework or categories were not used in observing the Design Bugs Out project as the observation was largely formal and already followed a certain agenda in terms of what needed to be observed.
In this case, the six dimensions of the information framework gave structure to the observation and guided the sessions by outlining what aspects of designers’
information behaviour had to be observed. The agenda in every observation setting was to seek answers to research questions one to six (outlined in Section 5.2.1), leading to a ‘category system’ for the observation (Robson, 2002).
5.4 Data analysis method
Observation data was collected in various formats including notations, checklists, meeting minutes, emails and research presentations and reports. The ‘template approach’ (Robson, 2002) was used in order to analyse the data, being mainly quantitative. The collected data from various sources was processed in two main stages of coding the data and then clustering the data of the same code into groups. Using this technique, the collected data from various observation sources was first coded with words or phrases derived from the six framework dimensions and their sub-dimensions, then the data labelled with the same code were clustered together. The data collected from follow-up questionnaires was collated and organised in the order of each information dimension.