Data collection methods and procedures

The primary data collected for this research were quantitative (numerical) and qualitative. Five data collection methods were used, which were deemed appropriate for this study: literature review, questionnaires, focus groups, a usability test, interviews data analysis, and trustworthiness. In addition, the design was synthesized from the literature review to assist the researcher when designing a digital prototype. From this, a model of development was adapted to guide the researcher when developing the digital prototype in this research. These data collection methods and procedures were as follows:

Literature review

The literature review gathers a range of documentary sources for the data collection; documents can include textual data, photographs, images, audio, or digital files. A key concern is to assess their credibility to determine their authenticity, correctness, and whether they are free from bias and error (Johannesson & Perjons, 2014). The main advantages of this method are that data sources from documents can be collected in a short period of time; it is relatively inexpensive;

and does not pose usually ethical problems. On other hand, a disadvantage of a literature review

is its credibility as some documents may not be acceptable (Johannesson & Perjons, 2014).

Nevertheless, a literature review was a crucial method to investigate and review research findings before the questionnaire design; explained and provided further resource for other methods in this research.

Questionnaire

Colin (2010) explained that a questionnaire is a data-gathering technique that is used to collect either qualitative or quantitative information from an individual unit. This is identified in the research sample through written self-reports regarding the unit’s knowledge, beliefs, opinions, or attitudes about or toward a phenomenon. In a case study, there are two ways to adopt research questionnaires, namely: 1) the questionnaire can be the primary technique of a research strategy for data collection, or 2) the questionnaire can be used in combination with other case study techniques, such as interviews (Colin, 2010).

In this study, questionnaires were used in two ways; firstly, they were delivered to research participants for them to complete by themselves over the Internet. A qualitative questionnaire was adopted (see Appendix 4) that employed both closed-ended and open-ended questions to collect the data. This is particularly appropriate for descriptive, explanatory, and exploratory studies. These answers were analysed as a qualitative method using coding and themes, which helped to build a final model for the data collection and analysis. Secondly, a questionnaire was used while research participants completed the performance measurement of the usability test. Participants completed the Software Usability Scale (SUS) questionnaire; these results were calculated and analysed to score the usability. The satisfaction score was defined in terms of its usability.

Design

After understanding users’ requirements, the prototype architecture was built as a master plan of the prototype design and development. It was used to develop a digital prototype based on the model of Creative and Critical Thinking in Problem Solving for Design (CCTPSD) and a tool for Problem Solving. These were modified in this research for the concept and system design in this research. The CCTPSD model integrates components of design, Lawson’s idea creation model of thinking (2006), and Torrance’s creative thinking (Alvino, 1990, cited by Cotton, 1991) (see p.77) in order to address creative and critical thinking in the design process.

The CCTPSD model helped the researcher to build user-interface and system design diagrams

and to develop a graphic user interface for the prototype in order to follow the architecture, as illustrated in Figure 4.5.

Figure 4.5 Architectural model of prototype (Source: Researcher’s own)

In this research procedure, the specification requirement was defined for the architectural model (Figure 4.5). Thus, the researcher implemented the specification requirements to build a digital tool for a portable device. A portable device platform was chosen, which uses a CloudKit framework to manage the data among the target users, who are AEC professionals involved in co-designing within concept design. From this, the researcher adopted the strategy addressed in the CCTPSD model. The user interface and system designs were first developed as diagrams by the researcher at the first. Then, the prototype’s graphic user interface was designed and developed. In the concept and system design process, it is necessary to consider prototype’s environment, requirement, and operation system whilst the diagrammatic and graphic user interface are designed in the individual process.

Development

After developing the user interface and system designs, a development method was used to help the researcher simultaneously develop the coding and testing in order to build the prototype through the appropriately modified software development approach (see p.210). This stage was possible because the researcher is a software developer. The modified approach was developed from the prototyping model (Davis, 1992) and integrated the incremental process so that the prototype could be developed in stage, from a small to a complete prototype (see, p.217).

Coding is a process of computer programming, and was undertaken by the researcher. The prototyping was divided and incrementally developed to complete each part within three to four weeks. Each part was iteratively improved and developed for new additional features and was then tested. The researcher ensured that the coding was tested, both in terms of its functions and via an integration test, which was conducted before research participants undertook the evaluation.

Usability test

A usability test assesses how software works after it is developed or before it is released as user software. Nielsen (1994) and McClelland (1995) explained the process steps required to set the instruments in place for the user test. The following outlines how this was conducted, which comprised two sub-procedures:

The first sub-procedure was a measure performance, where each user tested the prototype tasks.

The time performance of each task was captured and compared with an expert undertaking the same tasks; this helped to calculate the efficiency and learnability values. This part of the study used a quantitative method to collect numeric data, which complemented the results generated from other methods. Nielsen (2012) advised a sample size of three to five participants or three to five participants per group. The user performance is almost always measured by inviting a group of test users to perform a predefined set of test tasks while collecting time and error data.

The goal of the user performance measures is to collect usability attributes: learnability, efficiency, memorability, errors, and satisfaction. Thus “With 5 users, you almost always get close to user testing's maximum benefit-cost ratio” (Nielsen, 2012). The second sub procedure was the Software Usability Scale (SUS) Questionnaire, which was used to support the data collected from the other methods.

In this research, the performance measurement and the SUS Questionnaire were adopted to collect data from participant groups; the sample size of each group was three participants.

Participants’ activities were recorded, and this was used for the performance measurement when using the prototype. After that, the SUS questionnaire was handed out to collect the participants’ satisfaction levels (see the SUS questionnaire form, p. 341).

Group evaluation and focus group

This procedure gathered three groups to test the prototype in terms of collaborative design. In particular, it focused on: 1) colocation, 2) different locations or delocation, and 3) a mix of both conditions.

- Firstly, all participants were assigned to use the prototype on the same table to replicate the condition of colocation.

- The next condition was different location, when all participants were separated and assigned to work together through the Internet and a WIFI connection.

- The final condition was a mix of both, when two participants worked together, and another was separated but could work with other participants through the Internet and a WIFI connection.

Each collaborative design term was used in the group evaluations, and this method gathered three participants per group to use the prototype with each other. The purpose of this procedure was to prepare each group of participants to gain experience of the prototype in order to enable a discussion in the focus group on collaborative design.

Focus group

A focus group can be conducted using semi-structured and in-depth interviews that concentrate on enabling and recording an interactive discussion between participants on clear and precise topics (Saunders et al., 2016). A focus group emphasises participant interactions, which is a key feature in enabling an interactive discussion and thus obtaining views or attitudes about a product. The purpose of the focus group is to construct shared meanings through participant interactions, including their sense-making of a topic. The number of participants depends on the complexity of the topics; thus, for more complex topics, a smaller group of interviewees is recommended, such as two, four, or ten participants (Saunders et al., 2016). Barlow (2010) advised that three to five groups of participants could help to verify the data and determine whether these are influenced by particular group dynamics. In addition, a session length of 90 minutes is generally sufficient, although participants are usually asked to attend for around two hours (Barlow, 2010).

In this research, a focus group was used in the evaluation process to collect data from at least three groups of research participants. After completing the group evaluation and gathering participant experiences of the prototype, all groups were arranged into focus groups. Each group spent 60-90 minutes in the group discussion, and the Six Thinking Hats technique (De Bono, 2017) was adopted, which is explained further in Chapter 7.

Interview

Interviews are widely used to collect data when conducing a systematic inquiry; they are defined as ‘conversations with purpose and direction’ that seek knowledge and understanding (Barlow, 2010). The interview adopted in this research was semi-structured, which lies between a structured and unstructured format, and explores a number of predetermined topic areas with some flexibility. The purpose of the interview can follow that of the research, namely, it can be exploratory, descriptive, explanatory, or conduct an evaluation.

The semi-structured interview aimed to conduct a process of evaluation and required several participants in a face-to-face method. Nielsen (1994) advised that five participants for an interview method is usually appropriate. These participants were invited to attend an interview after participating in the focus group. The length of the interview ranged from 30-45 minutes in length (see Appendix 13 For the interview questions p.354).

Data analysis

Analysis was conducted for both the qualitative and quantitative data. This research adopted an abductive approach and chose a multiple phase methodology using a mix method (complex) approach. Therefore, quantitative (numeric) data were collected from performance measurements, which gave participants’ levels of satisfaction in the usability test (see p.121 and p.239).

Building coding and themes were used for the qualitative data analysis; this helped to describe the phenomenon in depth (Braun & Clarke, 2006, Braun, Clarke, Hayfield, & Terry, 2018, and Yin, 2011). Yin (2011) proposed qualitative data analysis through a cycle of ‘Five Analytic Phases’, whilst Braun and Clarke (2006) provided a linear ‘Theme Analysis’ process comprising six phases; these were similar to the code building and theme analysis on the interaction among phases (see Figure 4.6). Yin (2011) and Braun et al. (2018) advocate the gradual digestion of complex contents collected from qualitative methods. They suggest: 1) identifying the keywords or themes and, 2) assembling these keywords or themes into, an appropriate category through interpretation. Building coding, themes, and categories are adopted to analyse data in the qualitative methods of this research.

Figure 4.6 Five Analytic Phases and Theme Analysis (Source: Modified from Yin, 2011 and Braun et al. 2018)

The abductive approach in this study combined both induction and deduction to explore the phenomenon; this help to build and test the concept framework. Furthermore, this research conducted a two-step data collection (see the research approach, p.111 and methodological choice, p.112) as follows:

1) The first data collection was inductive: it continued the research process from through addressing the objectives and questions. This was achieved by collecting and analysing data from the sample. The qualitative data analysis helped to generate codes, define themes and build relationships. The themes were assembled into appropriate categories to further analyse the phenomenon. Triangulation was used to support the findings and to explain the coding and themes from multiple sources, for example the literature review and another data collection methods (Yin, 2014). These categories helped to build model of the prototype and to transfer the model and explain the prototype specification.

2) The second data collection was deductive and similarly, continued the research process by addressing the objectives and questions. This phase involved the selection of research methods, and the analysis of quantitative and qualitative data. The quantitative data from the usability test were analysed to measure and describe participants’ use of the prototype. In comparison, the qualitative data analysis involved the participation of subjects in the focus group, who engaged with the Six Thinking Hats technique to

evaluate the prototype. After analysing both qualitative and quantitative data, these results were explained to evaluate the prototype and the process.

The abductive approach is adopted and based on a combination study, which integrated exploration and evaluation. Furthermore, the data analysis for this study needed to: 1) explain

‘what’ or ‘how’ is the model, and 2) evaluate the ‘how’ and ‘what’ in relation to the prototype’s function. These need to follow the aim and objectives of this research and offer recommendations for improving design team collaboration.

Trustworthiness

Trustworthiness is necessary to assure the quality of a study and relates to the degree of confidence in the data, interpretation, and methods used (Pilot & Beck, 2014, cited in Connelly, 2016). Guba (1981) stated that the criteria for trustworthiness in qualitative methods include:

credibility, transferability, dependability, and ‘confirmability’. These criteria are required for both quantitative and qualitative methods, and are considered as follows:

1) Truth value: Credibility-Validity (internal validity),

2) Application: Transferability-Generalisability (external validity), 3) Consistency: Dependability-Reliability, and

4) Neutrality: Confirmability-Objectivity (Guba, 1981).

Guba (1981) described credibility as referring to the confidence in an accurate account or the truth of the findings carried out in a particular context; these are analogous to validity in the quantitative method. Application refers to the relevance of the data findings to a different setting; in the quantitative method, transferability is linked to generalisability or external validity. Furthermore, Connelly (2016) noted that transferability is required to provide a rich, detailed description of the context, location, and sample; this includes transparency about the analysis and trustworthiness. The term dependability is a similar concept to reliability in quantitative method and refers to the stability of the data over time and throughout the study conditions (Polit & Beck, 2014, cited in Connelly, 2016). Finally, neutrality refers to

‘confirmability’, which is similar to objectivity in the quantitative method. This concept demands the removal of the investigator and subjects’ biases (Guba, 1981).

Table 4.1 presents the keys and strategies of trustworthiness, which were adopted in this research. The first column details the keys whilst the second column lists the strategies used

by the researcher. These strategies were carried out in this research, as detailed in the Procedure column:

Table 4.1 Trustworthiness strategies used in this research

Trustworthiness Strategy Procedure

Credibility

• Varied field experience • Researcher is a professional architect with 21 years of experience

• Triangulation • The results from the first data collection were triangulated by comparing them to the existing literature and the second data collection

• Member checking • Participants checked their evaluated information in the focus group

• Participants invited into the interview checked their information

• Validity: Usability test • Groups of participants evaluated the prototype with performance measure

Transferability

• Snowball sampling and Purposeful sampling

• The samples chosen for the first and second data collections are

representative of professionals who design together in concept design

• Detailed description of sample, location, and context

• Participants’ information was presented in the overview

• Context was presented a vivid picture in literature in terms of collaboration in Thailand.

• This research is conducted in Thailand

• Data saturation • Participants’ answers were included in the data analysis

Trustworthiness Strategy Procedure

• Generalisability:

Usability test

• It is useful to understand the prototype tested by professionals and its result can generalise the finding to the wider population

Dependability

• Triangulation • The results from the first data collection were triangulated by comparing them to the existing literature and the second data collection

• Peer review • Peer review by academic researcher and team

• Reliability: Usability test • Performance measure was the

evaluation tool; the study adopted the usability test which can be used in other evaluations the same test should

produce the same results

Confirmability

• Triangulation • Methodological triangulation adopted to compare data between primary data collection methods and the literature review

• Peer review • Peer review given by academic researcher and team

• Member checking • Participants checked their evaluated information in the focus group