Construct Validity

Construct validity refers to the use of correct operation measures for the concepts being studied (Yin, 2003b). According to Yin (2003b), this can be particularly problematic in case-study research, where operational measures used to collect the data are often not well developed. This study applied the three tactics for increasing construct validity in case studies outlined by Yin (2003b). First, this research used multiple sources of evidence, with data collected from the OCL forum (discussion forum transcript), participant interviews (interview transcripts), and focus groups (focus group transcript) to provide a number of different lines of inquiry. Second, this research provides a chain of evidence from the different sources of data collected to the findings in Chapter 6, by

127

providing direct quotes from the sources to support interpretations made. Third, the findings from the case-study report were reviewed by SBOMs who participated in the research.

Learning is a difficult concept to measure. To ensure construct validity in this case study it, was important that the concept be clearly defined and an operational measure be used to assess. A plethora of approaches have been designed to assess learning in an ODF; however. they have been limited largely to user self-reports, descriptive statistics based on the number of posts, posting rates, frequency counts and other quantitative measures (Gunawardena et al., 1997; Henri, 1992; Marra, Moore & Klimczak, 2004; Mason, 1992; Strijbos & Stahl, 2007). This research aimed to move beyond the dominant quantitative measures and analyse the discussion forum transcript to determine if learning does occur. In order to analyse the discussion forum transcript qualitatively, it was important the measure used also reflect the principles of social learning theory that underpin this research.

Henri‘s (1992) computer and content analysis model was considered because it is based on social learning theory; however, the detail for coding discussion forum data was limited, and this was considered a weakness that would be difficult to overcome. More recent work in measuring ODF has been done by Pozzi et al. (2007). Their model builds on Henri‘s (1992) original model, but provides significantly more detail about what constitutes the five different levels of learning they propose. While Pozzi et al.‘s (2007) model is designed to measure formal university-based learning, the underlying assumptions are embedded in social learning theory and this theoretical underpinning was thought an important factor in choosing a measure for SBOMs‘ participation and learning. Thus, the Pozzi model was considered an appropriate measure of participation and learning that could be applied to the ODF proposed in this research.

Pozzi et al. (2007) outline dimensions and indicators that are broad, enabling the framework to be adapted to fit the informal nature of this study‘s ODF. The framework has five dimensions: participative, interactive, social, cognitive and teaching. Each of these is outlined in Table 5.1. The framework uses predominantly qualitative measures for assessing participation and learning. However, it does support participation with

128

counts of posts. Pozzi et al. (2007) provide clarity for how operational measures of participation and learning in an ODF will be assessed in this study. It also allows easier replication, because the process is well documented.

5.4.2 External Validity

External validity (or generalisability) refers to the extent to which the case study findings can be extended to other situations (Huberman & Miles, 2002; Merriam, 1998; Yin, 2003b). External validity has been problematic for qualitative researchers for some time (Merriam, 1998) and has been a major barrier in doing case studies (Yin, 2003b). However, as Cresswell (2009) asserts, the intent of qualitative inquiry is not to generalise findings outside those in the study. This is supported by Maxwell (2005) who states ‗external generalisability is often not a crucial issue for qualitative studies. Indeed, the value of a qualitative study may depend on its lack of external generalisability‘ (p. 115).

There are procedures that can be used by the researcher to strengthen external validity. These include within a single case sampling from a subunit, or using multiple cases to study the same phenomenon (Merriam, 1998). In this study, external validity was strengthened by using multiple case studies and by sampling a number of SBOMs (subunits) in the ODF case. Limited generalisability can be seen as problematic in qualitative case-study research. However, its opposing strength of providing a rich, thick description of a particular group in a given situation was seen as important. That is, the value of this case study is the rich, thick description and themes developed from SBOMs participating and learning in the context of using an ODF.

5.4.3 Validation Strategies

In order to ensure confidence in the methodology and analysis, the following three validation strategies were used: triangulation, rich, thick descriptions in the findings, and the presentation of alternative explanations. Each of these is discussed here.

129

Triangulation was used to ensure the findings are trustworthy. Triangulation is the use of multiple methods to help develop an in-depth understanding of the phenomenon being studied (Denzin & Lincoln, 2005). Multiple methods add rigour, breadth and depth to the inquiry, which was important to help ensure that a full and complete picture of SBOMs‘ participation and learning on an ODF was developed.

Triangulation of the data in this research occurred in three ways. First, the data were gathered from different sources and represented different data types. Data were collected through the ODF software, which established the SBOMs‘ pattern of use within the discussion forum throughout the study. The SBOMs‘ and the researcher‘s contributions to the ODF were archived, providing the opportunity to examine material in detail using Pozzi et al.‘s framework (2007) (see Table 5.1). Interviews with SBOMs were transcribed and analysed in conjunction with the ODF transcript. Focus groups with SBOMs were transcribed to assess why SBOMs chose not to take part in the discussion forums. Second, data were triangulated through the use of multiple data collection methods. Quantitative methods were used to gather information regarding SBOMs‘ participation patterns on the discussion forum. Qualitative methods included the analysis of the ODFs, interviews and focus groups. Third, the findings were triangulated through theoretical checking and double-checking to ensure that theories could be replicated and confirmed through examination of different perspectives within the data.

The second validation strategy used in this research was the use of rich, thick description to communicate the findings, which, according to Cresswell (2009), transports the reader to the setting allowing the reader to make decisions about the transferability of the findings beyond that described. This thick description makes the findings richer and more realistic. The findings are shown in Chapters 6 and 7.

The third validation strategy applied was the presentation, where possible, of negative or discrepant information (Cresswell, 2009). In particular, the use of focus groups with SBOMs who did not participate in the ODF enabled the researcher to develop a more complete understanding of the factors that both support and prevent participation. This allowed the differing perspectives of SBOMs to be more accurately reflected.

130

Discussion of contrary information about various themes regarding participation added to the credibility of this case study.

Details of the analysis and inter-rater reliability are discussed in Section 5.8.2.

5.4.4 Reliability

Reliability refers to the operations of the study, such as the data collection procedures being able to be repeated with the same results (Yin, 2003b). In order to do this, the procedures of this case study were documented in as many steps as possible (Cresswell, 2009). Other reliability procedures used in this study included the checking and re- checking of transcripts to ensure that they did not contain transcription errors. During analysis of the interview transcripts, the codes were described on initial capture. Then to ensure that the meaning of the codes did not shift during the coding process, data were constantly compared with the initial definitions as suggested by Cresswell (2009).

In addition, the reliability of information in this case was increased by maintaining a chain of evidence (Yin, 2003b), the principle being that an external observer is able to trace the steps from research questions to conclusion or, alternatively, from conclusion back to the initial research question.

5.5 Ethical Considerations

Ethical considerations regarding this research focus on four issues; first, avoiding harm to respondents; second, informed consent; third, privacy, anonymity and confidentiality; and fourth, analysis and reporting (Babbie, 2002; Leedy & Ormrod, 2005; Sarantakos, 2005; Saunders et al., 1997). Each of these four issues including how they were managed in the design of the current study is discussed in this section.

First, this research was designed to ensure participants were not exposed to physical, psychological or legal harm (Sarantakos, 2005). Although the risk was small, there was some potential that SBOM participants could reveal sensitive information regarding themselves or their business during the computer mediated discussion. They could

131

reveal information that could embarrass or endanger their small business. To avoid this, only the SBOM‘s first name and a brief description of the type of business (not the business name) were used, during the ODF.

Second, all participants in this research took part on a voluntary basis and completed an informed consent form prior to participating in this study. All SBOMs contacted were able to accept or refuse to take part. Those who chose to participate in this study did so knowingly and voluntarily, by reading and signing the informed consent letter, which outlined the research and the possible risks to the individual and their business. The informed consent letter described the nature of the research project, together with the type of participation required (Babbie, 2002; Cresswell, 2009; Sarantakos, 2005; Saunders et al., 1997). A copy of the informed consent letter is shown in Appendix A.

The letter explained what participation in the study would require and outlined that the data from the discussion forum would be collected from the online site. The letter also explained that SBOMs would be asked to take part in an interview, and clarified that their privacy would be protected because no participants would be revealed in any resulting publications. SBOMs were informed that their participation in this study was voluntary and they could withdraw at anytime without penalty. All SBOMs were made aware of whom they could contact if they had any concerns regarding the research. As such, the names of the SBOMs shown in the findings chapters have been changed to protect their identity.

Likewise, participation in the focus group was voluntary. The focus group invited SBOMs who had not participated in the OCL forum to take part in a discussion (focus group) about prior training. To encourage attendance drinks, food and a $50 gift voucher were offered. Invitations to attend one of two focus groups are shown in Appendix B. Third, this research respected the participants‘ right to privacy, by ensuring that the nature and quality of participants‘ responses remained strictly confidential (Sarantakos, 2005). Confidentiality was provided by ensuring that the name of the respondents do not appear on the research interviews or online forum data (Sarantakos, 2005). According to Saunders et al (1997) use of the discussion forums during data collection can lead to the possibility of ethical issues in regard to

132

confidentiality. To avoid harm participants used only their first name and a generic description of their business in the OCL forum. That is, when they introduced themselves online they used their first name, and did not reveal the name of their small business, but instead gave a generic description the type of business that they owned. They also used generic email addresses to access the forum, to avoid revealing their business or personal identity, thus enabling them to ask questions and reveal answers without revealing their identity. In addition, informed consent forms were kept separate from the research data to ensure that names could not be linked (Sarantakos, 2005).

Fourth, ethics were adhered during the research, the writing and dissemination of this research, by using pseudonyms and presenting a complete and honest report of the findings. Analysis and reporting of the findings of this study uses pseudonyms, as suggested by Cresswell (2009), thus quotes and names are used in the findings but the name used is a pseudonym, thus helping to ensure that the information provided cannot be linked with the respondent.

5.6 The Case

This section discusses the goal of this research and the planned research design. It describes how SBOM participants were recruited for the research, and the data collection and analysis process. In addition, it also addresses the specifics of how, reliability and validity, generalisability, triangulation, relevance and rigour are addressed in the data collection and analysis.