Research Design

The proposed approach, like any approach, cannot be tested for its “truth,” only its usefulness (Cook and Campbell 1979). I test its usefulness by empirically investigating whether measures of system usage that are selected according to the approach yield better explanations than other measures of usage in specific theoretical models. The theoretical context I use for this empirical investigation is the relationship between system usage and user task performance, which past researchers have suggested is a context in which better measures of system usage are needed (Chin and Marcolin, 2001, DeLone and McLean, 2003).

1.6.1 Empirical Tests

Table 1.5 describes the empirical tests. As Table 1.5 shows, Chapters 2, 4, and 5 use data from free simulation experiments. Each experiment examines one or more steps of the proposed approach. Chapter 3 is a conceptual paper with no empirical test.

An experimental approach was appropriate because this is the first test of the proposed approach and experiments generally provide strong empirical tests by providing greater control of external influences and rival, confounding explanations (Calder et al. 1981; Greenberg 1987). A limitation of experiments is their generalizability. To minimize this limitation, both experiments examine a common task in practice: analysts’ use of spreadsheets for financial analysis (Springer and Borthick 2004). This is a useful context for studying system usage because spreadsheets are among the most common end-user applications in practice (Carlsson 1988; Panko 1998).

Table 1.5: Empirical Tests*

Description of Empirical Test Chapter Step

examined

Design* Level of analysis

Methods Subjects Task/IT

Ch. 2 Structure,

Function

Free simulation

Individual level Usage measured by self- report questionnaire, performance measured by an independent measure Ch. 4 Structure, Function Free simulation Individual and collective level

Usage and performance measured by self-report questionnaire

Ch. 5 Method Free

simulation

Individual level Usage and performance measured by self-report questionnaire and independent measures Accounting students in a principles of accounting course in a southern US university Use Excel to build a spreadsheet model to recommend a method of financing an asset purchase

* A free simulation is an experimental design in which the values of the independent variable (e.g., usage) are allowed to vary freely over their natural range (Fromkin and Streufert 1976). This gives an insight into the relationship between the independent and dependent variable and the range over which it occurs.

1.6.2 Data Analysis

Table 1.6 summarizes the data analysis approach. I briefly explain each test below. In Chapter 2, I draw on theories of performance (Campbell 1990; March 1991) to propose that in the context of studying the relationship between system usage and task

performance in cognitively engaging tasks, each element of usage (i.e., user, task, and system) is relevant for explaining the relationship between system usage and task performance. Building on past studies (Agarwal and Karahanna 2000; DeSanctis and Poole 1994; Wand and Weber 1995), I then propose two measures of these elements, cognitive absorption and deep structure usage. I then empirically test whether these measures of system usage explain the relationship between system usage and task performance more effectively than a measure of system usage that would not be recommended by the proposed approach in this context (i.e., minutes of use).

Chapter 4 tests whether the results from Chapter 2 hold in a multilevel context. Drawing on theories of groups (Lindenberg 1997), I propose that in this theoretical context, user

Table 1.6: Data Analysis Approach

Chapter Sample Statistical Method Analytical Test*

Ch. 2 171 Partial Least

Squares (PLS)

Tests whether the relationship between usage and performance is stronger (in terms of R2_{) and more}

interpretable (in terms of direction) when usage is modeled via a measure that is tailored to the theoretical context rather than a measure that omits one/more elements (i.e., user, system, and/or task) that are proposed to be relevant in this theoretical context. Ch. 4 173 groups

633 individuals

Regression, Hierarchical Linear Modeling

Tests whether the relationship between usage and performance at the collective level of analysis and across levels of analysis (from the collective level to the individual level) is stronger (in terms of R2_{) when}

collective usage is modeled via a measure that includes interdependencies-in-use rather than a measure that omits this element (i.e., that only measures the user, task, and system elements). Ch. 5 171 representation

bias;

45 distance bias

PLS Tests multiple models of the usageÆperformance relationship using data from different methods and examines whether data obtained from the same method exhibit common methods bias (a form of representation bias) and whether the strength of the usageÆperformance relationship is significantly influenced by the degree of distance bias and representation bias in the data, i.e., β (with distance bias) ≠β (without distance bias); β (with

representation bias) ≠β (without representation bias).

* This table lists the primary analytical tests. Each chapter includes additional secondary tests to provide a complete analysis of the data.

user, task, and system elements. I then draw on past studies (Crowston 1997; Karsten 2003) to propose two relevant measures of interdependencies-in-use: coordination-in-use and

collaboration-in-use. I then empirically test whether a measure of collective usage that includes these additional measures yields a stronger explanation of the relationship between system usage and task performance than a measure of collective system usage that does not include these measures.

Chapter 5 tests the impact of the two sources of method variance (distance bias and representation bias) on the relationship between system usage and task performance at the

individual level of analysis. Utilizing the same measures of system usage and performance as in Chapter 2, I use two methods to collect data on each measure: self-reports and independent ratings. Self-reports are acquired via participants’ responses to validated instruments in a post- test questionnaire. Independent ratings are obtained via independent ratings of participants’ use of their spreadsheet program (MS Excel) and their final task performance. To enable an accurate independent coding of system usage, Screen Cam software video records are examined for a subsample of 46 user sessions. As Table 1.6 outlines, I test the impact of method variance by running multiple models of the usageÆperformance relationship that include different degrees of distance bias and representation bias and I statistically identify the degree of methods bias and distance bias within and across models.