An Enhanced Operationalization of Effective Use

Our research effort to operationalize the measures for EU as presented so far was successful. However, the mix of reflective and formative measures, and concern raised by item raters, respondents, and other researchers motivated us to improve on this initial effort. While developing the measures for EU, we experienced several challenges. EU introduces a much higher level of abstraction to the IS use concept. While this represents a significant contribution to the understanding of the underlying mechanisms of EU, it makes the operation-alization of EU much more demanding. The operationoperation-alization of an abstract phenomenon is a challenging mental procedure (Recker, 2013), particularly for complex, multidimensional concepts such as EU. Due to the complex and abstract nature of EU, many people involved in the research process (e.g., content validity raters or survey participants) seemed to have had difficulties in understanding elements of the theory such as the concept of representation. Hence, the goal of developing a generalizable EU theory is ever more challenging and research should always aim to contextualize to enable respondents to relate to the questions (Burton-Jones & Grange, 2013). Despite including this aspect in our operationalization efforts, we encoun-tered many raters and respondents, who perceived the questions relating to the sub-dimensions TI and RF as system properties or tried to understand them that way.

This is contrary to Burton-Jones’ and Grange’s (2013) conceptualization of TI, RF, and IA as assessments of use (i.e., as behaviors), not as properties of the system or the user. Burton-Jones and Grange (2013) explicitly state the differences between the EU dimensions and existing constructs such as perceived ease of use (Davis, 1989) or information quality (DeLone & McLean, 1992):

“Although our constructs bear some similarity to TAM constructs, there are several differences (e.g., our constructs reflect observable behaviors rather than user perceptions). More importantly, TAM explains IT acceptance whereas our theory explains what people need to do to use systems more effectively and increase their performance.” (Burton-Jones & Grange, 2013, p. 652)

“Representational fidelity is not the same as information quality. For example, one difference is that information quality is a property of a system whereas our concept of representational fidelity is a property of use.” (Burton-Jones & Grange, 2013, p. 652)

While Burton-Jones and Grange (2013) clearly demonstrate why the EU dimensions are behaviors and how they differ from perceptions of system properties on a conceptual level, this distinction is less clear when it comes to operationalizing and measuring these constructs and in the interaction with respondents. Fur-thermore, operationalizations of richer concepts of IS use in previous IS research have generally measured

7Parts of this section were accepted to the Pre-ICIS SIGADIT Workshop 2016, Dublin. Only the abstract was published. The Paper won the workshop’s Best Paper Award. This section is also part of a recent submission to the Information & Management Journal:

Haake, P., Schacht, S., Lauterbach, J., Gnewuch, U., Koegel, C., Mueller, B., Maedche, A., 2017. Operationalization and Measurement of the Concept of Effective Use. Information & Management, submitted.

usage behavior based on perceptions, such as perceptions of using a particular feature of a system (e.g., deep structure use):

 “When I use Excel, I use features that help me analyze the data.” (McKnight, Carter, Thatcher,

& Clay, 2011, p. 22)

 “I use the “feedback” feature to provide input to others on their work.” (Sykes & Venkatesh, forthcoming, p. 57)

This raises the question of whether the EU dimensions – although they are behaviors on a conceptual level – can be empirically measured based on perceptions that represent the product of a user’s behavior (i.e., the results of a user’s interaction with the system). Consider as an example the concept of RF which constitutes the extent to which a user is obtaining faithful representations from a system (Burton-Jones & Grange, 2013). Most likely, a user’s perception of the quality of the system’s representations is determined by his/her own ability to obtain faithful representations from it. For example, users who state that the system provides incorrect information reveal their inability to obtain correct information from the system. As a result, their perception of the system can be used as a proxy measure for RF. In addition to that, this ap-proach provides benefits from a practical standpoint since it is arguably much easier for someone to grasp the perceived quality of the system’s representations than to go through the thought process whether his/her interaction with the system results in the obtainment of accurate representations. While Burton-Jones and Grange (2013) have illustrated that EU and its sub-dimensions differ significantly from established measures, the comparison of measures and the insights from the item development process raise the much broader question of whether the operationalization of core EU constructs results in reconceptualizations of other concepts that represent system properties (e.g., TAM constructs).

To better illustrate this conjecture, Table 21 provides a comparison of the definitions of EU sub-dimensions with existing measures for other constructs. This provides initial evidence for the assumption that the EU dimensions can be linked to the perception of system properties, at least on an empirical level. Furthermore, we included an exploratory factor analysis (EFA) with SPSS Version 22 to test the initial measurement models for unidimensionality (Urbach & Ahlemann, 2010). We performed an EFA based on principle-component-analysis (PCA) because it is deemed the appropriate approach for exploratory research (Pett, Lackey, & Sullivan, 2003). Furthermore, we selected extraction based on eigenvalue >1 and promax rota-tion (kappa = 4), as recommended for human behavior research (Costello & Osborne, 2005). For the panel data set, all items converged in their corresponding construct. For BANK, several items did not converge in their corresponding construct (see the results of the final EFA in Appendix F). This added to our percep-tion that we should reevaluate and, when necessary, adapt our initial operapercep-tionalizapercep-tion. Beyond the points that we have already mentioned, we also wanted to address low loadings on some items (e.g. TI5), the high Cronbach’s alpha for IA and the formative nature of some of the RF items (RF1-3; RF5) to allow for a more convenient measurement.

Table 21: Similarity of Effective Use Measures with Other Constructs

Construct Definition Measures for Related Constructs

Transparent

Perceived ease of use (PEOU) (Davis, 1989, p. 340):

The system provides output that is exactly what I need.

Item Development

Thus, we decided to start another item development and data collection effort to investigate whether the EU-sub-dimensions are similar to measures of perceived system characteristics and to improve the individ-ual items. We developed completely reflective measures to reduce the complexity of the measurement, following the suggestions of current research (Hair, Hult, Ringle, & Sarstedt, 2014; MacKenzie et al., 2011). According to Hair et al. (2014), reflective measures represent the consequences of an underlying construct. To understand the consequences in more detail, we operationalized the items in the role of em-bedded researchers and altered item versions between contextualization and generalization, while testing the items with end users. For our pre-study, we conducted embedded research in a buildings material com-pany (BMC) in Central Europe that is currently setting up customer service centers and the accomcom-panying call-center system. One of the authors was an embedded researcher at BMC for half a year, while he got to know the company and developed the items (see Table 22). Thus, we incorporated the lessons learned from the previous operationalization effort to develop a reduced number of focused and pre-tested items. Never-theless, we again developed the new set of measures by generally following procedures suggested in IS literature (MacKenzie et al., 2011), as we did for the previous item set (see Appendix G).

We selected a Multi-Channel-Fashion-Retailer (MCFR) as the case site for our study, which has eleven different brick-and-mortar stores as well as a sizeable online shop. 525 personnel of full-time and part-time shop assistants as well as sales managers work in the different stores. The shops assistants routinely use a shop-assistant system (SAS) in their jobs. SAS is a software installed on desktop PCs, which can be found across all branches of MCFR. In addition, the desktop PCs are equipped with a scanner for product codes.

The SAS provides a wide array of features, which support the shop assistants and their first-level manage-ment. The SAS allows accessing the firm’s intranet, which works like a wiki-system and includes all rules and regulations relevant for employees. This ranges from human resources topics, information about the features and the use of software, to specific sales-information and company policies towards customers.

Furthermore, it includes a direct link to the web-shop of the company, which is understood as a different sales outlet (i.e. separate prices etc.). The core features of the system are designed to support the shop assistants

in their direct contact with the customers and allow them to deal with stock levels, loyalty card information, specific service offerings, and complaints management. Additional features designed to support the pro-cesses in the sales organization can also be found in the key index tab of the SAS. Thus, it is evident that SAS provides a wide array of features with very different purposes. However, the features need to be used effectively by shop assistants to take informed action for customers or for internal processes.

Table 22: New Effective Use Measures

Construct ID Item

Transparent interaction

TI1 The system is difficult to work with. (reverse) TI2 The system is easy to navigate.

TI3 The system allows easy access to its [content].

TI4 I can successfully interact with the system.

TI5 The system facilitates the access to its [content].

Representa-tional fidelity

RF1 The system's content is dependable.

RF2 I trust the system's representation of the content.

RF3 The system correctly reflects the real business object (e.g. customer, level of stock, services).

RF4 I am confident that the system provides a correct representation of its [data].

RF5 I do not have to crosscheck the data in the system.

Informed action

IA1 The system's content allows me to make better decisions in my job.

IA2 I act upon information in the system because they help me to do a better job.

IA3 Information drawn from the system's content allow me to avoid making mistakes.

IA4 I can leverage the system's content to avoid acting on false information.

(dropped)

IA5 The system's [content] allows me to correctly execute [my tasks].

Data Collection

We measured all items using a seven-point Likert scale (1=strongly disagree ... 7=strongly agree). Before we started the data collection, the embedded researcher at the Multi-Channel-Fashion-Retailer (MCFR) discussed the survey design and the contextualization of the items with several other employees at the firm.

S/He discussed the items and survey design mainly with the main trainer for SAS and with the responsible IT-department employee. Furthermore, s/he also discussed the contextualization of the items with a select group of two department heads, one deputy department, and the employees from the works council. For this purpose, we had to translate the items into German. We controlled the accuracy of this translation by having another person directly translating the German version back into English and a third independent person evaluating whether the items have the same meaning. Subsequently, we conducted several pre-tests of sections of the survey with end-users of SAS on the shop floor. All the people that we contacted provided feedback on the items and the survey structure, which we used to further iteratively develop the survey instrument. Based on their feedback, we removed and redesigned several items and modified the instruc-tions to reduce complexity and make the survey easier to understand.

It became evident that shop assistants as well as all other sales personnel understood IA based on the defi-nition by Burton-Jones and Grange (2013) as all behaviors that followed from system use and enabled them to provide a service to a customer in a better way. They believed that their ability to obtain IA is influenced by their perceived TI and RF of the system. We added some controls for individual factors that could in-fluence the ability to use a system and therefore added controls for prior IS use frequency (Wilson, Mao, &

Lankton, 2010) and individual use experience (Kim & Malhotra, 2005). Furthermore, we added the controls of perceived ease of use (PEOU) (Davis, 1989; Venkatesh & Davis, 2000) and information quality (InfQ) (Rai, Lang, & Welker, 2002; Sasidharan et al., 2012) to enable the cross-validation as outlined above (see Appendix H). We collected 169 responses overall. 22 respondents did not answer the complete survey and we therefore removed their answers. Subsequently, we checked several quality criteria to receive our final data set. We had to remove another 32 responses based on quality issues. For the removal of invalid re-sponses, we followed again a two-step approach. First, we used the implemented quality measure in the online survey software Questback⁸. We again decided to remove all responses (17 responses) with a quality value smaller than 0.25. Second, we considered the reversely coded item in our survey and removed 15 completed questionnaires based on the analysis of the reversely coded item (resulting in 115 valid re-sponses; effective response rate: 22%). Table 23 provides an overview of the descriptive statistics for the data collection. Appendix I shows that this sample is representative for the employees at MCRF.

Table 23: Descriptive Statistics of New EU Data Set

Statistic Values

Number of participants/ valid data sets 169 / 115 Number of female participants 81 (70%) Number of male participants 34 (30%)

Data Analysis and Results

This time we also performed the quantitative data analysis with a partial least squares structural equation modeling (PLS-SEM) approach, using SmartPLS 3.2.6 (Ringle, Wende, & Becker, 2015) and again fol-lowed the recommendations by Urbach and Ahlemann (2010) and Hair, Hult, Ringle, and Sarstedt (2014).

We conducted and exploratory factor analysis (EFA) with SPSS Version 22 with the same settings as in our previous effort. With the exception of IA6 and InfQ4/5, all items converged in their corresponding construct (see the results of the final EFA in Appendix J). While Smart PLS is the appropriate analysis tool for our explorative research, there is unfortunately no global measure for the goodness-of-fit of the different models that can be used (Hair et al., 2014). Hence, we evaluated both models separately.

Table 24 shows satisfactory values for CA and CR (> 0.7) (Nunnally & Bernstein, 1994). CA values range from 0.82 to 0.92, whereas CR values range from 0.88 to 0.94. Table 25 also shows satisfactory values for CA and CR (> 0.7) (Nunnally & Bernstein, 1994). CA values range from 0.82 to 0.89, whereas CR values

8https://www.questback.com/ (Accessed on June 20^th, 2017)

range from 0.88 to 0.92. Appendix J shows that almost all reflective items load highly on their parent constructs (> 0.708) (Chin, 1998). However, the loadings for IA5 (0.638), as well as the loadings for TI4 (0.694) do not load highly on their construct. However, these values are still larger than the suggested threshold of 0.6 (Gerbing & Anderson, 1988) for exploratory research as long as the primary loading is on the correct factor (Gefen & Straub, 2005). Therefore, we included these items in our further analysis.

Table 24: Results for Reflective Measurement Model (Effective Use New Operationalization)

Constructs CΑ CR AVE TI RF IA

TI 0.83 0.88 0.59 0.80

RF 0.92 0.94 0.75 0.41 0.87

IA 0.82 0.88 0.65 0.39 0.25 0.77

As Table 24 shows, AVE values are above the suggested threshold value of 0.5 for all constructs (Fornell

& Larcker, 1981). We measured the lowest AVE value for TI (0.59). For the alternative model, Table 25 also shows AVE values above the suggested threshold value of 0.5 for all constructs (Fornell & Larcker, 1981). IA has the lowest AVE value (0.65). We assessed discriminant validity based on the Fornell-Larcker-Criterion (Fornell & Larcker, 1981) (Table 24/25) and item cross-loadings. Therefore, we can confirm the validity of the reflective measurement model. Item loadings in Appendix J illustrate that all items loaded higher on their designated construct than on any other construct. All cross-loadings were lower than 0.7 with a gap of at least 0.1 between cross- and primary loading (Gefen & Straub, 2005).

Table 25: Results for Reflective Measurement Model (Effective Use New Alternative Operationalization)

Constructs CΑ CR AVE PEOU InfQ IA

PEOU 0.87 0.91 0.59 0.85

InfQ 0.89 0.92 0.75 0.50 0.87 IA 0.82 0.88 0.65 0.40 0.45 0.80

Finally, we employed Harman’s single factor test to assess the common method bias (Podsakoff et al., 2003). After we had loaded all variables in an EFA, we examined the unrotated solution. If a single factor emerges from an unrotated factor solution or one general factor accounts for the majority of the covariance in the variables, there might be risk of a common method bias (Podsakoff et al., 2003). However, this was not the case in our study, which means that common method bias is not an issue in this study.

Subsequently, we assessed the structural model (Figure 13). The variance explained in the original EU model is 6.2% for RF and 26.0% for IA. Variance explained in the alternative model is 17.2% for InfQ and 13.9% for IA. This indicates that both research models have some level of explanatory power (Chin, 1998).

However, the explanatory power for IA in the original model is higher than the explanatory power in the

alternative model, concerning a linear regression. We used the bootstrapping procedure with 5000 resamples to determine the direction and significance of the paths (β) within the structural model. As illus-trated in Figure 13, TI (β = 0.250, p ≤ 0.01) positively affects RF and positively affects IA (β = 0.309, p ≤ 0.001). RF positively affects IA (β = 0.336, p ≤ 0.01). The relationships in the alternative model are of the following nature: PEOU (β = 0.502, p ≤ 0.001) positively affects InfQ and positively affects IA (β = 0.235, p ≤ 0.05). InfQ positively affects IA (β = 0.335, p ≤ 0.01). In contrast to the model for the original opera-tionalization above, we observe for our new adapted operaopera-tionalization of the original EU model as well as for the alternative EU model based on PEOU and InfQ that the relationship between TI and IA is significant, with and without the potentially mediating variables (Hair et al., 2014) of RF and InfQ. In the case of the original model, there is a significant effect of TI on IA (b= 0.402, p < 0.001) without RF in the model.

Original EU model

ƒ² (RF → IA) = 0.143 ƒ² (TI → IA) = 0.121 ƒ² (TI → RF) = 0.067

Alternative EU model

ƒ² (InfQ → IA) = 0.112 ƒ² (PEOU → IA) = 0.055 ƒ² (PEOU → InfQ) = 0.336

*** p < 0.001; ** p < 0.01; * p < 0.05; n.s. = not significant

Figure 13: Comparative Assessment of the Structural Model for New and Alternative Measurement of Effective Use

When we introduce RF as a mediator in the model, TI still has a significant direct influence on IA (b=

0.309, p < 0.001). However, RF also has a mediating effect of 0.09 with a 95% confidence interval of 0.02 to 0.17 altogether indicating partial mediation. For the alternative model, there is a significant effect of PEOU on IA (b= 0.407, p < 0.001) without RF in the model. When we introduce InfQ as a mediator in the alternative EU model, PEOU still has a significant direct influence on IA (b= 0.235, p < 0.05). However, RF also has a mediating effect of 0.17 with a 95% confidence interval of 0.06 to 0.31 altogether also indi-cating partial mediation. Hence, we can accept H1 and H2 for both models, while we have to reject H3 for both models as well. Nonetheless, in both models RF and InfQ, respectively, partially but not fully mediate the relationship.

We evaluated the effect size based on Cohen’s f² (Cohen, 1988). A small effect has an f² of about 0.02, a medium effect of about 0.15, and a large effect of about 0.35 (Cohen, 1988). The effect of TI on RF is small (f² = 0.067), while the effect of RF on IA is rather medium (f² = 0.143). The effect of TI on IA is also medium-sized (f² = 0.121). The results of the evaluation of the alternative model of EU show an effect of

PEOU on InfQ is almost large (f² = 0.336), while the effect of PEOU on IA is rather small (f² = 0.055).

However, the effect of InfQ on IA is rather medium-sized (f² = 0.112). Additionally, we evaluate the mod-els’ capabilities of predicting dependent variables with a blindfolding procedure (7 cases). The Q² values are 0.144 for IA and 0.042 for RF in the case of the original EU model, indicating sufficient predictive relevance (Hair, Ringle, & Sarstedt, 2011). For the alternative EU model, we obtain Q² values of 0.139 for IA and 0.172 for InfQ, indicating sufficient predictive relevance (Hair et al., 2011).

Conclusion

Drawing on the recent conceptualization of EU (Burton-Jones & Grange, 2013), we initially developed a first measurement instrument (see Table 20) and conducted an empirical test to advance our understanding of individuals’ effective use in an enterprise system implementation (post-adoption) context (at BANK) and in general (within a panel). However, we felt compelled to challenge our initial results in light of the feedback that we received from respondents and other researchers during the data collection effort. Conse-quently, we conducted another item development (see Table 22; see Appendix G) and data collection effort with these issues in mind and tried to operationalize the measurement model in response to the feedback

Drawing on the recent conceptualization of EU (Burton-Jones & Grange, 2013), we initially developed a first measurement instrument (see Table 20) and conducted an empirical test to advance our understanding of individuals’ effective use in an enterprise system implementation (post-adoption) context (at BANK) and in general (within a panel). However, we felt compelled to challenge our initial results in light of the feedback that we received from respondents and other researchers during the data collection effort. Conse-quently, we conducted another item development (see Table 22; see Appendix G) and data collection effort with these issues in mind and tried to operationalize the measurement model in response to the feedback