Research and instrument development phases

Survey instrument validation has some categories typically used in MIS (management information system) literature (Boudreau, Gefen, & Straub, 2001; Straub, 1989). The most significant categories are the pretest or pilot test. Considering those categories and taking into account some recommended guidelines (Belfo & Sousa, 2011c; Carpi, Egger, & Kuldell, 2009; Cragg et al., 2002), five phases where considered on this research in order to better validate the used instrument and its previous associated research. Among the defined phases, four of them preceded the definitive survey (see Figure 34).

Figure 34: The conducted phases used to validate research and the survey instrument

First, there was a detailed literature review, which supported the first version of the instrument.

Next, a second phase which consisted on the presentation of papers at conferences or publication in academic journals. Third, there was a phase with a pretest and forth, a phase which pilot tested the instrument. Finally, after these phases, which intended to make the definition of the instrument, its improvement and validation, as much as possible, the full scale survey was performed.

(a) Literature review phase

The first version of the instrument was supported by a detailed literature review, already resumed at the earlier correspondent chapter. The measures related with the alignment construct came mainly from the Luftman’s instrument known as the Strategic Alignment Maturity (SAM). The work made by Belfo and Sousa (2013) analyzed seven other instrument proposals (Bergeron et al.,

Phase I

Literature Review

Phase III

Pretest

Phase V

Full Scale Survey

Phase IV

Pilot Test

Phase II

Conferences and Publications

RESEARCH METHODOLOGY

117

2004; Chan et al., 1997; Cragg et al., 2002; Kearns & Lederer, 2003; Reich & Benbasat, 2000;

Sabherwal & Chan, 2001; Segars & Grover, 1999) according to each dimension of BIA under SAM lenses. The six dimensions proposed by Luftman´s instrument (communications, competency/value measurements, governance, partnership, technology scope and skills) were used to analyze each instrument. This analysis revealed that Luftman’s instrument seems to provide a good (and mainly strong) level of coverage of important dimensions. Five of the six dimensions got a strong level of coverage. Although reasonably well covered, the technology scope dimension was the only one that got a good level of coverage evaluation. The empirical work previously found on SAM afforded a good starting point to get an acceptable validated instrument. Indeed, SAM seems to be one of the most promising alignment instruments in terms of content validity. Yet, aspects concerning its operationalization may require more research to be done on every component of construct validity, namely, the convergent, discriminant and nomological components (Belfo & Sousa, 2013).

The selected indicators to measure the incentive construct came from different sources. An extensive literature review allowed the identification of a relevant set of motivation assessment instruments which supported the indicators of the new instrument. Those reviewed instruments were the General Causality Orientations Scale (Deci & Ryan, 1985a), the Work Preference Inventory (Amabile et al., 1994), the Harter’s instrument (Harter, 1981), the Academic Motivation Scale, originally known as “Echelle de Motivation en Education” (EME) (R.J. Vallerand et al., 1992; RJ Vallerand et al., 1993), the Human Resources Survey (HR-Survey, 2011), the Work-Life Questionnaire (Wrzesniewski et al., 1997). and the Intrinsic Motivation Inventory (McAuley, 1989). Some indicators of these seven instruments were used or were adapted to be used by the instrument adopted at this research with respect to the incentive construct (Belfo & Sousa, 2011a).

(b) Conferences and publications phase

Conferences and academic journals with blinded reviewing philosophies represent an important role on the review process of any research work by peers and specialists. Although there are further mechanisms embedded within the process of science that supports the validation of the work of scientists and that a significant number of journals doesn't identify substantial paper flaws (Bohannon, 2013), the peer review still represents one of the most used. In a peer review process, comments are usually provided by reviewers regarding the validity of the methods used or the rationality of the data analysis techniques. Also, annotations may also be made about interpretations reasonableness made by the authors or and the quality of the writing (Carpi et al., 2009). According to an international survey of academics about peer review, the vast majority of respondents (85%) agreed that scientific communication is significantly helped by peer review and that without peer review there would be no control (Ware, 2008). Also, researchers prefer blind review (56% preferred double blind review and 25%

single-blind) compared to other peer review formats (like open review or post publication review).

Accordingly, the researcher³ submitted and presented some papers in international conferences and

3 The researcher main research interests are the alignment between business and information technology, knowledge management, business process management, project management, people & human factors in IT and technology innovation. On these research themes, he has made approximately two dozen publications, including book chapters, journal articles and papers presented at national and international conferences. A detailed list of publications can be consulted at https://sites.google.com/site/fpbelfo/.

published some others in journals, all of them adopting blind review principles. Table 13 (below) lists some of peer reviewed publications made by the researcher concerning subjects related to the research work.

Year Author(s) Publication

2010 Belfo, F. Influence of Incentive Policy in Strategic Alignment of Information Technology and Business. Proceedings of the Conference on Enterprise Information Systems (CENTERIS 2010). Communications in Computer and Information Science, 109(5), 421-430, PA: Springler. doi:10.1007/978-3-642-16402-6_44

2011 Belfo, F. and

Sousa, R.D. Employee Incentives in IT Companies: What can we learn from Google?

Proceedings of the IADIS International Conferences, Multi Conference on Computer Science and Information Systems, ICT, Society and Human Beings 2011 & e-Democracy, Equity and Social Justice 2011, 142-152 2011 Belfo, F. and

Sousa, R.D.

A Web Survey Implementation Framework: Evidence-Based Design Practices. 6th Mediterranean Conference on Information Systems (MCIS) Proceedings. Paper 43.

2011 Belfo, F. and

Sousa, R.D. Developing an Instrument to Assess Information Technology Staff Motivation. M.M. Cruz-Cunha et al (Eds.): CENTERIS 2011,

Communications in Computer and Information Science (CCIS), 2011, 220(3), 230-239. Springer-Verlarg Berlim Heidelberg. doi: 10.1007/978-3-642-24355-4_24

2011 Belfo, F. and Sousa, R.D.

Workforce Incentives at IT companies: the Google’s Case. IADIS International Journal on WWW Internet, 9(2), 69-84.

2012 Belfo, F. and Sousa, R.D.

A Critical Review of Luftman´s Instrument for Business-IT Alignment. 7th Mediterranean Conference on Information Systems (MCIS) Proceedings.

Paper 78.

2013 Belfo, F. and

Sousa, R.D. Reviewing Business-IT Alignment Instruments Under SAM Dimensions.

International Journal of Information Communication Technologies and Human Development, 5(3), 18-40, July-September 2013.

doi:10.4018/jicthd.2013070102

2013 Belfo, F. A framework to enhance business and IT alignment through incentive policy.

International Journal of Information Systems in the Service Sector (IJISSS), 5(2), 1-16. doi:10.4018/jisss.2013040101

Table 13. Peer reviewed publications about incentives, business-IT alignment or web surveys Two papers resumed some of the principles enunciated on this thesis supporting the influence of an incentive policy on the business and IT alignment. One, baptized as “Influence of Incentive Policy in Strategic Alignment of Information Technology and Business” (Belfo, 2010) was presented at the 2nd Conference on Enterprise Information Systems (CENTERIS´2010) and published on its proceedings on the Communications in Computer and Information Science´s journal. The other paper, named “A framework to enhance business and IT alignment through incentive policy” (Belfo, 2013) was published on the International Journal of Information Systems in the Service Sector (IJISSS).

Also, because the chosen research method was the survey, a paper named as “A web survey implementation framework: evidence-based design practices” was presented at the 6th Mediterranean Conference on Information Systems, a conference sponsored by the Association for Information

RESEARCH METHODOLOGY

119

Systems (AIS). This paper put together a set of important aspects to be taken into account through several phases of a web survey implementation. It proposed a framework developed to guide researchers in building a successful web survey implementation and is structured in terms of software tool selection, questionnaire design and survey administration phases (Belfo & Sousa, 2011c).

Furthermore, considering the two domains of the instrument, different papers were produced specifically about it. With respect to the domain of the incentives, one paper was presented in Rome on the Multi Conference on Computer Science and Information Systems, ICT, Society and Human Beings 2011 & e-Democracy, Equity and Social Justice 2011, about “Employee Incentives in IT Companies:

What can we learn from Google?” (Belfo & Sousa, 2011b). This paper, which received an award as been considered among the best papers of this conference, presented an interesting case about employee incentive policies in IT companies. Using a netnographic approach to look into a work life blog discussion with the participation of present and past Google employees, a total reward strategy framework is used to analyse the data and generate new insights. Later, based on this paper, another one, named “Workforce Incentives at IT companies: the Google’s Case” was produced and published on the journal IADIS International Journal on WWW Internet (Belfo & Sousa, 2011d). With these two paper major contributions, another paper was made, called the “Developing an Instrument to Assess Information Technology Staff Motivation” and reporting on the development of an instrument designed to measure the motivation of Information Technology people at their workplace. Psychology theories and work addressing intrinsic and extrinsic motivation have been studied. Some motivation instruments were reviewed and analysed. Specificities and special characteristics regarding IT workers were evidenced and combined with other more general motivation factors (Belfo & Sousa, 2011a). This last paper was the basis to the part of the instrument concerning the incentives. The selected indicators to measure the incentive construct came from different sources. The instrument proposed by Belfo and Sousa (2011) was developed according to the five dimensions of the Worldatwork framework, resulting in a set of scale items addressing several variables (WorldatWork, 2008).

With respect to the alignment domain, a paper named “A Critical Review of Luftman´s Instrument for Business-IT Alignment” (Belfo & Sousa, 2012) was presented on the seventh Mediterranean Conference on Information Systems (MCIS). Among many proposals about alignment, one of the best known and cited in the literature, showing some relevant empirical work, is the Luftman’s Strategic Alignment Maturity (SAM) assessment. So, taking it as a framework, this paper presented the examination of this and seven other proposals using the components of the SAM assessment: communications, competency/value measurements, governance, partnership, technology scope and skills. Moreover, on the sequence of this paper, another one, called “Reviewing Business-IT Alignment Instruments under SAM Dimensions” (Belfo & Sousa, 2013), was published on the International Journal of Information Communication Technologies and Human Development. Since this work evidenced that Luftman's instrument is one of the most used, promising and acceptable instruments to measure the alignment construct, this instrument was the basis to the part of the instrument used on this research concerning the alignment.

(c) Pretest phase

The third phase of the instrument development was supported by a pretest. As usual, this pretest was considered a preliminary trial of some aspects of the instrument to ensure that there are no unanticipated difficulties (Boudreau et al., 2001). This pretest, a small-scale trial of particular research components, was conducted to successively refine the instrument with the collaboration of some experts. The techniques used at pretesting aim to identify non-sample errors, proposing ways to improve or reducing those errors occurrences (Australian Bureau of Statistics, 2001).

(c.i) Approach of the pretest phase

With the objective of locating and correcting weaknesses in the questionnaire instrument, personal interviews were conducted with 11 participants. Below, Table 14 presents the demographic profiles of the experts which participated at the pretest.

Feature Number Percentage (%)

Role

Practitioner 4 36,36

Researcher 7 63,64

Education

Doctoral 5 45,45

Master 3 27,27

Bachelor 3 27,27

Specialty

Information Technology 6 54,55

Human Resources 3 27,27

Linguistic & Law 1 9,09

Business 1 9,09

Table 14. Demographic profiles of the experts participating at the pretest

The in-deep interviews with the respondents offered insight into the alignment problem and about what could be the influence of certain incentives on it. The experts were supposed to comment and evaluate the instrument in the same language that it will later be used to answer by the final respondents. So, before interviewing the experts, the questionnaire was previously translated into Portuguese.

In order to establish a high degree of equivalence at the translations of the survey instrument and according to best practices, differences between English and Portuguese languages were taken into consideration and occasionally, some cultural appropriateness (Davidov & De Beuckelaer, 2010).

Some specific aspects were taken into account in order to adequate the survey instrument translation.

These aspects were the equivalence in meaning and content (conceptual equivalence), the equivalence in grammar and syntax (technical equivalence), the readability and comprehensibility for the target population (linguistically appropriateness) and occasionally, the attention to cultural assumptions, norms, values, or expectations of the target population (cultural competency).

RESEARCH METHODOLOGY

121

Acknowledging the importance of cultural and conceptual equivalence need in a survey translation, significant time and effort was invested in the translation process. In order to get a proper translation and adaptation of measures, a fluent bilingual in English and Portuguese languages, a native speaker in Portuguese and with a nearly equal fluency in English (a logical consequence of having lived and worked in the USA during a significant number of years), was also invited to review the translation. Her comments were adequately taken into consideration.

The Appendix 3 presents the Portuguese version of the instrument, the one which was effectively used at pretest phase. The Appendix 4 presents the Portuguese version of the instrument, after the changes made in consequence of the pretest and which was used at pilot test phase. Some words or expressions are underlined at Appendix 4 to make visible the changes that were made through this process.

The design of the interviews considered an evolution from an open-ended to a highly structured format (Straub, 1989). The interviews had three sequent segments (see Figure 35). The first segment consisted in interviews conducted in an open-ended general discussion format. Secondly, interviews gradually moved to a semi-structured format and lastly, they moved to a highly structured analysis of each item of the draft instrument.

Figure 35: The three sequent segments adopted in the interviews design of pretest phase Each interview started by introducing the subject of the research and encouraging the participants to be discursive. They were provoked by a simple request for their opinion about the importance and characteristics of the alignment between business and IT and about incentives.

Participants were also asked about the possible influence of incentives on the alignment. The general perception of the interviewees was that incentives should influence the alignment. When asked about the main characteristics of the alignment and incentives (content validity), interviewees’ suggestions about these constructs were, essentially, already considered on the design of the presented instrument.

Secondly, in the semi-structured segment, questions from the interviewer focused on the proposed dimensions of the alignment and the incentive. Clarifications about the constructs and their dimensions were undertaken in this segment (content validity). Interviewees were also asked if the proposed questions actually measured part of each construct (construct validity).

The Appendix 2 presents the English version of the instrument presented to these participants.

Yet, as it was previously mentioned, since the universe is composed of Portuguese companies and the native language of the respondents is Portuguese, the questionnaire was translated into Portuguese and answered in that language.

On one hand, participants were highly challenged to propose changes to questions in order to better capture a property of a construct. On the other hand, interviewees were also asked about the clarity of the research questions. They were encouraged to propose an alternative formulation of a question if they feel it could be better understandable. The accepted changes on items tried to ensure a better correspondence and isomorphism between the syntactic and the semantic form of linguistic expressions. Some changes needed a deeper reflection, others, more obvious, are supposed more commonly accepted and were almost immediately adopted.

The conversations with the experts were very enriching, not only by qualitative contributions that were immediately expressed either by the successive incremental improvements that have happening over the exchange of views with the expert and, also later, after a more careful analysis of the issues, timely and briefly noted.

This section is organized as follows below. Firstly, there are some generic change proposals that are presented. The reason why these changes appear on first is because the rationale behind this type of changes affects not only one item, but a group of items. So, the first adjustments to be discussed are about items related with issues like the reverse coding and the results of the content validity ratio test which was previously made. Secondly, the results of the content validity ratio test are analyzed.

And third, each item that was challenged by an expert is analyzed. The order of this last analysis is the same order the items have on the instrument. Each challenged item is discussed and a decision is made and presented about each one wording.

(c.ii) Reverse coded items

Three of the items were initially reverse coded (item I04, I14 and I21). The original objective of this procedure was to force respondents to pay attention to the questions before answering. At the proposed instrument, only a short number of questions were worded such that high values for the construct are valued by low scores on the item. For example, the following example is a translation of the item I14 of the initial instrument using this reverse coded procedure:

“It is difficult for me to get time off because of maternity/paternity or sabbatical reasons”

The original idea consisted on inverting the answers on this kind of items. The selected option (one of the five options, from 1 - "strongly disagree" to 5 - "strongly agree"), should be converted into precisely its symmetric value.

RESEARCH METHODOLOGY

123

Although the reverse coding may have some benefits and so, it has a significant number of supporters, there are also some problems with this approach. Several interviewees manifested that reverse coding could confuse respondents. As a matter of fact, the use of negatively worded and consequently reverse-scored items is not consensual and has motivated abundant debate. It has strong supporters and opponents. On one hand, supporters argue that reverse-scored items may reduce response set bias. On the other hand, opponents argue that the usage of few of reverse-scored arbitrarily distributed within a measure may have a damaging effect on psychometric properties of a measure (Hinkin, 1998). Because some interviewees believed that negatively worded items may create confusion, and since it seems this approach has pointed out problems with internal consistency, factor structures, and other statistics (Barnette, 2000; Eys, Carron, Bray, & Brawley, 2007; Roszkowski &

Soven, 2010), it was decided to convert the negatively worded items into positively worded items.

(c.iii) Content validity test

Content validity may be defined as the degree to which items in an instrument reflect the content universe to which the instrument will be generalized (Straub, Boudreau, & Gefen, 2004). Content validity shows that a sample of the universe in which the investigator is interested is covered by the test items (Cronbach & Meehl, 1955).

Straub (1989) summarizes the content validity concept with the following question:

Are instrument measures drawn from all possible measures of the properties under investigation?

Figure 36 presents a symbolic model of the content validity of an instrument. It considers that the domain of a given construct can be represented by a circumference and that an instrument tries to measure the most important properties of the concept, by using several items to do it. Each item of the instrument represents one point on the universe of all possible measures of the concept and captures a fragment of it.

Literature review and expert judges or panels are typically employed to established the content validity (Straub et al., 2004). As it was suggested by Cronbach (1971) (as cited in Straub et al., 2004), a review process should involve experts in the field, familiar with the content universe, that evaluate versions of the instrument again and again until a form of consensus is reached.

Literature review and expert judges or panels are typically employed to established the content validity (Straub et al., 2004). As it was suggested by Cronbach (1971) (as cited in Straub et al., 2004), a review process should involve experts in the field, familiar with the content universe, that evaluate versions of the instrument again and again until a form of consensus is reached.

In document Influence of incentive policy in the alignment of business and information technology (Page 141-187)