Sampling and Data Collection

Data was collected only once, over a two-week period. This can be regarded as a one-shot or cross-sectional survey (Sekaran, 2003).

As similarly carried out in experiment 1, the materials were first pretested with 2 Web designers and 3 senior MIS academics to ensure that the instrument was unambiguous, possessed face validity and that the prototypes were realistic.

For the conjoint study, a total of 102 postgraduate students were invited to participate in the study. Participants who were both familiar and unfamiliar with the balanced scorecards participated in the study.

Prior to sending out email invitations, an application for ethical consent was submitted to the University of Surrey, Board of ethics (see Appendix A9). Following the successful review of the proposal for the study, an email invite (see Appendix A6) was sent out through the University’s email system to only postgraduate students at the School of Management. The experiment was conducted online. Therefore, following a short introduction to the experiment, participants were asked to complete some demographic questions and a series of questions measure awareness and familiarity of the subjects to balanced scorecards (see Appendix A7). Each respondent was thereafter presented with the twenty prototypes- each identical to the first but with different arrangement of the information features, which they evaluated by ranking from the most preferred to the least preferred. A total of 102 completed set of responses provided 2,040 profile evaluations (20 profiles x 102 respondents) for the purpose of statistical analysis. A £20 gift voucher was given out as incentive to 20 participants (selected at random), who completed the experiment.

7.6.9 Summary Using Recommended Guidelines for Conjoint Analysis

The recommended guidelines for conjoint analysis study were closely observed throughout the data collection process and are summarised in Table 7.3

Steps This Study Select a model of

preference

Part-worth: A part-worth function model was adopted. This is because the three levels of critical display of information systems adopted in this study- critical, important, and desirable- as well as the two modes of presentation formats- graphs and tables- can be considered as categorical (Green and Srinivasan, 1978).

Data collection method

Full-profile: A full-profile approach was adopted due to the number of factors being less than six. Thus, all the four information features were presented at a time without any problem of overload.

Data collection design

Fractional Factorial: A fractional factorial design was adopted. This method generated twenty hypothetical Web page prototypes of which two were holdout profiles. These holdout Web page profiles made it possible to determine how consistently the conjoint model could predict user preferences for new balanced scorecard prototypes that were not evaluated in the survey and as such serve to validate the fitted conjoint model (SPSS, 2007).

Stimulus presentation

Visual Presentation: A visual presentation is most suitable for a study of this nature as it deals with the visual order of presentation of features.

A brief definition of each feature was first presented to each respondent at the beginning of the interview, followed by the evaluation of the prototypes

Data collection procedure

Online Experiment: Participants took the experiment online. The survey was scripted in HTML format using ASP program and the data was collected using MS Access database in the back end

Estimation Scale Non-metric: The author favoured the use of ranking scales in the current study because of their perceived practicality when comparing stimuli. Respondents were therefore asked to rank the prototypes in order of their individual preference by giving the prototype most preferred a rank of 1 and the one least preferred a rank of 20.

Benefit Estimation Method

MONANOVA: Since the Preference model adopted in the experiment is non-metric and as a result preference ranking scales were used, the MONANOVA algorithm is most appropriate for estimation. Cattin and Bliemel (1978) proved the superiority of MONANOVA as compared to an OLS estimation for deterministic data.

Table 7.3: Steps in conjoint analysis (current study).

7.7 Results

Participants were 102 students enrolled on a three-year business program at the University of Surrey. An analysis of the demographic variables of the sample revealed a nearly even gender split; 52 per cent of subjects were males and 48 per cent were females.

Table 7.4: Gender

Frequency Percent Valid Percent Cumulative Percent

Valid Male 53 52.0 52.0 52.0

Female 49 48.0 48.0 100.0

Total 102 100.0 100.0

In terms of age, 96 per cent of subjects were aged 30 years or less. In particular, a higher skew towards younger age groups especially 21-25 years (53.9%) which is generally reflective of graduate student age is displayed in the histogram and normal curve in Figure 7.6. The skewness and kurtosis values of 0.915 and 1.604 are indicative of a positive skew.

Table 7.5: Age

Frequency Percent Valid Percent Cumulative Percent

Valid 18-20 years 28 27.5 27.5 27.5

21-25 years 55 53.9 53.9 81.4

26-30 years 15 14.7 14.7 96.1

31-35 years 3 2.9 2.9 99.0

36-40 years 1 1.0 1.0 100.0

Total 102 100.0 100.0

Age

tDD-I IÎ

Agt

5M Of ï "3 795

N.tB:

Figure 7.6: Age distribution- histogram and normal curve

The bulk of the subjects were level 2 students (52%) followed by postgraduate MSc students (28.4%) and as such were expected to have gained an appreciable knowledge of business performance indicators in general. The skewness and kurtosis values of 0.748 and 0.290 are a further indication of a positive skew.

Table 7.6: Student Status

Frequency Percent Valid Percent

Cumulative Percent

Valid Level 1 UG-SOM 6 5.9 5.9 5.9

Level 2 UG-SOM 53 52.0 52.0 57.8

Level 3 UG-SOM 6 5.9 5.9 63.7

MSc PGT-SOM 29 28.4 28.4 92.2

MBA PGT-SOM 4 3.9 3.9 96.1

PHD PGR-SOM 4 3.9 3.9 100.0

Total 102 100.0 100.0

stud«nt st*(u$

tlcifl M

o « v ►! m

N.I2

Figure 7.7: Student status distribution- histogram and normal curve.