Information Systems evaluation approaches

M u m fo rd (1 9 8 3 ) highlighted th e social and technical n atu re o f in fo rm atio n systems and

recom m ended th e d evelo p m en t o f systems th a t w ould enhance jo b satisfaction. She

argued th a t hum an factors w e re often not taken into account w h e n introducing co m p u ter

systems into organisations, resulting in failure to achieve th e desired outcom es. Klecun and

Cornford (2 0 0 5 ) noted th a t th e trad itio n al focus o f in fo rm atio n systems' evaluation had

been on im m ed iate usability needs, th e ir technical aspects, perform ance, reliability,

robustness and security and cost-benefit assessments. They argued th a t evaluation o f

in form ation systems rem ained "controversial, biased, incom plete and superficial" (Klecun

and Cornford, 2 0 0 5 ). W alsham (2006) argued th a t th e increasingly pervasive and com plex

n atu re of in fo rm atio n systems and th e ir use as tools fo r organisational change en tailed n ew

dem ands fro m evaluation activities. Klecun and Cornford (2 0 0 5 ) also noted th a t th e

com plexity o f n ew systems introduced new sociotechnical aspects th a t w e re difficu lt to

evaluate. Land (2000) recom m ended th a t inform ation systems evaluations should include

w id e r perspectives at technological, economic, organisational and individual levels. Land

suggested th e use o f a m odified version o f Kaplan and N orton's (1 9 9 5 ) balanced scorecard

to cover a range o f issues such as custom er, financial, internal efficiency, learning and

grow th and em ployees' perspectives. M artinsons e t al. (1 9 9 9 ) also looked a t ho w th e

balanced scorecard could be adapted to th e m an ag em en t o f business functions,

organisational units and individual inform ation systems projects. O th e r researchers looked

a t aspects such as alignm ent o f new technologies w ith strategic and business goals,

understanding o f form al and inform al w o rk structures, diversity and com peting interests

o f stakeholder groups and th e ir inform ation needs (Symons, 1991; Sm ithson and

Hirschheim, 1 9 98). Cornford and Klecun-Dabrowska (2 0 0 1 ) looked a t ethical issues and th e

effects o f n ew systems on legitim ate interests o f stakeholders. Others found evidence of

evaluation being used fo r legitimising vested interests (Friedm an and W y a tt, 200 6 ; Rigby,

2001

The concept o f benefits realisation m anagem ent, or simply benefits m an ag em en t or

benefits realisation in IT projects relates to how resources are utilised to m ake desirable

im provem ents (Sapountzis e t al., 2 0 0 8 ). It also aims to align project outcom es w ith business

strategies, and actively m anage th e benefits th ro u g h o u t th e life cycle o f th e investm ent

(W ard and Daniel, 2 0 0 6 ). This concept was introduced in th e late 1980s (Farbey e t al., 1993)

and has becom e an increasingly im p o rta n t fa c to r in planning fo r how benefits fro m IT

projects are realised and m easured (Glynne, 2 0 0 7 ). The increasing use and com plexity o f IT

systems (Ashurst and D oherty, 2003; W a rd and Elvin, 1999; Bradley, 2 0 1 0 ) and th e need to

ensure th a t benefits from investm ents are realised and th a t decision m akers are held to

account have also contributed to th e popularity o f this m ethod (Sapountzis e t al., 2 0 0 8 ). In

th e UK, th e M cC artney report, Successful IT: M odernising G overnm ent in Action and th e

CSSA rep o rt G etting it Right f o r G overnm ent (W heeler-C arm ichael, 2 0 0 0 ) stated th a t

programs can only be regarded as successful if th e ir intended benefits are realised. Reiss

(2006) also noted th a t most unsuccessful programs w e re characteristically vague regarding

th e ir expected benefits. O ver th e past decade, benefits realisation has gained popu larity in

healthcare settings because o f th e challenges in ascertaining w h e th e r benefits have been

achieved due to com plex interplay b etw een m u lti-stakeh o ld er team s and various IT

systems. The benefits realisation m odel requires expected benefits to be o utlined fro m th e

outset and a project is only considered successful if th e intended benefits have been

stakeh o ld er plays and manage all stakeholder groups and w h e re necessary, assign benefits

to d iffe re n t stakeholder groups.

A n o th e r w id ely used technology evaluation approach is health technology assessment

(HTA). HTA is defined as a "tool to review technologies and provide evidence of th e value

these technologies can deliver to patients and th e ir fam ilies, health system stakeholders,

and to society m ore broadly" (INAHTA, 2 0 1 5 ). According to D rum m ond e t al. (2 0 0 8 ), HTA

seeks to inform decision makers regarding th e benefits, risks and costs o f n ew and existing

technologies. This m ethod has historically been used to produce inform ation th a t can be

used by a w id e range o f decision makers. H ow ever, th e focus o f HTAs has shifted to

inform ing particular resource allocation decisions regarding healthcare technologies,

evaluating costs and benefits as w ell as paying a tte n tio n to th e decisions influenced by th e

HTA (D rum m ond e t al., 2 0 0 8 ). There is increasing awareness o f th e im portance o f hum an,

organisational and social factors th a t m ay affect th e introduction o f n ew technology

(Kaplan, 2001a; Kaplan, 2 0 0 1 b ). Kaplan (2 0 0 1 b ) argued th a t th e re should be a shift in

attitu d es around health inform ation technology evaluation fro m th e tra d itio n a l focus on

technical and econom ic issues to sociotechnical issues. She noted th a t no single m eth o d

could ad eq u ately provide decision makers w ith all th e inform ation th a t was required to

m ake inform ed decisions about health inform ation systems.