Evaluating end-user computing competence

Studies have shown a gap between the computing skill expectation of an employer and the actual skill of an employee (Berezina et al., 2011; Bunker, 2010; Gibbs et al., 2010; Murray & Perez, 2014). A problem that both employers and those assessing their own computing skill face can be the ambiguity in definitions of computer literacy. Some regard computer literacy as being able to use specific and common workplace software applications, while others regard it as being able to navigate the Internet (Gibbs, et al., 2010; Murray & Perez, 2014; Perez & Murray, 2010).

Although the use of end-user applications is a requirement in many jobs, skill level may be at a very basic level and therefore the maximum benefit from using the technology is not achieved (Bunker, 2010; Eschenbrenner & Nah, 2014; Torkzadeh & Lee, 2003). Effective end-user computing capability within an organisation ideally includes effective task completion. This type of capability may be

40

related to the competency of an organisation as a whole (Bunker, 2010; Yoon, 2009). For example, in a report on the digital literacy standards in New Zealand organisations, Bunker (2010) found that, although the New Zealand government had introduced digital strategies into the education sector, there was little in place to support improving digital literacy in organisations. The findings of this report suggest that workplace digital competency levels are low, resulting in lost personal and organisation productivity.

Often the type of computing skill required in a workplace is assumed and not measured in any way (Murray & Perez, 2014). If measurement does take place, it will often involve some form of self- assessment (Gibbs et al., 2010; Grant, Malloy & Murphy, 2009; Gravill et al., 2006, 2001). Self- assessment is frequently used to assess end-user computing, in both educational and workplace settings. However, self-assessment has been found in other domains to be an inappropriate method of measuring computing knowledge when used in isolation as it is often subject to biases such as the above-average effect (Gravill et al., 2006, 2001, Grant et al., 2009; Stoner, 2009). When inaccuracies in self-assessment involve an over-estimation of skill the result may be that an employee is not performing at an expected or acceptable level (Gibbs et al., 2010; Grant et al., 2009; Gravill et al., 2006, 2001). Having reliable measurement of skill would mean that the constructs that make up this skill-set are less likely to be affected by social biases, and appropriate indicators would be available for measuring the impact of end-user skill in an environment such as the workplace (Torkzadeh & Lee, 2003). End-user computing competence has been defined as having a complete set of

knowledge, computing skills and attitudes that combine to allow a user to complete tasks efficiently and effectively (Eschenbrenner & Nah, 2014; Suen, 2012; Leahy & Dolan, 2010; Yoon, 2009; Yoon et al., 2008). Such competence has an influence on how well an individual can apply IT knowledge when using software or systems required for completing workplace tasks (Eschenbrenner & Nah, 2014; Suen, 2012).

Various models of end-user competency have been proposed. For example Huff, Munro, and Marcolin (1992) endeavoured to create a model that would accurately measure end –user computing sophistication where sophistication is the level to which the software is used. For their study, they classified end-user computing knowledge and skill on three dimensions: breadth, depth and finesse. The dimension of breadth refers to the range of computing knowledge a person may have. Depth was used to classify how much a person knew about certain aspects of EUC. The final dimension, finesse, refers to an end-user’s ability to apply their knowledge “creatively” in an end- user situation. Those who could be deemed true novices were those who had experience in only a

41

small number of the applications. Those who could be deemed true experts were those who had experience and knowledge in a wide range of applications and computing concepts. Experts also featured in the finesse dimension by rating themselves as highly creative.

More recently, Yoon (2009) attempted to produce a ‘reliable’ instrument to measure end-user

competency in a business setting based on computing competencies defined in prior research. Yoon defines an end-user as someone who directly interacts with a computer in a business situation and defined end-user computing competency (EUCC) as being

“a total set of knowledge, technology, skills and attitudes which function as action

characteristics of an organisational member who can outstandingly and efficiently do his or

her tasks in a computing environment” (Yoon, 2009, p. 47).

According to Yoon (2009), from his review of prior research, end-user computing competency can be explained by four components: computing mind; computing knowledge; computing application and computing potential. Yoon (2009) surveyed end-users from a variety of different industries. The survey consisted of a questionnaire designed to measure end-user competency by asking

respondents to rate their ability on a number of end-user tasks and included questions relating to computer security, hardware, software and network knowledge as well as some questions relating to Internet and computer use and etiquette. Questions were then assessed against the four criteria previously mentioned. An example of a question related to computer mind-set was “How many computer magazines do you subscribe to?” whereas computing knowledge questions included “How much do you know and understand computing technology, applications and computer systems?” All questions were presented using a 5 point Likert type system where the respondent could answer on a scale from 1(Not at all) to 5 (A great deal). Yoon (2009) concluded that this instrument was a valid and reliable measure of end user competency because it not only asked all the general “what can you do” questions, it also included the competency constructs that other researchers had identified. Models, such as those promoted by Huff et al. (1992) and, more recently, Yoon (2009) have been useful in recognising the user as central in extracting the best from software or a system. However,

these models rely on an individual’s subjective self-rating of their computing skill, confidence using

IT or knowledge of IT and it is not possible to assess computing skill reliably when a computer is not part of the assessment method (Stoner, 2009).

42

Previous research in self-assessment of skill in areas such as computer literacy has found differences between perceived and actual ability (Anderson, Benamati, Merhout, & Rajkumar, 2010; Ballantine, McCourt Larres, & Oyelere, 2007; Gravill et al., 2006). For example, Gravill et al. (2006) reported that participants with a greater breadth of experience using computer software had self-assessments more aligned to actual knowledge than those with a lesser exposure to IT. Gravill et al. (2006) used the cognitive skills dimension of a three-dimension user competence cube to represent the

relationships between self-assessed knowledge of IT, procedural knowledge and declarative knowledge and the influence of experience on these factors. The three conceptualisation dimensions included in the cube were cognitive outcomes, skill-based outcomes and affective outcomes. Each dimension was measured using a combination of self-report, paper and pencil testing, a hands-on test and observer assessment. The cognitive dimension was identified by Gravill et al. (2006) as being most appropriate for their study because this dimension, which refers to the knowledge a user will have about technology and their use of it, comprised declarative, procedural and strategic outcomes that could characterise the incremental stages of knowledge attainment. In their study, experience factors included years of use, the breadth of use and a control or anchoring factor where two groups were given the same instruments but one had the self-assessment before the knowledge tests. Results from this study found that self-assessments were more closely related to procedural knowledge than to declarative knowledge and that varying the order of the self- assessment resulted in a closer alignment between self-assessment and declarative knowledge than with procedural knowledge. Their results also showed that those participants with greater exposure to technology (measured as years of use) were no more accurate in their self-assessment than those with fewer years’ experience. The findings reported by Gravill et al. (2006) are interesting and relevant to both the areas of end-user computing and research in social biases. While Gravill et al. (2006) did not specifically look for cognitive explanations of the self-report results; their findings are in direct contrast to work in the area of the AAE and more specially the DKE. Kruger & Dunning (1999, 2009) contend that the AAE is influenced by a person’s level of expertise in a particular domain. Kruger & Dunning (1999, 2009) say that those with low levels of expertise often do not recognise this, believe they have more expertise than they do and are unlikely not to recognise expertise in others. They also say the reverse is true, that those with more expertise may be more likely to under-estimate this in themselves, all of which is somewhat different to the results present by Gravill et al., (2006).

To measure skill and knowledge accurately without the need to rely on inaccurate self-assessments it is necessary to have some method of benchmarking skill level (Bunker, 2010; Gravill et al., 2006;

43

Leahy & Dolan, 2010). One way of providing a benchmark of skill is to require employees to attain some type of certification or standard (Bunker, 2010; McGill & Dixon, 2004; Vakhitova & Bollinger, 2011). Skill validation certification can be not only associated with high skills and therefore be of value to an employer, but it may also send signals about a person’s motivation to keep up to date with changing technology and to improve their knowledge (Vakhitova & Bollinger, 2011).

There are a number of computing learning and testing systems available through educational institutions or accessible via the Internet. Some of these, including SAM (Skill Assessment Manager)2 MOS3 _{(Microsoft Office Specialist), and ECDL}4 _{have been used or recommended for use as industry} standards on which to benchmark workplace computing skills (Bunker, 2010; Calzarossa, Ciancarini, Maresca, Mich & Scarabotto, 2007;Davis & Cleere, 2003; Grant et al., 2009; McLay & Brown, 2006; Panicos & Sotiris, 2010; Townley, 2004; Wallace & Clariana, 2005). Each of these end-user computing systems claims to have been informed through a rigorous development process by subject matter experts and to be quality assured (Leahy & Dolan, 2010). However, in some instances this type of learning system has been found to be inflexible in particular workplace situations (Gravill et al., 2006). Some employers value computing certification more highly than they value some degree qualifications (McGill & Dixon, 2004; Vakhitova & Bollinger, 2011). This may be because certifications give an assurance that a person has specific skills, whereas the skills gained in a degree may be regarded by some employers as more general (McGill & Dixon, 2004; Vakhitova & Bollinger, 2011 ). Some employers also believe that employees with certification will require far less workplace training than those without industry certification (McGill & Dixon, 2004; Vakhitova & Bollinger, 2011). Although these certifications are promoted as assuring workplace-ready practitioners, there are some disadvantages associated with them. Risks include the fast rate of technological change and graduates of these courses who may be unable or unwilling to keep up to date (Gravill et al., 2006; McGill & Dixon, 2004; Vakhitova & Bollinger, 2011). It is important to note that perceived skill level may be just as important as actual skill level in the effect it has on a person’s attitude toward using and extending their use of technology (Torkzadeh & Lee, 2006).

2_{http://www.cengage.com/samoffice2013/}

3 _{http://www.microsoft.com/learning/en-us/mos-certification.aspx} 4 _{http://www.ecdl.com}

44