Evaluating tests for local use

Assessments with known psychometric properties are intended to provide a

standardised metric for comparative purposes, for example, comparing individuals in pool of job applicants. Despite the intention of many test developers to develop tests for use in more than one country, there is debate in the literature about the cross-cultural transportability of assessments.

It is well recognised that there are difficulties associated with importing an assessment

for use in a population different to that in which the assessment was developed (Dana, 1993; Hambleton, Merenda, & Spielberger, 2005; Samuda & Wolfgang, 1985; Suzuki & Ponterotto, 2008). For example, tests developed in one culture, may not be equally valid in another culture, as the test takers are likely to have different values and knowledge than those implicitly

assumed by the test (Greenfield, 1997). The implication is that the test might not measure what it was intended to measure (e.g. ability), but instead measure respondents’ (deficit) knowledge about another culture’s cultural conventions (Greenfield, 1997).

Many test publishers are based in the United States and United Kingdom, and the tests are imported for use in other countries. Most of the commercially available psychometric tests in New Zealand were developed by international test publishers, and this issue has been

recognised by New Zealand researchers (cf. Rodriguez, Treacy, Sowerby, & Murphy, 1998). Although local (New Zealand) norms are usually available for referencing respondents’ scores, as the tests were designed and initially validated in other countries, items within the test and

the test itself may not have the same validity in the local setting.

Importing and using a test in New Zealand therefore requires careful consideration as to whether it is appropriate to use the test in the local setting. The setting does not only include the country (i.e. the appropriateness of using a test designed in the UK in New Zealand), but also the appropriateness of using a test in particular occupational setting.

Assessments from other countries can be adopted or adapted for use in a new country (van de Vijver, 2002). Various strategies have been proposed to assist the process of importing and adapting tests for local use. Greenfield (1997) emphasised assessing the degree and type of difference between the test’s origin culture and the culture in which it is has been imported to, and where there is a large degree of difference, considering the use of completely different

assessment methods.

Test adaptation and modification is another strategy, and researchers have proposed guidelines to assist test importation and translation (cf. Hambleton, 1996, 2001; Hambleton & De Jong, 2003). Guidelines focus on processes such as test development, reliability assessment,

validity assessment and norming, establishing item equivalence, establishing score equivalence, test administration and test-taking guidelines, and the methodologies associated with these test adaptation processes.

Attempts to adapt a test for a local setting must involve a review of the adapted test and statistical analysis to ensure that the modified test maintains its integrity. For example,

Rodriguez et al. (1998) investigated the applicability of Australian adaptations of the Wechsler Intelligence Scale for Children (3rd edition) and the Stanford-Binet Intelligence Scale (4th edition) in a sample of Dunedin children. The researchers found that the test means of the Dunedin sample were comparable to the US norms, providing some initial reassurance about the use of

the modified tests in this country.

The General Reasoning Test

The General Reasoning Test (GRT2) is a commercially available psychometric test developed in the United Kingdom by the test publisher Psytech. Although it has been used in various occupational settings in New Zealand, and New Zealand norms are available, it is newly

occupational setting, at a minimum the test needs to be re-validated using a sample from the population in which the test is intended to be used (i.e. corrections officers working in New Zealand). As its psychometric properties are unknown in the new setting, the study will

examine the factor structure and construct validity when the test is used in the local sample and re-estimate its reliability.

The General Reasoning Test (GRT2) is designed to assess three forms or subtypes of general reasoning ability (verbal, numerical, abstract reasoning). The assessment is comprised of three separate subscales, with the items in each subscale designed to assess the same facet

of reasoning ability. It is expected that the underlying structure (latent dimensions) in the GRT2 data will also resemble the three-scale structure in the local setting. To test this

assumption, factor analysis will be conducted using item-level data from all three subscales of the GRT2, in an attempt to reproduce a three-dimensional structure. As the original factor

loadings are not available in the technical manual for the GRT2, exploratory rather than confirmatory factor analysis will be used (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Thompson, 2004).

Professionally constructed assessments of cognitive ability tend to have excellent psychometric properties, including good levels of reliability (Ones et al., 2012). The technical

manual for the GRT2 reported that the test had good levels of internal consistency reliability in a range of occupational samples, with most Cronbach’s α coefficients above .8 (Psytech International Ltd., n.d). It is expected that the GRT2 will also show good internal consistency reliability in the present sample. In addition, the study will examine the item difficulty and item discrimination statistics for each subscale.

• Assumption 1: The GRT2 is expected to reveal a latent structure comprising three dimensions representing verbal, numerical, and abstract reasoning, on the Corrections sample.

• Assumption 2: The GRT2 subscales will have good levels of internal consistency reliability as evidenced by the Cronbach’s α, on the Corrections sample.

Job performance appraisal

It is well recognised that job performance is a multi-faceted construct (Borman & Motowidlo, 1993, 1997; Sackett, 2002; Schmitt, 2014). Several models have been proposed for the dimensionality of job performance, for example, task and contextual performance

through the literature did not find evidence of any published research into the facets of job performance for corrections officers working in New Zealand.

The study uses a custom-designed job performance appraisal tool. The tool is concise

but it does permit assessment of sub-constructs of performance. Information on underlying factor structure or other evidence for construct validity is not available in existing publications for the tool in its current form. An exploratory aim is to gather evidence for factor structure over the item set. Factor analysis will be used to investigate dimensions of performance in corrections officer data. This may determine whether performance as assessed by this tool is

best explained as a single, global factor, or as an array of multiple facets. The conceptual nature of any facets will be examined. The reliability of the performance measure will also be analysed.

• Research aim: To explore the dimensions in the measured performance data for corrections officers working in New Zealand.