Standardised usability questionnaires

Standardised usability questionnaires (SUQs) are designed to assess a participant’s sat- isfaction with the perceived usability of products during or immediately after usability testing [231]. Hence, an SUQ can be used to assess the perceived usability of a software language. An SUQ usually undergoes psychometric qualification that reports measure- ments that can determine its reliability, validity and sensitivity [231,237].

Nunnally [237] and Lewis [238] agree that among several other advantages, an SUQ can offer the following benefits: i ) Objectivity – a standardised measurement can allow measurement statements of other practitioners to be independently verified by others; ii ) Replicability – an SUQ can make it easier to replicate a particular study through the usage of standardised methods; iii ) Quantification – notwithstanding that the ap- plication of statistical methods to multipoint scale data is a point of controversy, stan- dardised measurements allow usability practitioners and researchers to report results in finer detail that use mathematical and statistical methods so that results can be better understood; iv ) Economy – in spite of the substantial amount of work that is required when developing standardised measures, the measures in an SUQ becomes economical to use once they have been developed; v ) Communication – standardised measures allow usability practitioners and researchers to efficiently and effectively communicate their usability results; vi ) Scientific generalisation – SUQs provide a platform for standardis- ation that is essential for assessing the generalisability of results.

Several SUQs have been introduced over the past few decades [231, 238, 239]. These SUQs have been introduced primarily to assist in the assessment of usability following participation in scenario-based usability tests [231]. SUQs can be classified into two categories. The first category of SUQs includes those that administered immediately following the completion of a task or test-scenario within a study [231,240,241]. The

second category of SUQs includes those that are used for administration at the end of a study [231,241]. A posttest SUQ can be used as a post-study questionnaire in a study that evaluates a CIG modelling language. The rest of this section discusses some of the post-test SUQs that have undergone a psychometric evaluation process.

The questionnaire for user interaction satisfaction (QUIS) is a post-study questionnaire that was first published in 1988 [242]. The original version of QUIS had 90 items that used a 10-step bipolar scale that was numbered from 0 to 9, and was aligned with the negative response on the left in addition to an off-scale ‘NA’ response to indicate non-availability [231]. A psychometric evaluation of the early version of the QUIS was reported for a short-form version of the questionnaire with 27 items covering usability aspects related to software, screen, terminology and system information, learning, and system capabilities [231, 242]. The evaluation showed evidence of construct validity and sensitivity although items in screen factor did not group as expected [231]. The current version of the QUIS is available in five languages (English, German, Italian, Portuguese, Spanish); in two lengths, short with 41 items and long with 122 items; and using a nine-point bipolar scale with an off-scale ‘NA’ response similar to the initial version [231]. A licence for the QUIS is required before usage of the questionnaire as follows: i ) A student licence requires 50 United States Dollars (USD50.00); ii ) An academic or non-profit license requires 200 United States Dollars (USD200.00); iii ) A commercial licence requires 750 United States Dollars (USD750.00) [231].

The software usability measurement inventory (SUMI) is a post-study SUQ that was introduced in the 1990s by the Human Factors Research Group at the University Col- lege Cork in Ireland [243, 244]. The SUMI is a 50-item questionnaire with a global scale based on 25 of the items five subscales for measuring efficiency, effectiveness, help- fulness, control and learnability [231, 244]. The SUMI contains a mixture of positive and negative statements for its items that are measured on a three-scale step namely: Agree, Undecided, and Disagree [244]. The SUMI is currently available in 12 lan- guages (Dutch, English, Finnish, French, German, Greek, Italian, Norwegian, Polish, Portuguese, Swedish, and Spanish) [231]. SUMI underwent a considerable amount of psychometric evaluation which provided some evidence of construct validity and sensi- tivity [231, 244]. Licences are required to allow one to use SUMI as follows: i ) One thousand Euros (EUR1000.00) for offline usage; ii ) Five hundred Euros (EUR500.00) for online usage; iii ) No fee for students [231].

The post-study system usability questionnaire (PSSUQ) is a post-study SUQ that was introduced in the early 1990s as an internal project at International Business Machines (IBM) Corporation [245]. The current version of the PSSUQ is a 19-item questionnaire with a single global scale and three subscales that measure system quality, information

quality and interface quality [246]. The items in PSSUQ are scored using values between one and seven, with lower scores indicating a higher degree of satisfaction [245,246]. The PSSUQ underwent a considerable amount of psychometric testing that provided some evidence of construct validity and sensitivity [245–247]. The PSSUQ does not require any licence fee for its use [231].

The system usability scale (SUS) is a post-test SUQ that was developed in the 1980s [232, 248]. The SUS is 10-item questionnaire that is scored using a five-step scale [248]. The scores from an SUS questionnaire are used to derive a single global scale [231,248]. The scores from an SUS questionnaire can also be used to reliably compute two subscales that can measure usability and reliability [249]. Like other more prominent SUQs in both research and practice, the SUS underwent a considerable amount of psychometric evaluations [231, 250]. The 10 SUS items were selected from a pool of 50 potential items [231, 248]. The psychometric tests provided evidence of reliability, sensitivity and validity [231]. The SUS does not require a licence fee as a prerequisite for its use [231,248].

The usability metrics for user experience (UMUX) is a recently developed four-item post- test SUQ that is based on a seven-point Likert scale designed to provide results on a single global scale that is similar to SUS [251]. The UMUX is organised around the ISO9421- 11 definition of usability [252]. The UMUX has a shorter form variant UMUX-LITE which only has two items that align it to the technology acceptance model [252, 253]. The UMUX has undergone limited psychometric evaluation. The validity, reliability and sensitivity of UMUX as an SUQ has been criticised [253–255]. A recent study that was conducted to evaluate the validity, reliability and sensitivity of UMUX provided some evidence of construct validity and sensitivity of the UMUX questionnaire [252].