User Group Knowledge Base

For each user group in a given population, behaviour is modelled following a calibration exercise involv-ing several user group representatives (participants). User groups are generally established based on cit-izen demographics, however the calibration process may shape the development of such groups through emerging behavioural traits, informing the creation, modification, merging or retirement of groups. This may in turn affect the requirements development process through the introduction of new (and possibly unexpected) project personas, which would in turn inform the development of use cases and scenarios.

Table4.12provides an example of several project personas that share three user groups’ behavioural traits (i.e., formalised through statistical behavioural models).

Table 4.12: Multiple project personas linked with different user groups, distinguished by some common factor(s)

User Group Project personas

Care is required to avoid over generalisation when constructing user groups. For instance, a Sec-ondary school teachers user group (using the plural, not to be confused with personas) can be considered to be an over-generalised group. Although secondary school teachers generally share a common educa-tional background, teachers across different age groups may exhibit significant behavioural differences when interacting with computer systems (e.g., due to different levels of experience and confidence). A 50 year old teacher working in a secondary school might have had less exposure to electronic means

4.4. User Group Knowledge Base 135

of learning and teaching, and thus system designers may want to distinguish him from other teachers in the 25–35 year-old bracket. For this reason, calibration should focus on multiple user groups: sec-ondary school teachers (22–40), secsec-ondary school teachers (40–55), and secsec-ondary school teachers (55+). During calibration one might notice that different user groups share similar behavioural patterns and workload perceptions, and thus these could be merged (e.g., secondary school teachers (40+)). User groups should be re-usable across projects and thus generic group names are recommended. For in-stance, instead of referring to secondary school teachers (22-40) one should abstract this group even further and consider graduate professional (22–40). This makes this user group persona-agnostic and thus re-usable across project personas (i.e., not necessarily related to e-learning projects). A project may require several personas however these may share one or more calibrated user groups. This means that a user group called university graduates (under 25) could be linked with a number of unrelated project personas (e.g., nurse persona for an e-health system and teacher persona for an e-learning platform) as long as contextual differences are taken into account for the e-services being developed. Both personas can be university graduates and under 25 years of age, and although they are used in separate projects they may still share the same behavioural patterns and attitudes when dealing with security tasks. This does not apply when contextual calibration is adopted for a specific user group since the resulting models could be influenced by contextual nuances that may not be applicable outside that specific environment (consider calibration conducted in a hospital emergency room as opposed to a clerical environment). In these specific cases, user groups should be flagged with the context in which calibration took place (e.g., the underlying models for the 35-45 business-person (office) user group may differ significantly from the 35-45 business-person (travelling) user group). In situ and contextual calibration is discussed in Sections 4.3.3and9.4.1.4.

A number of user group differentiating factors exist, which could include both demographic and biographic user properties (e.g., age group, education level, geographic context and computer literacy).

Algorithmic techniques (e.g., clustering) may also be used to uncover subtle differences in behaviour between participants (from a pool of participants with the same set of properties). This may help fine tune the user group creation process by exposing potential clusters of users from less obvious behavioural patterns in data (see Section10.3.3). User groups are not user archetypes, but are mere containers for behavioural models created for (and through representatives of) individual groups of users. User groups are in turn associated with project personas (turning them into Calibrated Personas) which can then be used to simulate user feedback to assist in early-stage decision making. The existence and quality of this feedback depends on (1) the models available for each group of users and (2) the strength of such models.

The upkeep of the user group library is strongly suggested, by:

1. Conducting routine re-calibration exercises on existing groups (e.g., yearly). The model’s predic-tive power may increase with more participants although a saturation point exists.

2. User groups may need to be updated (or retired altogether) especially when participants who were used to build the original behavioural models (through calibration) are replaced with a new

gen-eration of users who may exhibit different behavioural patterns and attitudes towards enrolment processes. Tests such as the Wilcoxon Signed Rank test (no assumptions of normal distribution) or the paired-samples t-test (assuming a normal distribution) can help the researcher determine whether two sets of data from the same population (assumed) are significantly different from each other. This may happen when a user group has been in existence for a number of years (e.g., five or more years).

3. Creating new user groups when specific patterns emerge from the calibration data. Data can hold interesting insights on behavioural patterns and through clustering techniques one may identify recurring deviations in data collected from a single user group.

Given a shared user base across government agencies, the author argues that government entities would benefit from the creation (and upkeeping) of a user group knowledge base. For instance, over 200 e-services were rolled out (or updated) in the last two years in Malta, all serving a specific subset of the same population.

Figure 4.31: Weighting for the various NASA-TLX workload dimensions generated from the user group calibration exercise conducted with 55+ year old computer newbies. These are then used to generate warnings on design aspects that can have a stronger negative impact on this particular group of users

In practical terms the user group library consists of a database of user groups together with their statistical models explaining their reaction towards the various design factors – in this case related to en-rolment. When personas are constructed for a new e-service, the project team should determine whether an applicable user group exists within the library. User groups give a ‘voice’ to the persona construct (turning it into a Calibrated Persona), adding analytical power which could be used to simulate user feedback on critical design decisions during the requirements development process. Furthermore, Cali-brated Personas can also be used to glean practical design recommendations for the different user groups involved. These recommendations would be automatically generated based on the underlying data pro-duced during the calibration process. Figure4.31and4.32depict two Calibrated Personas together with

4.5. Summary 137

Figure 4.32: Weighting for the various NASA-TLX workload dimensions generated from the user group calibration exercise conducted with young urban professionals (30–40 years old). These are then used to generate warnings on design aspects that can have a stronger negative impact on this particular group of users

practical design recommendations based on the respective groups’ workload weightings. These warn-ings are generated for the project team’s consideration, highlighting design aspects that have a negative impact for each of the two project personas. Weightings shown are based on median values derived from NASA-TLX data (final pairwise comparison) generated during the user group calibration exercise with each respective group of participants.

4.5 Summary

The enrolment process is considered to be a critical aspect for e-service design and development, po-tentially affecting adoption rates and overall success (see Section2.3). A number of enrolment-specific design factors have been presented (see Table 4.1). These factors have been uncovered through an empirical exercise designed to explore issues and negative experiences encountered by regular internet users during enrolment (see Section4.1). Based on these design factors as well as a survey of common e-service enrolment processes, this chapter then presents a technique to build behavioural models for different groups of users that help explain and predict the level of perceived workload and their will-ingness to enrol for and use an e-service (i.e., given a specific enrolment process and type of service).

Data modelling (i.e., regression), as opposed to black-box algorithmic modelling (e.g., Artificial Neural Networks) was found to be the more practical and sensible approach for this thesis. This was mainly due to the exploratory nature of the study whereby full control of the underlying processes as well as the availability of well documented tests (to monitor output quality) were highly desirable. Two regression techniques were used, one for perceived workload (multiple linear regression) and one for the users’

willingness to complete the task (binary logistic regression).

Individual enrolment-related design factors may have a different impact (i.e., different intensity) on the users’ lived experience (ULX), and in turn this impact may also vary across the different groups

of users. For this purpose the User Group Calibration exercise (UGC) was developed – a systematic data collection protocol to measure and aggregate behavioural patterns exhibited by different groups of users when facing different enrolment processes. The calibration exercise consists of a set of fictitious enrolment tasks based on a survey of commonly found e-service enrolment processes across the globe.

Following each tasks participants provide their feedback on the perceived workload involved, across six workload dimensions, as well as on their willingness to complete the task, across four types of e-services. This data is then pre-processed and used to fit the two regression models out of which two sets of regression coefficients are produced. These coefficients could then be associated with project spe-cific personas (turning these into Calibrated Personas) which could in turn be used to predict (or rather, indicate) possible user reactions towards enrolment-centric e-services (see Chapter5). Knowledge on different user groups is stored in a user group knowledge base for future reuse across different projects within the same governmental context (e.g., region or nation). As with other knowledge bases, mainte-nance is an important activity – and behavioural models need to be maintained by (1) re-calibrating them (to improve model strength) and (2) replacing them with new models to respect generational nuances (e.g., 25 year old undergraduates of today may not exhibit the same behaviour as their counterparts five years down the line). Risks to validity were also discussed and techniques such as UGC participant sampling as well as contextual calibration were presented as measures to mitigate these risks. The user group calibration exercise provides important insights on user attitudes (e.g., towards workload) which can be used to inform design decisions (see Figures4.31and4.32).

Chapter5will outline Sentire, a requirements framework that adopts Calibrated Personas to gener-ate simulgener-ated user feedback on critical design decisions (i.e., enrolment-based use cases). By embedding user feedback simulations at the requirements stage practitioners can test the impact of their design decisions before building the actual product (or prototype). This also introduces the idea of testable experience-related requirements which are decorated with measurable fit-criteria. Any proposed use case design must be run through the simulator to determine whether such requirements are observed (see Section5.1).

Chapter 5

A Requirements and Design Framework for