Enrolment to Build Behaviour Based Models
Phase 5: Refining Themes Once satisfied with the thematic map Braun and Clarke [18] suggest that the researcher should go back to the codes and underlying data to identify the essence of each theme
1. Dependent variable is non-continuous. A case in point would be a study on family planning
4.3 User Group Calibration (UGC)
4.3.1 Calibration process
The workflow in Figure4.20outlines the steps required to calibrate a user group followed by a descrip-tion of each step.
Figure 4.20: Calibration process overview
1. Invite a number of representatives from each of the various user groups associated with project specific personas. At the time of writing the calibration process was hosted at http://calibrate.devbell.com (see Figures 4.21, 4.22 and 4.23). Tasks are presented in a ran-dom order so as to avoid unnecessary bias based on apparent patterns (e.g., expecting subsequent tasks to be more demanding or intensive).
Figure 4.21: One of the nine tasks presented during calibration
2. After each task, participants are asked to complete an online feedback form (see Figure 4.24) which captures NASA-TLX specific workload ratings as suggested by Hart and Staveland [70].
Figure 4.22: Another task presented during calibration
Figure 4.23: Participants are presented with notifications whenever interruptions or delays are present
This form also captures the participants’ willingness to complete the task in four different sce-narios, reflecting the four Types of Service (ToS) identified in Section4.1.1. The first scenario involves no legal obligation for use while the other three pose incrementing frequencies of use and levels of compulsion (i.e., requiring compliance within specific legal time frames). Scenario four represents an e-service that is used frequently and citizens are required to comply in a timely manner otherwise penalties apply. In all cases the alternative is to visit a government office or to send forms by post. These four scenarios are customised according to the user group under inves-tigation (e.g., filing tax returns as an example of a ToS 3 service may not be relevant to students, in which case an alternative example is provided – e.g., study-unit add/drop form which needs to be submitted 2 weeks before the start-of-term).
4.3. User Group Calibration (UGC) 125
Figure 4.24: Calibration task evaluation form
3. The collected data is preprocessed, grouped (by user group) and prepared for model fitting (see Table4.10).
4. Using a statistical package (e.g., SPSS) two regression models are then generated: one for per-ceived workload (multiple linear regression model) and one for the willingness to complete the task (binary logistic regression model). The modelling step will provide the researcher with a number of regression coefficients that explain the user groups’ reactions towards the various de-sign factors (e.g., for each additional item to recall introduced in the enrolment process, the like-lihood that the task is abandoned increases by 35%). These user group regression coefficients are then associated with the respective project persona(s) and eventually used to predict the expected perceived workload as well as the number of people that might complete the task for a given e-service being developed (i.e., ToS), per persona (i.e., Calibrated Personas), and for each alternative
Table4.10:Sampleoutputfromacalibrationexercise NASA-TLXmeasurementsTaskcompletionmeasurements Weighting(tallycount)RatingWillingnesstocompletetask TaskMDPDTDPEFMDPDTDPEFMWWType1Type2Type3Type4 ParticipantAA1221450000000010.90.9 ParticipantAH12214535254002510210.30.90.90.9 ParticipantAB1221450015105107.330.50.70.81 ParticipantAG1221455101551058.330.90.90.90.9 ParticipantAC12214515152510102016.330.20.70.81 ParticipantAF1221455103510152519.330.40.80.91 ParticipantAE1221451565602045304100.70.80.9 ParticipantAD1221455510552010.670.40.70.70.7 ParticipantAI12214525657030504048.3300.70.81 ParticipantBA2521141500101503.670.70.50.71 ParticipantBH25211475156030503537.670.50.60.71 ...
4.3. User Group Calibration (UGC) 127
enrolment process design.
It is important to note that at this stage the analyst should monitor the responses and identify any participants that provided data which is noticeably different from that of other participants within the same user group. This might indicate the existence of a new or different user group altogether.
Thus, at the analysis stage it is important to consider outliers as evidence for the existence of potentially new user groups, leading to fresh insights into the e-service’s target users.
SPSS (or any alternative thereof) can be used to support model fitting and generate the required coefficients. Manual model fitting is extremely difficult due to multiple dimensions introduced by the various predictors (ItR, ItG, D, I and ToS). Once the regression coefficients for the two models have been generated, they are assigned to the respective user group, which can in turn be associated with one or more project specific persona, turning them into Calibrated Personas.
Recurring UGC exercises will result in more realistic and fine-tuned coefficients. It may be the case that certain predictors are statistically not significant in any of the models for a particular user group. In that case, the respective portion of the model is removed (i.e.bnXni for which Xni is the value of the insignificant predictor for the ithobservation while bnis the regression coefficient for the same predictor).
Table 4.11: Regression coefficients generated for the young urban professionals (30–40) user group. These coeffi-cients explain the user group’s reactions to the various enrolment-related design factors
Regression coefficients
Task completion (see Figure4.25) Perceived workload (see Figure4.26)
B-Coefficient 5.866 3.888
Items to Generate -0.78 NA
Items to Recall NA 2.183
Delays -1.434 34.332
Interruption -1.925 24.127
Type of Service 1 -2.339 NA
Type of Service 2 -1.448 NA
Type of Service 3 -0.718 NA
Type of Service 4 NA NA
Figure 4.25: Task completion parameter estimates for the young urban professionals (30–40) user group
Figure 4.26: Perceived workload parameter estimates for the young urban professionals (30–40) user group
More participants to a calibration exercise will yield stronger predictive models, however a sta-tistical saturation point exists [128]. Saturation is hereby defined as the point at which stasta-tistical models do not exhibit significant improvements in prediction with the addition of more calibration data (e.g., an additional 10 participants will not yield more than 2% improvement over predictions generated by the original model). Out-of-sample tests can be used for this purpose whereby up-dated models are tested against a set of known observations. A score based on residual values (predictions vs actual observations) will help determine by how much the model has improved with the addition of new calibration data – based on the original model’s score (without the new data).
Further to this the calibration process may uncover unexpected clusters of behavioural patterns from within the same set of participants (who might have initially been assumed to share common
4.3. User Group Calibration (UGC) 129
behavioural patterns). Clustered behavioural patterns may indicate the existence of different user groups and further investigation might be required, which may then lead to the discovery of new (and unexpected) user groups altogether. A case in point can be observed in figures4.29a,4.29b, 4.30aand4.30bin which potential patterns in workload weighting is evident.