Research protocol

Each participant went through the experiment independently of other study participants. All participants used the same computer at different times in a secluded room in order to reduce design contamination for the experiments and to enhance my ability to carry out effective observations. Each participant followed the following procedure:

1. The purpose of the experiment was explained to the participant

2. The participant completed an informed consent form. Refer to Section H.1 in AppendixH for more details

3. The procedure of the experiment was explained to the participant. Refer to Section H.2in AppendixH for more a script that was read to each participant

4. The participant was randomly assigned to either experimental Group one or Group two where language A was Arden Syntax and FCIG respectively

• The experiment administrator assigned a sealed and unmarked envelope con- taining written instructions for the experiment. The instructions were pre- viously randomly set for either Group one or Group two through a digital randomisation application called Random UX1_.

• There were two sets of instructions for the experiment. The first set of instruc- tions was for participants in Group one2 and the second set of instructions was for participants in Group two3

5. The participant was allocated a preset computer

6. The participant followed appropriate instructions for Group one or Group two depending on the group allocated to them

7. The participant undertook self-paced basic training for language A using an ap- propriate tutorial for Group one4 or Group two5 accordingly

1

https://play.google.com/store/apps/details?id=ru.uxapps.random&hl=en

2_{https://github.com/yamiko/CIG-Modelling-Languages/wiki} 3

https://github.com/yamiko/cig modelling languages/wiki

4

https://github.com/yamiko/CIG-Modelling-Languages/wiki/Language-A

8. The participant was issued with an appropriate set of instructions for the CIG modelling tasks on a printout depending on the sequence of the experimental conditions. Refer to Section H.3 and Section H.4 in Appendix H for Group one and Group two instructions respectively

9. The participant was observed whilst carrying out tasks for the experiment

10. The participant carried out the tasks using a guideline modelling language accord- ing to the group one was allocated to in step three

11. The participant was asked to fill an online post-test questionnaire. Refer to Section H.5in AppendixH for a list of questions asked in the post-test questionnaire 12. The participant was given a five minute break

13. The participant undertook self-paced basic training for language B using an ap- propriate tutorial for Group one6 or Group two7 accordingly

14. The participant carried out the same set of guideline modelling tasks as those issued in step six using the CIG modelling language allocated in step 11. Refer to Section H.3 and Section H.4 in Appendix H for Group one and Group two instructions respectively

15. The participant was asked to fill an online post-test questionnaire after completing tasks with the second experimental condition. Refer to Section H.5 in Appendix H for more details on the set of questions asked in the post-test questionnaire 16. The participant was interviewed to get a qualitative description of the CIG mod-

elling language that the participant preferred

17. The participant signed in the participant log and received ZAR50.00 to compensate for their time.

7.2.6 Data collection methods

I used the following data collection methods in the study:

• Low level computer logs from the Fluorite plugin [303] of the Eclipse IDE were used to collect data on guideline modellers’ interaction with the guideline modelling environments.

6_{https://github.com/yamiko/CIG-Modelling-Languages/wiki/Language-B} 7

• Questionnaires were used. Where possible, online questionnaires were deployed for data collection. In addition, the SUS that uses a Likert scale was used as a standardised questionnaire to collect data about participants’ perceptions on a particular CIG modelling language.

• Observations were used to record details as modellers were carrying out the CIG modelling tasks. I observed from close enough a distance to see what was happen- ing but not so close as to interfere with the CIG modelling tasks.

7.2.7 Criteria to address the research question RQ2

In order to address research question RQ2, I compared collective performance measure- ments from FCIG modellers and Arden Syntax modellers. Each participant attempted to use the benchmark and HL7 standard guideline modelling language, Arden Syntax, as experimental Condition A. And at another point each of the participants used the new fine-grained CIG modelling language, FCIG, as experimental Condition B. Hence the independent variable was the particular CIG modelling language used during the experiment. Participants were randomly divided into two groups, Group one and Group two, as categorised in Section 7.2.5. As each participant was subjected to perform the same set of tasks using both CIG modelling languages, data was collected to measure: task time, success rate, error rate and efficiency. Success rate, task time and errors are fundamental metrics that can be applied to scientific or user research [231]. In order to measure and evaluate these various dimensions separately, sub-questions are devised for each of the dimensions as discussed in the following sub-sections.