This section contains the plan of the experiment and the detailed implementation procedure. This exercise differs from the pilot deployment at Kingston (Section 4.8.1) as it incorporates strategies to avoid the encountered failure points (Section 4.8.2).
Figure 45: The within subjects design for the evaluation. All participants complete a pre-test, then watch the Bushfire Training Video. Participants split into groups. One group use Static Map examples followed by the interim test, then they swap to use the tangible Interactive Map examples
followed by the post-test. Group two follow a similar format but use tangible Interactive Map examples first followed, interim test, Static Map examples, finishing with the post-test.
The general public comparative study is now a within subjects design comparing the
Tangible Preparing for Bushfire interface (Section 4.8) using the map-based
constructivist tasks (Section 4.9) to the traditionally available alternative (Section 4.10). The within subjects style means that all participants use the touch table. The map-based tasks are now standardised. Furthermore, they are always implemented exactly the same way with exactly the same scripted introduction and all have a stated directed goal. There is very little interaction from the instructor. The instructor’s role is to introduce the scenarios and to explain the interface. They are not allowed to prompt the participant, or correct them as this may bias results.
The detailed system implementation procedure is listed step by step in Table 8 while Figure 45 shows the overall structure of the experimental process. These are presented so that other researchers can recreate the experiment for this research. Every participant follows the sequential steps listed in the table.
When recruiting participants it is important to mention that the average completion time is 53 minutes for individuals and 1hr 20mins for groups. All participants must have sufficient time to complete the session in one attendance.
Table 8: The implementation procedure for the experiment with the general public.
Step Description
1. Hand Participant Information Sheet to participant to read.
2. Ask participant to read and sign Consent Form
3. Novelty play,
4. Answer section one of the pre-test in the questionnaire
5. Answer section one pre-test practical component using the photos from
NSW Bushfire education,
6. Watch 4:30 min Tasmanian Bushfire Education video titled ‘Create a
Defendable Space’.
7. Record the sequence number (Section 5.2.1) obtained from the pre-
randomised list for this participant. Use this to identify the required cubes for the Static Map portion.
a. Place the required Worked Example Cube and Practical Cube on the PixelSense Bezel.
b. For this example assume sequence 4 (from Table 8). I Ex2 Test 1, S Ex1, Test 2
Participant Starts with Interactive Table using Worked Example Set 2 with Test number 1 (Practical Test Set 1) as interim test. The second phase switches to Static Maps using Worked Example Set 2 with Test number 2 as Final Test (Practical Set 2)
8. Start Audio Recording
9. Complete 3 Interactive worked examples (progressing from simplest to
hardest) [Use the touch buttons, to activate each exercise by placing the
appropriate face down on the surface.]
a. Introduce the Interface Objects
Sample dialogue for introduction of objects:
‘You may use these models with the table (Indicate the house, chainsaw and rake).
When you place the House on the table it shows the boundaries for the defendable space zones. These boundaries are calculated to include the correct slope for the house. The boundaries are removed when the house is removed.’
b. Briefly describe each scenario at their start. Sample dialogue:
‘Consider this property:
It is at the base of a hill. The property boundary is here (quickly trace by finger), their access road is here (indicate by pointing), These white lines are 10m contours indicating the land is relatively flat in front of the house here, but steep at the back here.’
c. Instructor records all pans, zooms and frequency of use of each interface object.
10. Start interim test – (Test 1) Recall section. Participant is moved away from the table to a chair and desk where they complete the recall section of the test by writing the answers to 3 questions on the questionnaire.
11. Return to the touch table for practical section.
12. Complete 3 Practical Scenarios (progressing from simplest to hardest).
Activate using the labelled touch buttons.
State: ‘You may use the interface at this point to help with your answer’
For each scenario:
Briefly describe each scenario at their start.
Ask question ‘What do you think is their greatest danger?’ Ask question ‘What do you think the owner could do to better prepare their property for bushfire?’
Ask question ‘In the event of a severe bushfire – would you stay or leave?’
Instructor records keyword and phrase responses, all pans, zooms and frequency of use of each interface object.
13. Switch to alternate system. If starting with Interactive switch to Static Maps, if starting with Static Maps switch to Interactive Maps.
14. Complete 3 worked examples (progressing from simplest to hardest) [Use
the cube objects, keeping with the TUI theme, to activate each exercise by placing the appropriate face down on the surface. Faces are labelled Worked Example 1, Worked Example 2 and Worked Example 3]
Briefly describe each scenario at their start.
15. Start Post Test – (Test 2 from sequence) Recall section. Participant is moved
away from the table to a chair and desk where they complete the recall section of the test by writing answers for 3 questions on the questionnaire.
16. Return to the touch table for practical section.
17. Complete 3 Practical Scenarios (progressing from simplest to hardest).
disabled in the interface. Participants may still touch the table or use the objects but no visible effects occur.]
For each scenario:
Briefly describe each scenario at their start.
Ask question ‘What do you think is their greatest danger?’ Ask question ‘What do you think the owner could do to better prepare their property for bushfire?’
Ask question ‘In the event of a severe bushfire – would you stay or leave?’
Instructor records keyword and phrase responses, and even though they do not perform any function still record any attempts to pan, zoom or use of each interface object.
18. Complete the overall review section of the questionnaire.
Explain that: ‘This section concerns the experience as a whole. It may seem like a lot of questions, but it takes between 3-10 minutes to complete. Please try to answer with your first impression.’
19. Participant is moved away from the table to a chair and desk where they
complete the remaining overall exercise review section of the questionnaire checking for missed questions.
20. Check through the completed questionnaire searching for missed or
incomplete questions.
21. Stop Audio recording.
22. Thank participant.
23. Offer selection of thank you mini chocolates or UTAS T-shirts.
24. Rename Touch Logfile
25. Paper clip Consent form, Supplementary Information sheets and the
Questionnaire together – put in locked cabinet.
5.2.1
Avoiding Bias by Changing the Sequence
To avoid any possible bias from a specific combination of worked example and test order the two sets of worked examples and the two tests were designed to be interchangeable. Therefore these could be swapped around. There are 8 unique configurations for Static Map/Interactive Table, worked examples and test combinations (Table 9). These 8 sequences were pre-ordered in a spreadsheet so participants were assigned their sequence based upon the order in which they arrived to undertake the exercise. The spreadsheet contained a group list and an individual list.
Table 9: There are 8 combinations of exercise elements which comprise the unique sequences. Participants were assigned a specific sequence in a preselected random order. An equal number of
participants complete each sequence.