The study was carried out to a design approved by the university’s Hu- man Research Ethics Committee under reference HREC/3016/Hall. The interviewer’s script is appended at Appendix A.
8.2.1 Format of the study
Twelve volunteers were recruited from GORS and related analytical profes- sions. In the results below, they are referred to anonymously by randomly assigned letters of the alphabet. The volunteers first attended a group pre- sentation, in which I summarised the background to the lish and provided a demonstration of the lish editor. I then conducted a one-to-one session with each participant. This consisted of a warm-up task for familiarisation with the editor, a main task which the participant was asked to carry out unassisted so far as possible, and a semi-structured interview.
8.2.2 Task description
Participants were provided with a “crib sheet” of editor functions, com- mands and shortcut keys, shown at Appendix B. They were given the small spreadsheet shown in Figure 8.1, showing weekly production figures for a gizmo manufacturer, and were first asked to replicate it in lish form (with the total calculated using a formula). They were then supplied with data for two further sites and asked to extend their lish to encompass those sites as well. Finally they were asked to write a formula that would list the managers of sites whose weekly production exceeded 100 units. A model solution in lish form is shown in Figure 8.2; the frames have been turned on in the editor to show a representative portion of the nested structure.
Figure 8.1: The initial spreadsheet for the task in the user study
Of course, the task had to be kept very small in order to fit within the time constraints of the session and to be suitable for a user who had only just met the lish. But it did exercise the fundamentals of lish modelling: using lists to make tables, and applying formulae upon ranges of cells via a single template cell. It also exercised some concepts with no direct counterpart in spreadsheet modelling, namely:
1. Column groups. Although spreadsheet users may mark off groups of columns using secondary notation such as vertical lines, they are not an intrinsic part of the structure. In a lish, columns that are to be operated on as a single object must be grouped.
2. The use of templates to initiate a family of similar objects. A tem- plate based on the ‘Arkwright’ data was used in this task to obtain a consistent structure for the other two sites.
3. Ranges consisting of non-adjacent cells. Such ranges are available in spreadsheets if the user selects each cell individually, but in a lish they can be selected via a single template cell.
4. Dynamic allocation. The final query was of a form that might produce an output of arbitrary size, but which could be embedded in a single cell.
8.2.3 The interview questions
An initial question asked about the participant’s impressions of an example dataset in long compared to wide form.
The next set of questions concerned the task. Their aim (along with observations on how the participant carried out the task) was to assess the lish in a cognitive dimensions framework. There is a difficulty here: the user interaction is with the actual editor presented to them, so we cannot
entirely separate to what extent their experience is driven by the editor itself (which is a rough prototype having no great pretensions to facility of use) and to what extent by the underlying formalism (which is hypothesised to be well suited to the task, and is what we would like to evaluate). The “notation” whose cognitive dimensions are being assessed in this trial can only be the lish as visualised on screen, along with the editor that forms its environment, rough edges and all.
This problem cannot entirely be resolved until a better editor is available! However, as noted in subsection 2.4.1, certain dimensions are heavily driven by the underlying representation, even though in the final analysis they can only be judged in the system as a whole. So as long as the editor is good enough, we might still expect to obtain some relevant information on how the lish as a model influences those dimensions. The ones I focused on in this study were:
• closeness of mapping
• abstraction gradient
• role expressiveness
• viscosity.
Participants were asked in this section to reflect on how easy or otherwise they found various aspects of building the model, from the point of view both of forming the structure and implementing the calculations. There followed two more open questions about whether the behaviour of the system was surprising or its operation felt unwieldy.
The other main sequence of questions looked at the relationship between the lish and the participant’s own work. The point of these questions was to take the RQ on effect on workflow from a different angle. It has been fairly easy to show that the lish can have an effect (and a beneficial one) on some workflows, but do those actually reflect the practices of real world analysis? And if so, do users perceive the aspects that have changed as being of any importance?
Finally, the participant was given the opportunity to make open com- ments about anything not already covered.