In Chapters 4 and 5 of this thesis a new framework to support the study of behaviour is
proposed. In Chapter 6 the information system P AC is introduced which provides a
prototype implementation for this framework. The case studies described here were performed to investigate the claim that the ideas introduced in the framework provide advanced support for the study ofbehaviour. The implementation PAC was used to conduct the case studies.
Employing the research method of case studies was seen as most appropriate because it allowed a number of behavioural domain experts to gain intimate knowledge and understanding of the concepts suggested for a new multimedia system to support the study of behaviour. The domain experts would use PAC to analyse their behavioural data. Based on the analysis results achieved, the feedback given by the domain experts, and the observations made by the author during the duration of the case studies, the usefulness of
the framework suggested could be evaluated. It was further assumed that the domain
experts, through the intensive work with PAC, would achieve enough familiarity with the concepts of PAC to be able to comment of the usefulness of P AC beyond the limitations of both the specific studies and the prototype implementation.
The emphasis throughout the case studies was on evaluating the concepts behind the implementation of P AC. While the implementation made it possible to conduct the case studies its main purpose was to facilitate the application and analysis of the concepts. To enhance its readability this case study report mostly refers to 'PAC' instead to of using a construct like 'the concepts behind P AC' .
7 . 1 Case Study Design
The next sections describe the research design for the case studies following the recommendations in Yin ( 1 993). First the case study question and the more detailed propositions are formulated. This is followed by a discussion of the selection of cases for the multiple case design. The data collection and interpretation steps for the individual case studies are outlined and followed by a section on cross-case evaluation.
7 . 1 . 1 Case Study Question
In this research a framework for a multimedia information system to support the study of behaviour is introduced. This framework contains features like the structured yet flexible coding and query languages (FSCL and FSQL), the use of digital media, and the support of quantitative and qualitative analysis techniques. The claim is made that implementing a computer application according to this framework leads to advanced support for the study of behaviour. Based on this claim the two main research questions for the case studies were formulated:
' Can PAC be used to support the study of behaviour?'
' Does PAC provide advanced support for the study ofbehaviour?'
The main research questions were divided into a number of propositions to be evaluated: • Usefulness of the flexible structured coding and query languages, F SCL and FSQL;
• The advantages of digital as compared to analog video;
• The support for combining quantitative and qualitative research methods;
• The advantages of the combined analysis of behaviour recordings of multiple media formats;
• The benefits of a multimedia conclusion construction;
• The possibility of building up a domain knowledge base.
Some of the ideas introduced with PAC could not be fully tested within the scope of these case studies. For example, building up domain knowledge within a behavioural domain can only happen over time by performing a series of studies. This was not possible within the scope of this research and therefore the domain experts could only be asked to comment on the usefulness of this idea based on their expertise.
Further restrictions arose from the fact that PAC is a prototype implementation. As such PAC has some limitations regarding the number of features available to the user. The quantitative interpretation tool, for example, offers only basic descriptive statistical functions and not the range of functions a user would expect from a full implementation. The tool for the conclusion construction is not implemented to the extent suggested in the theoretical description of the concepts which was due to the technical limitations posed by
the development environment (see Section 6.5).
7 . 1 . 2 Multiple Case Design and Study Selection
A multiple case design was chosen to produce more compelling, richer evidence compared to performing only a single case study (Yin, 1 994). The design logic aimed in the first instance at theoretical replication (Yin, 1 994; Rose, 1 991) and in the second instance at
diversity (Rose, 1 99 1 ). The research question posed for each single case study was to find evidence supporting or contradicting the claim that the proposed framework would provide advanced support for the study of behaviour. The combination of multiple case studies was to provide replication of the findings. P AC has been designed to support the study of behaviour in a wide range of behavioural domains utilising both quantitative and qualitative analysis methods. Selecting case studies representing a range of behavioural domains and research methods was to show the suitability of the framework for a diversity of studies. All three case studies selected were built around existing studies of behaviour. In one study PAC was used to analyse the behavioural data according to the analysis plan already defmed for the study. In the two other cases PAC was used to add an exploratory line of analysis to the existing studies.
There were a number of advantages of building on existing studies. Using existing studies, which had been designed independently from P AC, would help to answer the first research question, showing that PAC was a suitable tool to support the study of behaviour. Applying P AC to their existing studies made it easier to motivate the domain experts to invest a considerable amount of time into the understanding of PAC. The domain experts were interested in advancing their studies and recognised PAC as a tool which could help to further their goals. Another advantage of using existing studies was a considerable time saving compared to setting up new studies. In all three studies selected the behavioural data had already been collected and the data analysis could start.
For two reasons the three case studies were performed in parallel. Firstly, the total duration of each study was about one year and it was therefore, in the given timeframe for the whole research, necessary to perform the studies in parallel. Secondly, with the aim of replication, the case studies were supposed to evaluate the same theoretical framework and therefore could easily be conducted in parallel.
7 . 1 . 3 Case Study Evidence
Burns ( 1 997) lists four sources of case study evidence, namely documents, interviews, participant and non-participant observation, and artefacts, which all have been used in the case studies described.
The case studies were performed in close interaction between the domain experts and the author. Numerous meetings were held to prepare the case studies and discuss analysis steps. The author produced summaries of these meetings and distributed them to the domain experts. These summaries captured the ideas discussed in the meetings and outlined the developing analysis steps. In one case study all preliminary discussions were conducted by
email. In this case complete documentation was available by nature of the communication mechanism.
The application program PAC was initially operated by the author, demonstrating its use to the domain experts. Step by step the domain experts learned to use the program themselves until they became relatively independent users of the main functions of P AC. During the use of the program PAC to perform the analysis of the behavioural data, the role of the author changed from participating observer to non-participating observer. In the initial phases of working with P AC the domain experts needed a relatively high amount of guidance. The author not only operated PAC but also suggested which features ofPAC could be applied to a specific analysis task. This occurred mainly in the phase of the case studies where the domain experts were still fmalising their analysis steps. By the time these steps had been fmalised, the domain experts had become familiar enough with P AC to operate the program
largely by themselves. All analysis of the behavioural data was performed solely by the
domain experts. The author remained present while the domain experts were operating P AC, but was mainly there to give technical advice in using P AC and to correct any problem which might occur considering the prototype status of PAC.
The analysis with P AC produced a number of artefacts. These were listings of the vocabulary, the description sentences, the queries and query results, information about the set structure chosen for the studies, and graphical presentations derived from query results. All these artefacts documented the way PAC was used to support the studies ofbehaviour.
After the domain experts had finished their studies, semi-structured interviews were conducted. The domain experts were asked to both comment on their experiences of using P AC in the concluded studies and in relation to potential future studies in their areas of
expertise. Figure 7 . 1 shows the questions the interviews were based on.
7 . 1 . 4 Analysis of Evidence
The analysis of the case study evidence was aimed at providing answers to the case study propositions and, based on these, the main case study questions. To prepare the analysis, the audio recordings of the interviews were transcribed. The documents, artefacts and notes taken during the observations were sorted and reviewed. Table 7. 1 shows which main
sources of evidence were used to answer the propositions.
The evidence not used directly to evaluate the propositions contributed, together with the findings for the propositions, to answering the main case study questions. This was evidence contained in the documents, artefacts, and observation notes as well as discussions following some of the interview questions. Questions 4 and 8 aimed at a wider context beyond the limitations of the current case studies, whereas questions 9 and 1 0
Questions to the Domain Experts
These questions are asked in the context that you have conducted a study of behaviour using P AC. Please answer the questions based not only on your experiences of using P AC for your current study but in context of the range of studies you have performed or plan to perform.
I . PAC supports the combined analysis of behaviour recordings of several media types (video, text). How can you make use of this feature in your range of studies?
2 . What impact do the concepts o f the PAC coding language and query language have on your studies of behaviour?
3 . Did the development o f the vocabulary help you to gain a better understanding o f your data or planned analysis steps?
4 . Does using P AC influence your method o f analysis? If yes, how?
5 . Two of the concepts PAC uses to support both quantitative and qualitative analysis methods are: - the flexible and structured coding language/query language to allow both rich and precise descriptions of behaviour
- the provision of interpretation tools to view behaviour recording sequences, to perform basic calculations/comparisons, to study the extracted behaviour descriptions
Do you agree that these concepts facilitate the use of both quantitative and qualitative analysis methods?
6. The PAC implementation indicates how the presentation of the conclusions of a study can be supported by analysis data (graphs, behaviour recordings, .. ). Can you see how you could use this feature?
7. The concept behind P A C suggests that domain experts could over time, b y conducting several studies using P AC, build up a collection of 'domain knowledge' consisting of behaviour recordings, vocabulary, analysis data and conclusions. Do you think that this would be useful?
8. How would you use P AC for further behaviour analysis studies?
9. Would you like to make further comments on any aspect of the concepts of PAC?
I 0. Would you like to make any comments on the strength or the weaknesses of the implementation of PAC?
Figure 7. 1 : Questions to the domain experts
prompted the domain experts for general comments on any of the aspects of concept and implementation of PAC. The case study reports for the individual studies were presented to the respective domain experts for comment and verification. The feedback from the domain experts was integrated into the reports.
7 . 1 . 5 Cross Case Evaluation
The cross case evaluation followed two directions in accordance with the multiple case design logic looking for replication and diversity. To check for replication the results of the
Main Source of Evidence
Usefulness of the flexible structured coding and query languages, Questions 2 and 3
FSCL and FSQL
The advantages of digital as compared to analog video Observations and discussions during case studies
The support for combining quantitative and qualitative research Question 5
methods
The advantages of the combined analysis of behaviour recordings of Question I media formats
The benefits of a multimedia conclusion construction 6
The to build a domain base 7
T able 7. 1 : Propositions and main source of evidence
individual studies were compared on a proposition and main case study question level.
Additionally the single cases were investigated in regard to diversity. Characteristics of the individual studies were identified with the aim to highlight different needs for support.
7 . 2 The Individual C ase Studies
All three case studies were largely performed according to a common format. The first
phase was the familiarisation phase which was a two-sided process. The behavioural domain experts had to become acquainted with the concepts and implementation of P AC. The author had to understand the behavioural studies the domain experts were working on to be able to suggest possible uses of PAC. Following on from the first familiarisation the scope of the case studies was defmed. The needs of the domain experts to gain some benefit for their own studies and for the author to test the propositions had to be balanced. Time and resource restrictions had to be considered. The next phase consisted of technical preparations for the studies like setting up necessary structures in P AC and digitising the
analog video recordings. From there the development of the PAC vocabulary followed. As
will be seen in the individual case study reports this step required considerable time because the vocabulary is closely linked to the concepts of the coding and query languages and therefore central to the main concepts of P AC. In the next steps, the behaviour recordings were coded and the resulting descriptions queried.
Following the exploratory nature of the case studies, all the phases described so far did not occur in a strict linear sequence. A number of the issues were addressed repeatedly. The growing understanding of the domain experts for P AC raised new questions and produced new ideas ofhow to use PAC to advance the studies of behaviour.
As the practical work with PAC had come to some form of conclusion and the domain experts had developed a good understanding of PAC, the semi-structured interviews were
conducted. The domain experts were asked to report on their experiences of using P AC in their current studies and on the prospects of using PAC for other studies in their fields of expertise.
7 . 2 . 1 The Reading Groups Case Study
The Reading Groups case study was based on a study titled 'Contexts of Support: Learning to Read in Small Groups in New Zealand' which builds on work by Wilkinson and Townsend ( 1 999). The purpose of this study was to analyse the parameters which contribute to learning to read in New Zealand primary schools. The context of the study was strong criticism of teaching reading in small groups that have been formed based on the ability of the students. Overseas research suggests that this kind of grouping disadvantages weaker readers. This study attempted to show that group tuition is successful in the New Zealand context, where the group membership of students is very flexible and the instructional practices of teachers may overcome the potential disadvantages for the weaker readers.
The study entailed a detailed analysis of the instructional procedures of selected "best practice" teachers and the reading abilities of a number of selected students over a year. Three different types of data were collected: video recordings of reading lessons, audio recordings of post-lesson stimulated recall interviews with the teachers, and results of word knowledge tests given to the students before and after the reading lessons. The study design included three different forms of coding. The video recordings were to be coded for 'word learning opportunities', describing every instance where one of the target words (as defmed by experienced teachers who were not involved in the data collection) featured in the lesson in any form. In addition, the video recordings were to be coded for 'procedural engagement' , capturing the attention status of each student. The third form of coding was to be based on transcripts of the interviews with the teachers. These interviews were to be
coded for 'substantive engagement' of the students in the lessons according to the statements made by the teachers as they watched the videotapes of the lessons. The
descriptions resulting from the three different forms of coding were to be combined and
interpreted in the context of results from the word knowledge tests.
7 . 2 .1.1 F amiliarisation with P AC and Scoping of Case Study
The familiarisation of the domain expert with PAC occurred over a period of nearly half a year in the form of ernail messages and electronic exchange of documents. The domain expert was located about 600krn from the site where PAC was implemented. The prototype status of the PAC implementation and its hardware requirements made it impossible to transport P AC to the user.
Initially descriptions of the behavioural study and general descriptions of P AC were exchanged. The behavioural study was well defined and a number of parameters for the analysis had been specified. Yet for one central part of the analysis, the analysis of the word learning opportunities, neither theory nor analysis steps had been fmalised. These were to evolve in the course of the study.
Utilising the initial project description, the domain expert was presented with suggestions of how P AC could be used to facilitate his analysis. These suggestions started the exchange of a series of documents with the aim of developing a better understanding of the behavioural study on the one hand and of PAC on the other.
Once the conceptual issues had been clarified and the domain expert could see the value of