THE RIGOROUS EVALUATION

In Zwarenstein et al’s (1999) systematic review of interprofessional education they concluded that IPE providers should build into their research projects both quantitative and qualitative evaluations. Although this paper was published after my study had been planned and begun, preliminary data from it was presented at a conference, reported by Boaden (1998), which was highly influential in my thinking regarding the rigorous evaluation.

The two day international conference was convened on 27th and 28th November 1997 it was entitled Interprofessional education: Does it work? Two of the keynote speakers at the conference were Sir Kenneth Calman the Chief Medical Officer and Professor Michael Orme, Director of Education and Training at the NHS executive, two key individuals who both demonstrated their support for the development of interprofessional education. But the most interesting speaker from my perspective was Dr Merrick Zwarenstein, Division Head of Health Systems Research at the Centre for Epidemiological Research in South Africa. He presented a paper on behalf of an IPE review group called "Findings from a retrospective systematic review of the effectiveness of interprofessional education".

They suggested that evaluations of IPE should take the form of randomised controlled trials. Yet there are several difficulties inherent in trying to set up such trials in an educational context.

Firstly, it is as yet unclear as to which variables affect the outcomes of interprofessional education therefore it is not possible to control for such variables with any certainty. Secondly the idea of random allocation of subjects to control and experimental groups is problematic. When using aware human beings in education it is extremely difficult to obtain a valid control group as it is likely that they would know if they were not having an interprofessional educational experience. This would bias the results, as those who were in the control group would expect not to score highly on post intervention measures. Thirdly many interprofessional education initiatives involve relatively small groups of students once randomised into control and experimental groups the numbers are likely to be so small as to make inferential statistical analysis weak.

Fourthly there is an ethical dimension to this type of educational intervention. If interprofessional education is beneficial to students’ collaborative practice it may be unethical for education providers to deny them the right to join in with such educational initiatives. Finally one of the strategies that has been employed to overcome the organisational difficulties of delivering IPE at undergraduate level is to make such programmes optional. This means that the pool of students for IPE would be drawn from those who had self-selected for the programme and this may also bias the results. Another strategy was required to evaluate the IPE programme that was still considered by the research community to be rigorous. The Cochrane Collaboration discussed in the last chapter, were influential in directing me to another research design.

5.2.1 Research design: The interrupted time series with non-equivalent groups

The Effective Practice and Organisation of care (EPOC) subgroup of the Cochrane collaboration state that other methodologies such as the controlled before and after study and the interrupted time series design have sufficient rigor to produce meaningful results (Barr et al 1999b). These are quasi-experimental designs and are different from traditional experimental designs.

A ‘true’ experimental design studies rigorously, cause and effect relationships between variables. Cook & Campbell (1979) describe how experimental design features have been used since as far back as the seventeenth century. They state that more recently theory of experimental control through randomisation of treatment subjects has been used. This is where variables can be manipulated in order to demonstrate that one variable or group of variables, the independent variable(s), has an effect on another variable, the dependent variable. By randomly assigning subjects to experimental or control groups it is possible to extricate the effects of other variables, confounding variables, on the outcome and therefore demonstrate the relationship between independent and dependent variables.

Another type of experimentation, which is considered to be less rigorous than true experimentation (Payton 1994), but by the Cochrane Collaboration to be of equal value is the quasi-experiment. Cook and Campbell (1979) quote Stouffer (1950) and Campbell (1957) who describe quasi-experiments as those that have treatments, outcome measures and experimental units (subjects) but do not use random assignment to identify the

treatment inferred change. Pollit and Hungler (1978) also state that such experiments lack the randomisation component but go on to say that the control component or both randomisation and control components may be missing.

So the two quasi-experimental designs suggested by the Cochrane Collaboration were the before and after control and the interrupted time series. Cook and Campbell (1979) discuss the relative merits of these two designs. The key confounding factor in both are the effects of history. Factors that occur prior to establishing a control group may have an effect on the outcome measures and reduce the validity of the causal relationship between variables. The interrupted time series enables a greater degree of history to be taken into account and was therefore the one that was selected for this research.

In the interrupted time series design (Payton 1994 and Cook and Campbell 1979) the effects of the treatment are determined by taking several measures of performance or criterion measures over time both before and after treatment. There are several different types of interrupted time series and Cook and Campbell suggest that by combining the longitudinal component of the time series with the cross sectional comparability of non- equivalent group designs it may be possible to improve the quality of the design.

Subjects for control groups for each profession were sought who could be closely matched for independent variables that may have an effect on the performance measures. This would provide an interrupted time series with a non-equivalent no treatment (IPE) group design.

(O1, O2, O3, O4, X, O5, O6, O7, O8) experimental group

(O1, O2, O3, O4, O5, O6, O7, O8) control group

---> time

O = performance measure, X = treatment (IPE programme)

Figure 2: The interrupted time series with a non-equivalent, no treatment (IPE) group design.

5.2.1.1 Non-equivalent control groups.

There were four professions involved in the pilot IPE module and control groups had to be found for each. When randomisation is not possible, as in this case, it would be usual to select subjects for a control group that were matched for specific characteristics with those in the experimental group. These characteristics would be the independent variables that are known to have an effect on the outcome measures. In this way several key variables can be excluded as having had an effect on the outcome measures. However in IPE the variables which affect the outcome are not well known.

Shaw (1994) used control groups in his evaluation of an IPE programme for those involved with learning disability but does not state how he allocated individuals to either group. He does state that he used control subjects that he matched to the experimental subjects using the same unit (workplace), grade and client ability mix presumably because he believes these might have an effect on the outcome of IPE. Yet he provides no evidence for this. As well as this problem of lack of knowledge of the independent variables involved in IPE there is also another practical problem with this type of field experiment. Absolute control of a field experiment by the researcher is not possible and therefore it is often necessary to compromise. When selecting control groups for the pilot IPE module institutions were identified which delivered similar professional programmes as the students at Salford but did not contain any formal MPE.

The occupational therapy students taking part in the module had a ready made control group as only some of the cohort had volunteered for the pilot IPE module. Those not taking part were selected as controls. Controls for the radiography, nursing and midwifery students were drawn from institutions in London, the Midlands and North of England and each had comparable programmes as the experimental group but with no formal MPE within the programme. The radiography control subjects were from one institution, the midwifery and nursing subjects were drawn from two institutions. This was to increase the sample size that was considered to be too small if it was hewn from any one institution. The nursing subjects came from both those Salford nursing students not taking part in the IPE pilot module and nursing students from another institution. Again this was intended to increase sample size.

Profesions Experimental Group Control group Radiographers 30 28 Midwives 10 23 Occupational Therapists 7 26 Nurses 4 26 Total 51 103

Table 6: The numbers of subjects in control and experimental groups.

5.2.1.2 The interrupted time series

The intervention was planned for the month of February 1999 so the strategy was planned around this. Two pre-intervention and two post-intervention measures were organised. Influential in this planning was the work of Shaw 1994. His study evaluated the outcomes of an Open University Course. The course aim was:

“to equip students with the knowledge and skills necessary to encourage beneficial changes in the lives of mentally

handicapped children and adults.” p43

which puts it into Level 2 of Kirkpatrick’s framework. The participants were a range of professionals, volunteers and parents involved with learning disability. The method used was a before and after intervention design with control groups. Subjects were measured before the programme, immediately after and then five months after the programme.

There was evidence of a change in perception by participants of the value of work of another organisation or profession. In some professionals there was also evidence of a change in the perception of the role of the other professions involved. This change became less pronounced five months after the programme.

This interesting finding at five months suggests that the longevity of the measurable effect of IPE might be short lived. So an eight month follow up measurement was

incorporated into the strategy to see if the pilot IPE project had effects which lasted after the end of the programme and if so whether they lasted longer then Shaw’s. It was also thought that a longer-term follow up might be of value. The students on the pilot would qualify and be practising professionals at the time of the eight month measurement so several practical difficulties were anticipated including loss of subjects as they dispersed into their first professional placements.

Having considered the most appropriate research design and the non-equivalent control groups there were still several questions which needed answering before completing the research strategy. Which quantitative and qualitative measures should be used? What were the key outcome measures? Where there any valid and reliable tools published that I could use?

The development of the research strategy did not proceed in a logical and sequential order. Rather articles and research texts I had read influenced my thinking and I developed my research awareness as I went along. Triangulation had been used by several authors who used it to get at the 'truth’ when researching human beings.