Choice of methodology

The nature of the problem and the research paradigm

The problem that initiated this research is that of allocating educational resources to a heterogeneous and sparse population with special needs in order to provide the best outcomes. At the beginning of the research, the nature of the educational resources provided, the characteristics of the population, the desired outcomes and the relationships between them were all ill-defined. The approach to the problem needed to include defining each of the inter-related areas before beginning to explore the inter-relationships.

This problem involves both “soft” or qualitative and “hard” or quantitative aspects. The soft aspects are those that relate to human activities and the diverse nature of the individuals involved. People do not fit neatly into categories, nor are they easily measured in the way that manufactured goods or physical effects can be. Quality of service and level of need are both qualitative concepts that can be quantified to a certain degree, but only by approximation or proxy measures. The term “hard” is used to refer to aspects that can be directly quantified such as hours of service provision, and the funding provided for individuals and groups of students. These are more easily measured and the figures manipulated to give information. However even the so-called “hard” figures may mask a high degree of variation or hidden softness. For instance, an hour of teacher aide time may be allocated to a student, and it can be used for one-on-one tutoring, assistance in a large classroom, production of materials or helping a small number of students, including the student for whom it is ostensibly provided.

In this research the problem is approached from an Operational Research paradigm. Operational Research/ Management Science is a discipline which seeks to improve a problem situation, using modelling. Operational Research as a discipline was originated/developed to deal with problems involving both hard and soft aspects, and consequently the discipline has diversified in hard, soft and mixed directions. Operational Research is used to solve hard (numerical) problems related to physical systems such as coal mines, electricity generation and distribution networks, and at the other extreme it is used for interventions into hospital closures, homeless teenagers and company restructuring. In between lie problems such as flight crew scheduling, which uses a quantitative method, integer programming, to provide solutions that will suit the people involved, who have diverse and often conflicting needs.

As the application area for this problem is Education, the problem could be classified as falling within the area of Educational Evaluation; this also provides a way of approaching the problem. Educational Evaluation, like Operational Research, deals with problems that are both qualitative and quantitative in nature. Evaluating a school or educational programme will include both human aspects and quantitative aspects. The emphasis is less on problem solving than it is in an Operational Research study, and more on providing an analysis as to the state and efficacy of the programme in question for a particular set of stakeholders.

It is interesting to explore the comments of Berliner (2002, p. 18), who, in a play on the meaning of “hard”, calls Educational Research the “hardest science of all”. In response to an emphasis by the US Government on “evidence-based practices” and “scientific research”, he suggests that though physics, chemistry and geology are often called “hard sciences” and contrasted with the social sciences which are considered as “soft sciences”, the distinction is really between “easy-to-do” science (physics, chemistry etc) and “hard-to-do” science (social science and educational research). Many of the key practices or possibilities of physics, chemistry etc: controlling the context, replication, generalisation over setting and time are not possible in Educational Research. He suggests:

We should never lose sight of the fact that children and teachers in classrooms are conscious, sentient, and purposive human beings, so no scientific explanation of human behaviour could ever be complete…. When stated this way, we have an argument for heterogeneity in educational scholarship.”(Berliner, 2002, p. 20)

The process that was used in this research combined a qualitative interview-based inquiry with statistical modelling to define the aspects of the problem and provide the desired insights. This approach is now examined with respect to both the OR and the Educational Evaluation disciplines.

The Operational Research Approach: Multimethodology

Ackermann, Eden, & Williams (1997, p. 49) suggest that some Operational Researchers “are developing methods to try to resolve some of the limitations of the quantitative methods, to add to the power of quantitative methods and to provide further benefit to managers by focussing on predominantly qualitative data and unstructured problems.” They then give as examples, SODA(Strategic Options Development and Analysis), Strategic Choice and Decision Conferencing which are soft methods developed within the OR literature. In their study, “Modelling for Litigation: Mixing Qualitative and Quantitative Approaches”, a combination of “soft” and “hard” Operational Research methods was used in order to meet the needs of the problem.

In “Multimethodology: Towards a Framework for Mixing Methodologies”, Mingers & Brocklesby (1997) take a closer look at the practice of combining different OR methods and how this can deal more effectively with the richness of the real world and better assist through the various intervention stages. They suggest that there are four arguments in favour of multi-methodology: the complexity of real world problems, the multi-phase nature of many interventions, the observation that people are already using it in practice and that “arguments from a postmodern perspective also support pluralism in methodology.”(p. 492) They also observe that most management scientists who are competent in both hard and soft methodologies have been competent first in the hard aspects, then moved towards softer methods.

Ormerod (1997) draws on his own experience in applying Operational Research to explore the use of mixed methods. He describes, in chronological order, seven interventions which covered traditional O.R., Hard Systems, Soft O.R. and mixtures of these methods. He concludes that O.R. consultants “should adopt an eclectic approach. The key is to hone one’s craft skills, learn a number of methods and note when and where they seem to work.” (p. 57).

Mingers (2003) proposed general characteristics that Management Science/Operational Research methodologies share. He stated that a distinguishing characteristic was that “All management science method(ologies) … share the idea of developing models (representations) of aspects of the situation.” The models can be “mathematical, computer-based, logical, diagrammatic, or linguistic.” (p. 561). I agree that the use of models is central to O.R. The main aim of this research is to develop a model to explain the relationships within the system in question.

In this respect, this research can be classified as Operational Research. It is concerned with developing and exploring models – linguistic, diagrammatic and mathematical, in order to provide information to improve a problem situation. It begins by developing a linguistic model of the nature and purpose of the service to the learners with vision impairment. A performance measure is identified – opportunity- to-learn or access to the curriculum. Diagrammatic models are developed from the literature that explore and clarify the elements of opportunity-to-learn and its precursors and dependants. Statistical modelling and index development are used to create an instrument that can measure opportunity-to- learn. Its usefulness is examined by analysis of data from the regular population. Then quantitative data regarding learners with vision impairment is used to build statistical models that aim to inform decision makers.

This thesis illustrates a pluralist paradigm, by using qualitative research, comprising interviews and a case study analysis, alongside quantitative analysis, using index development and statistical analysis of “hard” data about individual children.

The Educational Evaluation Approach: Mixed methods

In the area of Educational Evaluation, there is a move towards mixed methods, which draw on the strengths of both qualitative and quantitative methods of enquiry. This comes from a practical need, similar to that of Operational Research, to capture the full picture in an evaluation, in a way that is richer than that which either the quantitative or qualitative paradigm can accomplish individually. Like Operational Research, it also has been influenced by “the challenges to conventional scientific wisdom raised by philosophers of science and theorists of methodology.” (Greene & McClintock, 1991, p.13)

In the chapter on program evaluation in the foundation text, “Handbook of Qualitative Research”, Greene (1994) discusses the contexts and roles of evaluation. She identifies four major genres of evaluation methodologies with their corresponding philosophical frameworks, post-positivism, pragmatism, interpretivism and critical, normative science. This research aligns with the second genre, pragmatism, which promotes practicality, control and utility. Greene states that this genre arose as a result of “the failure of experimental science to provide timely and useful information for program decision making.” (p. 532) The following description of the pragmatic genre is descriptive of the research undertaken here.

Characteristic of these methodologies are their orientation to decision making and hence to management, their primary emphasis on producing useful information, their practical and pragmatic value base, and their eclectic methodological stance. Evaluators in this genre pragmatically select their methods to match the practical problem at hand, rather than as dictated by some abstract set of philosophical tenets. (Greene, 1994, p.533)

The work in educational evaluation by Patton (2002) encourages a pragmatic approach. Patton believes the issue is about making sensible decisions about methods, depending on the nature and purpose of the inquiry. He expresses his aims as follows:

My pragmatic stance aims to supersede one-sided paradigm allegiance by increasing the concrete and practical methodological options available to researchers and evaluators. Such pragmatism means judging the quality of a study by its intended purposes, available resources, procedures followed, and results obtained, all within a particular context and for a specific audience. (p. 71).

Miles & Huberman (1994), in their sourcebook for qualitative data analysis discuss the links between qualitative and quantitative data. They describe how a qualitative focus can help in a quantitative study thus:

Qualitative data can help the quantitative side of a study during design by aiding with conceptual development and instrumentation. They can help during data collection by making access and data collection easier. During analysis they can help by validating, interpreting, clarifying and illustrating quantitative findings as well as through strengthening and revising theory.” (p. 41)

The qualitative data in this research was used in design by helping to define the problem, it aided in data collection by identifying what data was needed and from whom, and in the analysis, the insights from the preliminary study were used to help make sense of the quantitative findings.

In essence this research followed a two-phase developmental approach wherein the first method (qualitative inquiry predominantly using interviews) was used sequentially to help inform the second method (quantitative data collection and analysis), as defined by Greene, Caracelli, & Graham (1989). The pragmatic and somewhat aphilosophical approach is similar to that proposed by Patton (2002) and congruent with much Operational Research thinking.