Validity of the MB

Nunan (1992: 15) defines validity as the extent to which a research study actually investigates what the researcher claims to investigate. Validity can be divided into internal and external validity.

Internal validity

Internal validity is concerned with credibility and whether the research findings match reality, that is, is the researcher observing or measuring what they think they are measuring? And has the researcher taken steps to ensure that the research results are not affected by external variables?

According to Seliger et al. (1989: 94-104), factors that can affect internal validity or the credibility of the research findings are:

Subject variability: as one assumes that the population used in the research is

representative of the general population to which the research results would apply. The research can achieve greater representativeness by using random sampling in which subjects are chosen at random from a larger pool of potential subjects;

o In this research study, a purposive or convenience sample was used and there was no need to randomly assign participants to different groups or to undergo different procedures. However, there was an element of random selection in the

fact that the TESOL teachers who participated in the research study were from different private language schools, ages, genders and backgrounds;

o Strict limitations were also put in place to control subject variability (see Subjects).

Size of subject population: small populations tend to magnify the effects of individual variability which can cause possible distortions to the data. The greater the size of the population, the smaller the effect of individual variability and the smaller the number of subjects, the more the study is susceptible to biases created by an over-representation of some subject characteristic;

o Merriam (2009), however, maintains that a small sample can be selected precisely because the researcher wishes to understand the particulars rather than what is generally true of many;

o Although this research study involved a small number of participants (forty- three), the researcher selected participants from different private language schools as it was possible that over-representation of TESOL teachers from a particular language school would have distorted the data as those teachers may have been experiencing elevated stress levels due to internal conflict at the school.

Time allotted for data collection: this factor is important when the researcher is investigating a change in the sample population over time;

o In this research study, the main aim was to investigate burnout at a particular point in time, that is, to obtain a ‘snapshot’ view of the phenomenon. This

supports Nunan (1992: 140) who states that the “purpose of a survey is generally to obtain a snapshot of conditions, attitudes and events at a single point in time.” The passage of time was thus not of concern in this research study. Comparability of subjects: the researcher placed strict limitations on the participants in the study so as to increase the comparability of the participants (see Subjects).

History, attrition and maturation; these factors are important when time is necessary for a particular treatment to have an effect, that is, the research may be negatively affected by the passing of time. In terms of history, longitudinal studies may be negatively affected by the passage of time. Attrition is when the composition of the population changes the longer the study continues which affects the data negatively, and maturation is particularly significant with research involving younger subjects who may mature and thus change over the course of the research study;

o In this research study, the effects of history, attrition and maturation did not affect the study.

Instrument and task sensitivity; refers to the effect that the test or instrument has on the subjects as participants may become test-wise with the pre-test or pilot creating a

practice effect, that is, the pre-test instrument or pilot study may affect the attitudes of the subjects by sensitising them to the questions;

o Instrument sensitivity is one of the reasons why Merriam (2009) advises researchers to use multiple methods of data collection such as surveys,

interviews, documents and observations. This enhances the credibility of a study. Thus data can be compared and cross-checked.

o All the participants completed phase one, the MBI-ES and then selected

participants completed phase two, the semi-structured interviews. Participants who did not complete phase one could not take part in phase two. The MBI-ES is a confidential document and participants were assured that people other than the researcher and the researcher’s supervisors, would not have access to the completed surveys and the results.

o Some participants may have falsified their responses on the MBI-ES so that burnout could not be detected. Conversely, some participants may have chosen responses which indicated high burnout, when in fact the responses were not indicative of their true feelings. However, there was no advantage to the participants in falsifying their responses on the survey. The researcher ensured that participants understood that there were no right or wrong answers. External validity

External validity refers to being able to generalise the findings to situations outside that in which the research was conducted, that is, the transferability of the research results from the sample to a wider population (Edge et al., 1998). Lincoln et al. (1985) maintain that the burden of proof lies less with the original researcher than with the person seeking to make an

application for the research elsewhere as the original researcher cannot know the contexts to which transferability may be sought. The onus is, however, on the original researcher to provide sufficient descriptive data to make transferability possible.

According to Seliger et al. (1989: 106-110), factors that can affect external validity are: Population characteristics: this is concerned with the degree to which the sample population in the study have the same characteristics as the population to which the research findings are to be applied, that is, is the population used in the research a specific subset of the larger population?

o In this research study, the sample population consisted of TESOL teachers working at private language schools in Johannesburg. The teachers came from a range of backgrounds with different qualifications and teaching experiences and also differed in terms of age and gender. The researcher did, however, restrict the sample to first language speakers of English so that the results could be

applied to the general population of English-speaking TESOL teachers working at private language schools in South Africa. The researcher would be cautious though about applying the results of the research to native speaker TESOL teachers working in other countries as the working conditions of TESOL teachers and perceptions of the profession differ from one country to another.

Interaction of subject selection and research: this is a problem with paid or volunteer subjects as they may not be representative of the wider population to which the research findings are generalised.

o In this research study, the participants did not receive any form of payment and were not volunteers from the general public. There was no danger of subjects participating in the study for monetary reasons or subjects volunteering to participate in the study so as to attract attention.

The effect of the research environment: the fact that subjects are aware of what the study is about may change their behaviour which distorts the research results. Subjects may, therefore, feel they are suffering from burnout when in fact they are not because just knowing the study is about burnout may heighten the participants’ awareness of the phenomenon.

o In this research study, it was not possible to conduct the research without the participants being aware of the nature of the research. Burgess (1984: 48) maintains that covert research, in which the subjects are not aware of being studied or the true nature of the study, can actually limit the range of research activities as the researcher only has access to those situations which are

observed. It is, therefore, not possible to conduct interviews, collect life-histories or documentary evidence produced by a particular group of people.

Researcher effects: the researcher may unintentionally ask leading questions so as to obtain the information being sought or provide clues through a change in the tone of voice or body language;

o All data was collected and analysed in an objective manner and the findings reflect the actual data obtained from the subjects. In addition, the researcher kept a research journal which provided a detailed account of the methods, procedures and decisions made in carrying out the research.

o Seliger et al. (1989: 104) mention the importance of retrievability which refers to the researcher being able to ‘retrieve’ their research records such as

participants’ responses on surveys and interview transcripts. It is important that the researcher keep meticulous records of data collected so that the data can be inspected and reviewed if necessary. Thus the researcher should collect the data by mechanical means such as audio recording and transcripts of interviews should be made.

The effect of time: this is concerned with the degree to which the time frame of the research context can be applied to the real world to which the results will be

generalised.

o In this research study, the aim was to obtain a ‘snapshot’ in time and not to conduct a longitudinal study of burnout. The researcher wanted to discover whether TESOL teachers currently working in private language schools were experiencing burnout.

According to Maslach et al. (1981: 7), convergent validity, the degree to which a measure is correlated with other measures that it is predicted to correlate with, was demonstrated by respondents MBI scores being correlated with behavioural ratings made independently by a person who knew the individual well such as a co-worker or spouse. MBI scores were also correlated with the presence of certain job characteristics that were expected to contribute to experienced burnout such as lack of control and work overload. Finally, the MBI scores were correlated with measures of various outcomes that had been hypothesised to be related to burnout such as dissatisfaction with opportunities for personal growth and development on the job and the belief that one’s work was not very meaningful or worthwhile. The three sets of correlations provided substantial evidence for the validity of the MBI.

Maslach et al. (1981: 9) state that it is important to distinguish the MBI from measures of other psychological constructs that might be confounded with burnout such as feelings of

dissatisfaction with one’s job. By so doing, discriminant validity, which tests whether concepts that are supposed to be unrelated are actually unrelated, would be demonstrated. One would expect the experience of burnout to have some relationship to lowered feelings of job

satisfaction, however, it was predicted that they would not be so highly correlated as to suggest that they were actually the same thing. Maslach et al. (1981: 9) conducted a comparison of ninety-one social service and mental health workers scores on the MBI and the JDS measure of ‘general job satisfaction’ to provide support for this reasoning. The researchers found that job satisfaction had a moderate negative correlation with Emotional Exhaustion (r = -.23, p < .05) and Depersonalisation (frequency only r =-.22, p <.02) as well as a slightly positive correlation with Personal Accomplishment (frequency only r =.17, p < .06). The researchers concluded that since less than 6% of the variance is accounted for by any one of these correlations, one can reject the notion that burnout is simply a synonym for job dissatisfaction.

It might also be argued that scores on the MBI are subject to distortion by a social desirability response set because many of the items describe feelings that are contrary to professional ideals. To test this idea, forty graduate students in social welfare were asked to complete the MBI and the Crowne-Marlowe (1964) Social Desirability (SD) scale. Maslach et al. (1981)

maintained that if reported burnout is not influenced by a social desirability response set, then the scores on the MBI and the SD scale should be uncorrelated. The results supported this hypothesis as none of the MBI subscales were significantly correlated with the SD scale at the .05 level.