THREATS TO THE VALIDITY OF THE STUDY

As with all research designs, there are various methodological weak-points that can threaten the studies’ validity at various stages of the research process. These issues need to be taken into consideration by researchers and need to be minimised or eliminated where possible. In an experimental research design there are various internal and external validity threats that should be considered as possible limitations when interpreting the findings.

A study possesses internal validity when the results of the research, scientific observations and measurements, are true representations of some reality (LeCompte & Goetz, 1982). When the studies’ findings follow in a direct and unproblematic way from its methods, the study will possess internal validity, meaning that all finings can be attributed to the independent variable, in this case the EI intervention (Terre Blanche, et al., 2006). Any inferences drawn from the experimental results, that do not accurately reflect what happened in the experiment itself, therefore threaten the studies’ internal validity. Thus internal invalidity occurs when anything else, but the experimental stimulus, affects the dependent variable.

Babbie and Mouton (2010) expanded on the works of Campbell and Stanley (1963) and Cook and Campbell (1979) and identified twelve of these internal validity threats namely: history, maturation, testing, instrumentation, statistical regression, selection bias, experimental mortality, causal time-order, diffusion of treatments, compensation, compensatory rivalry and demoralisation.

History refers to any historical event that occurs during the course of the experiment that will

most likely influence the results. These events can occur anywhere between the first and the last test session. Maturation stresses the fact that people are continuously growing and changing, whether in an experiment or not. Therefore, long term experiments are especially vulnerable to this type of invalidity. Furthermore the process of Testing and retesting will influence individual’s behaviour. Once a pre-test is conducted participants often become sensitive towards the topic, and are therefore more thoughtful in their second test opportunity. Individuals also try to be on their best behaviour during tests, and try to provide answers that they think researchers want, or which makes them look good. In the current study the threat of test, retest invalidity could certainly have played a role as participants received a report after each assessment to reflect on their EI profile. It could be that when participants were

confronted with low scores in their profile they may have attempted to answer the questions in a more favourable manner in the subsequent assessments.

Instrumentation refers to the conceptualisation and measurement of variables. It is important

not to change the conceptualisation or the way in which variables are measured, or by which instrument it has been measured during the course of the research process. In the current study all measurements remained the same and no changes in the operationalisation of variables were thus present. Statistical Regression is mainly considered with extreme scorers where change can only happen in one direction. Once the change does occur it is often mistakably interpreted as a result of the intervention. Selection Bias refers to the different ways of selecting subjects that might not be comparable to one another. Experimental

Morality is when subjects drop out or leave the experiment halfway through the process while

only those interested in the matter, or who side with the interventions’ viewpoints remain. This results in biased results. Similarly participants who willingly join an experiment often have a certain inclination towards the subject matter. Other individual’s opinions are then lost as they do not form part of the study due to their different disposition. Although attempts were made to include students from two faculties in order to better stratify the sample, experimental morality was still evident and a lot of participants withdrew, probably due to lack of interest, resulting in only those who felt strongly about the development of EI, to remain in the study. Causal Time-order refers to any ambiguity about the time-order of the stimulus and the dependant variable, and whether the dependent variable caused changes in the stimulus and not the other way around.

Diffusion of Treatment occurs when the control group becomes contaminated due to

interaction with experimental group individuals. Similarly Compensation can contaminate the control group as they are often deprived of something (in this case no EI training) and therefore are often compensated by something else. This compensation then causes the control group to no longer be a genuine control group. Compensatory Rivalry occurs when subjects who are deprived of the stimulus to compensate by working harder in an attempt to beat the “special” experimental group subjects. Finally Demoralisation is the opposite of

Compensatory Rivalry where control group subjects feel demoralised due to the absence of

the stimulus and therefore give up, or reduce their normal levels of effort.

Internal validity however only accounts for some of the complications faced by researchers. Even if results can be obtained uncontaminated and accurately reflect what happened during

the experiment, it still does not mean they can be generalised to the real world. In addition, there is what Campbell and Stanley (1963) refer to as external validity, that relates to the generalisability of experimental findings to the real world (Babbie & Mouton, 2010). According to LeCompte and Goetz (1982) external validity refers to the extent which the true representations of scientific measurement and observations could be legitimately applied across groups. A study therefore demonstrated external validity when its findings and conclusions can be generalised beyond the boundaries of the design and the study setting (Terre Blanche et al., 2006).

Campbell and Stanley (1963) describe external invalidity as any interaction between the testing situation and the experimental stimulus. The authors identified four possible forms of this problem. The first is the reactive or interaction effect of testing. A pre-test might increase or decrease the respondent’s sensitivity towards the experimental variable. The second form is the interaction effect of selection bias and the experimental variable. The third form is the reactive effects of experimental arrangements and the final form is due to multi-treatment

interference, likely to occur where multiple treatments are applied to the same respondents,

as the effects of previous treatments cannot be easily erased. Although some of these forms are hard to evade, both internal and external validity threats should be minimised as far as possible. When the results are presented the possibility of any further internal or external validity threats would be considered and discussed where necessary.