VALIDITY

According to Golafshani (2003:598) ‘validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are’. Salkind (2006:106) expands on this statement regarding validity, by asserting that it is essential that researchers check three issues: 1) the results of their tests, 2) the degree of validity of these tests and 3) the context in which the test occurs. However, Le Grange and Beets (2005:115) argue that one must be careful of placing the emphasis on the test, but rather concentrate on the interpretation of the data. In order to do this, it is necessary for researchers to choose their instruments of testing with great care to ensure that reliability does not skew the interpretation of the data. Henning (2007:146-148) claims that one validates the interpretation of data in a study by checking for bias, neglect, lack of precision, and questioning all procedures and decisions critically before asking peers and other researchers to criticize one’s theories. She stresses that precision throughout the whole research process is of the utmost importance.

Nieveen (2009:89) defines educational design research as

the systematic study of analyzing, designing and evaluating educational interventions in order to solve complex educational problems for which no ready-made solutions are available and to gain insight in key design principles.

If the principles of design research are to lead us to design similar interventions for similar settings, precision and accuracy of validity are crucial in all cases.

In this particular study it became clear that four types of validity, as discussed in Cohen et al (2008:133–139), were evident:

 internal validity that shows how well the interpretation of data is corroborated by the source which is used to support it;

 external validity which refers to the degree with which results can be generalised to other cases, people or situations;

 content validity which refers to the depth of all the areas measured (Creswell, 2009:137); and

 catalytic validity which helps participants to understand their world in order to transform them and to empower the people.

To test whether the researcher was accurately examining the impact of the Natural Approach intervention on their communication skills it was necessary to answer certain questions, which tested the truthfulness of the results and the accuracy of the instruments. The tests were set according to standards used in Communications Skills classes and they were discussed with another Communication Skills lecturer. They were then marked according to the same rubric and moderated. Catalytic validity was tested by the students’ own comments. Therefore, all possible care was given to the designing of the written test, the oral presentation and interview to ensure that pertinent, accurate and relevant data would be obtained to contest or support the arguments set out in the literature review.

In the following section, I will discuss how internal, external, content and catalytic validity influenced my study. To conclude each section, I have mentioned one possible threat to validity. However, the final paragraph mentions two further threats to the validity of my research.

3.9.1 Internal validity

Internal validity seeks to establish whether the interpretation of research data can be corroborated by the data obtained. ‘Empirical data helps researchers to gain insight into the quality of the tentative intervention principles’ (Nieveen, 2009:91).The findings should show precisely what sources were used to assess the phenomena being researched accurately. Henning (2007:146) agrees that the investigation must be done in a way that will reflect the truthfulness of the subject and enough time must be spent on the research to establish the relevance of the focus.

Problems arising here could be from a lack of time to delve deeply and that participants may not want to answer too many questions in a questionnaire (Cohen et al, 2008:137). The accuracy sought can, however, also be threatened if the researcher does not check that the participants’ experiences within the testing do not affect the researcher’s ability to explain his/her findings The nature of researchers is to be biased when they investigate the world, because they are human and part of the world. Therefore, it is very important to check if other researchers may have different perspectives of the same subject and these points of view (Marshall & Rossman, 1995:143-145).

In this study, the internal validity of the written test and the oral presentation was checked by using the criteria from a syllabus created by the Western Cape Education Department to establish the standard the students were required to have when entering the university and before the intervention took place. In an attempt to prevent bias a Communication Skills lecturer was also asked to use the same rubrics as can be seen in Appendices 3 and 4 on pages 179 and 181 to moderate the tests. I had used these rubrics for the written tests and the oral presentations. These rubrics for marking and moderating written tests (Jacobs et al, 1981) and the oral presentations (University of Cape Town, 1980) are established criteria and were obtained from a neighbouring university.

3.9.2 External validity

External validity means the degree to which results can be generalized to other cases (Creswell, 2009:165). It seeks to establish whether we can transfer these findings to another set of people or circumstances (Marshall & Rossman, 1995:143-145). In design research in particular, accurate information could lead to educational designers being able to design similar interventions for similar settings (Nieveen, 2009:91).

The comments made by respondents and critics during the interviews could provide improvements with the aim of making an intervention generally usable in a similar setting. Nieveen (2009:98) also recommends that the design researcher herself does

formative evaluations of the prototype. As I had only done one cycle, I used my learning experiences, in addition to the comments of a colleague teaching Communication skills, to consider transferability and comparability (Cohen et al, 2008:137). This would allow me to look at studying the typical situations for similar circumstances. In the post-interview, observations made by respondents, which was based on their reflections, also played an important role in shaping the intervention so that it could used by in similar situations.

A possible threat to external validity may be the knowledge and attitude of the facilitator who wanted the intervention to be a success and had to guard against this.

3.9.3 Content validity

Ensuring content validity in a study means that the instruments used should show that they thoroughly cover the items being researched in enough depth. The weightings of all the elements of the main issue discussed must also represent the wider subject fairly. Researchers should ensure accuracy of data when they look at the depth of a situation, as there may be many representations of a situation (Cohen et al, 2008:136-137).

To ensure content validity in this study it was necessary to ascertain whether the testing covered a fair representation of what the results should represent (Cohen et al, 2008:147). The content of the intervention programme discussed in chapter 3 is clearly reflected in the content of the rubrics of all three instruments used (written work, oral presentation, and interviews). The criteria of the rubrics added content depth and validity to my study. The content of the intervention is fully supported by the NCS of the Western Cape Education Department (WCED) as can be seen in Chapter 1 on page 15 in its aims to create lifelong learners who are confident, independent, literate and numerate as well as compassionate and able to participate in society as critical and active citizens. It also fulfils the requirements of the Washington Accord (see Chapter 2 page 51) which requires a module within the engineering degree dealing with communication, team work and ethics as opposed to the technical training of engineers.

A possible threat may be personal bias in the interpretation of marking the pre- and post-tests. Lincoln and Guba (1986:76) claim that the researcher should bear fairness in mind here at all times.

3.9.4 Catalytic validity

According to Cohen et al (2008:139) catalytic validity embraces the paradigm of critical theory. The agenda for catalytic validity is to help participants to understand their worlds in order to transform them. It is not only about improving the participants’ experiences of the world but also improving the empowerment of the participants.

In this study, participants were from a multicultural diverse background. One of aims of the intervention programme was for students to define themselves and what they stood for. At the beginning of the year, ground rules were formulated. Respect for others and fairness was paramount. During the classes, they were given the opportunity to discuss the problems of their worlds and how they were affected by the views of their peers. In this manner, they were exposed to the different South African cultures as well those of the rest of the world.

This reflective practice sought to empower them to make their own decisions based on what they believed to be right and the results can be seen in Chapter 4. This empowerment relates to Freire’s policy of conscientization (1970) which allows participants to understand their situation and make their own decisions.

A possible threat to catalytic validity may be that the students’ reflections were superficial and lacked depth.

3.9.5 Threats to validity

the growth of the students as people, the size of the intervention class and the enjoyment of the intervention could have made the students less critical than they should have been.

In addition to Communication Skills, other subjects, lecturers and university experiences could have played a role in the students’ growth and development. These external influences were not taken into account and instead the intervention relied on the students’ emotions. As the students were responding directly to the facilitator’s questions, they could have wished to please the facilitator.

For this study, a sample of fifteen students was selected from the class of 90 ECP Engineering students. All other students who wished to be part of the intervention could not be refused as I felt ethically obliged to help everyone who requested assistance. If I had been able to work with only these fifteen students, I may have achieved more depth in the study.

Students appeared to have enjoyed the experience of the intervention, which could and did become very emotional at times. The remembered involvement during the sessions could have made them concentrate on these emotions, instead of looking at the programme critically.

3.10 RELIABILITY

While validity measures whether an instrument does what it should, reliability checks the consistency and accuracy of these measures. While validity may not be necessary for reliability, reliability is an integral part of validity. This means that a test can be reliable but not valid, but a test cannot be valid without first being reliable. When a test is measured more than once and the outcomes are the same, reliability is high (Salkind, 2006:105-118). Salkind also explains that

what researchers know about human behaviour, ultimately depends on how they measure what they are interested in studying. In other words, the measurement technique used and the questions asked, go hand in hand and are very closely

The meaning may change in quantitative and qualitative research. In quantitative research, measurement is about numbers and objective hard data (Golafshani, 2003:597). The result of a test should be repeatable. In qualitative research, the emphasis is on the researcher who seeks trustworthiness, understanding and extrapolation. When using both quantitative and qualitative methods in a mixed- methods study, Patton (2002:247) asserts that triangulation can strengthen a study by combining methods. However, Barbour (1999:39-43) believes mixing methods within a qualitative or quantitative paradigm can be problematic because each paradigm has its own assumption according to its theoretical framework. This made it necessary, in the present study, to guard against interpretations that could be understood from different perspectives.

In my research, I used parallel forms as a measure of equivalence by administering two different forms of the same test to the same group of participants (Salkind, 2006:110-112). This means that the pre- and post-test for the written work, oral presentations and the interviews used the same format while the questions were different, yet of the same standard. A colleague in the communications field was asked to compare the standard of each test. She found both tests to be of the same standard.

3.11 ETHICAL ISSUES

In order to conduct my studies, I was obliged and ethically bound to obtain permission from the students involved and the Head of Engineering (ECP). A copy of this letter from the Head of ECP (Cape Town campus) is in Appendix 8 on page 190. I asked students for their informed consent from all parties concerned to conduct pre- and post-tests of written work, oral presentations and interviews, as well as to film the presentations and record the interviews. Only students that agreed to this procedure could be allowed to take part in the study. The access to my students formed part of my daily work, apart from the interviews, which I conducted privately in the Communications centre. Full confidentiality and anonymity was guaranteed to each participant. I used numbers to represent each student so that no-one would recognize them as people. Students were also guaranteed the right to withdraw from

the project. Having more students than I needed, I had allowed for an attrition rate. All participants were unconcerned by anonymity and preferred to concentrate on improving their results.

3.12 SUMMARY

In this chapter I stated the research question, outlined the processes of sampling and the research approach. I also identified the instruments I used and explained how the data was analysed in terms of theories and beliefs. I have discussed the validity, reliability and ethics of the process. Although the allocation of marks in oral presentations and written work is always difficult in terms of subjectivity and bias, I attempted to prevent any skewing of data by moderation from a Communications Skills expert.

4.1 INTRODUCTION

I shall present the results of this research project in this chapter and examine the benefits of the intervention on the written work tests, the oral presentations and the interviews. Each section will start with a quantitative discussion of the results of the written work tests and oral presentations according to pre- and post-tests. This will be followed by a qualitative discussion around the emerging benefits of the intervention. The students’ comments regarding the intervention in the post-testing will be regarded as the benefits or limitations of the intervention. Their insights will be tabulated and revealed through the comments taken from their written work, oral presentations and the interviews.