Establishing credibility, transferability, dependability, confirmability:

Phases 1 and 3

The social nature of educational research can pose particular challenges in terms of measuring reliability and validity (Pappas and Tucker-Raymond, 2011). For these reasons, it became imperative within my role as researcher, to ensure a high degree of confidence in the fidelity of the data and findings. Guba’s framework (1981) provided a useful construct for these purposes. In particular, attention was given to the criteria of credibility, transferability, dependability and confirmability of the data.

During the interviews, triangulation occurred through member checking, follow-up questions formulated as necessary, creating the opportunity to seek clarification on certain issues, all helping to reduce ambiguity, as well as providing the opportunity to explore key issues in further analytical detail which allowed for development of rich, thick descriptions. In order to ensure participants’ felt that they could provide an honest and full account of the CPD and change processes, it was important to work to ensure a good rapport with the participants and explicit

114 communication on my part that I was not seeking a particular answer to the

interview questions, was an important part of this process.

Bias was avoided as much as possible by recording a full and honest account of the interviews, and working to ensure a good rapport with interviewees so that participants could provide a full and honest account, and that they were not unduly influenced by my position as a CPD provider/leader within the CPD process.

These attempts at providing ‘distance’ between participants, helped to ensure that participants could provide a more unbiased account of the nature and extent of impact following their engagement with the CPD. It is still important to note, however, that validity of findings can only be made as valid as possible, and cannot be ‘proved’ (Gronlund, 1990). After the interviews, interviewees received full interview transcripts to ensure these represented their accounts accurately.

Themes common within transcripts were highlighted and coded, and specific statements from participants have been used directly within the analysis and discussion, to support the authenticity, of the study’s findings. Interview notes and transcripts were prepared in the same standardised way, in an attempt to improve dependability (Silverman, 2011). However, ensuring findings are repeatable at another time, may depend on similar levels of rapport between interviewer and interviewee.

115 3.5.2 Establishing reliability and validity: Phase 2

In the process of completing the questionnaires that are seeking details of the nature and extent of impact, it should be acknowledged that participants may feel that they need to ‘look’ for impacts – and so may over-report implemented

changes to practices and to pupils’ learning. As the CPD workshop occurred, on average, several years prior to completion of the questionnaire, the passage of time may affect the accuracy of a participant’s memory of specific changes.

Questionnaire items were established and adapted from existing peer-reviewed research instruments, including TIMSS Surveys (IEA, 2007) and Braund (2011), helping to ensure credibility and validity of measurement items.

Data have been drawn from larger sized sample than could be achieved through interviews, which helps to support the reliability of the findings. In order to avoid response shift bias (Howard and Dailey, 1979) of a traditional pre and post-test survey, a retrospective pre-test (Lamb and Tschillard, 2005) was used to provide more reliable data, by ensuring that the participants’ understanding of the various items measured were consistent in time. They were required to draw upon their professional judgement to consider the nature of changes to their practices as a result of engaging with the CPD.

The questionnaires provided a number of advantages over other data collection techniques: The questionnaires were relatively easy to administer, a large number

116 of questionnaires were completed, and so a significant amount of data collected. A number of questions elicited data in relation to participants’ motivations, teacher efficacy, and expectations, attitudes, opinions, beliefs, values, and behaviours, which provided considerable opportunity for rich data analysis. The large quantity of data collected was sufficient for a range of descriptive and inferential statistical analytical techniques.

Although the questionnaires did undergo a period of pilot testing to ensure that questions were clear to the reader, and that participants completed the

questionnaire items in the way intended. It may be possible that when used for the actual data collection, some participants experienced a level of fatigue or

experienced a lack of interest when completing the questionnaires. However, in response to feedback from the pilot testing of the questionnaires, I have mitigated against this to some extent by reducing the number of questionnaires items, so that the questionnaire could be completed within approximately 20 minutes.

The questionnaires included a large proportion of closed-ended questions, which may have a lower validity rate than more open-ended question types, as

participants are ‘forced’ to respond to one option or another. Alternatively,

participants may choose not to respond to a questionnaire item at all. Additionally, the participants who choose to respond to a particular question may be different from those who chose not to respond to others, thus creating bias. However, only a small number of ‘non-responses’ were identified within the questionnaire data.

117 Questionnaire answer options may have affected the validity of findings as certain answer options may be interpreted differently by different participants. For

instance, the answer option “partly agree” may represent different meanings to different participants. This is true also of answer options that are framed in terms of the ‘extent’ of the use of particular new resources or practices, for example,

‘occasionally’ may take on different meaning with different participants, depending on the participants’ context.