Chapter 7 The Pilot Study
7.7 Evaluation of the pilot study including resultant impact on the further
This section evaluates the research instruments used as part of the pilot study and outlines resultant changes. Consideration was given to the quality of data generated including areas of omission and duplication.
MSCEIT
Whilst the test did not yield as high a tutor EC score as anticipated, it has raised some pertinent points for consideration. Chapter 2 outlined the hypothesis, tutors exhibiting high levels of emotional competence are perceived as effective in blended learning environments by their learners, and this was not apparent in this case. High Average scores in Perceiving and Understanding emotions together with Consider Improvement scores in Using Emotions and Managing Emotions were noteworthy. This emotional competence profile should be further investigated to evaluate its contribution to success in blended learning environments and to consider whether a broader mix of emotional skills and competences are associated with tutor effectiveness. When questioned about the MSCEIT, a point was raised about the tutor’s state of mind at the time due to family bereavements, making answers more negative than in normal circumstances. As a
138
result, tutor feedback about the MSCEIT’s completion was added to the interview schedule.
The MSCEIT has, however, allowed useful associations to be made with both the tutor’s feedback and the data from the learner questionnaire and therefore was used in the full empirical research.
Tutor Interview
This yielded a wealth of information which complemented the MSCEIT data and learner questionnaire results. As stated in Chapter 2.5, a qualitative approach would help address some of the MSCEIT’s limitations and provide examples of practice that illustrate tutor EC, and the interview was effective in generating such data. This covered all aspects of factors outlined in the literature review chapters regarding the skills and qualities required for effective blended tutoring. Strong aspects of the interview included the tutor’s background and experience, technical skills, training, and learner support mechanisms.
The main difficulty found with this research method was identifying and classifying emotionally intelligent competences, such as, when considering the following statement:
I meet all my deadlines and if I say I’m going to do something I do it. I find out answers that I don’t know and I always get back to them.
On first reading it appears as just the statement of a conscientious tutor, however, when considered in the context of the interview and the passion in the tutor’s voice at the
139
time, it was quite revealing of a number of emotional competences such as Self-
Awareness (including Self-Confidence), Self-Management (including Trustworthiness
and Conscientiousness), Social Awareness (including Service Orientation) and
Relationship Management (including Influence and Communication) (Goleman, 2001).
During the preliminary phases of analysis, Goleman’s Framework was identifying relevant emotional competences with greater clarity than Mayer and Salovey’s (1997) Four-Branch Model of Emotional Intelligence. To illustrate, the above quote potentially exhibits elements of two branches of the Four Branch Model, namely, the capacity to accurately Perceive emotions and the capacity to Manage Emotions. However, as the interview was after module completion, it took a more reflective nature with limited reference to emotion. This resulted in difficulty in accurately identifying emotional competences in relation to the Four Branch Model.
The tutor interview can be improved to elicit further discussion around emotional competences with further analysis in relation to Goleman’s (2001) Framework of Emotional Competences. Broader EI competences were elicited from the tutor, mainly from follow up questions, therefore some interesting data was revealed. To encourage further tutor discussion of emotion, questions about the tutor’s emotional state were added; were they, for example, calm or stressed throughout the module in online and face-to-face environments. Secondly, a question was added that explored relationship management, conflict, and other issues that arose during the module, including tutor management of these situations. Questions around student and tutor workload allowed consideration of Self-Management but not necessarily from an emotional point of view. The questions revealed an organised tutor, able to plan an effective teaching and
140
learning experience for her and the students. A useful follow up question, after discussing workload, would consider the tutor’s mood and stress management throughout the module and whether this was noticeable to learners or had an impact on the learning experience. This would enable easier associations with MSCEIT scores, particularly the Managing Emotions branch and other emotional competences.
Amendments were made to the template following the pilot and these continued to be made throughout the research. The final version of the template is presented in Appendix 7.
Learner Questionnaire
The learner questionnaire was useful in triangulating a number of aspects of the tutor interview but, due the low number of respondents in the pilot phase, was only analysed with descriptive statistics. The questionnaire was used in its present form for the main study but with amendments following the full analysis of scale reliability and validity (see Chapter 6.4).
Analysis of the VLE Content
This provided a valuable source of information that triangulated the findings of the tutor interview and learner questionnaire. The interview template had limited value in the analysis of the comments as they were largely descriptive with less reference to thoughts, feelings and emotions than the tutor interview. When analysing the VLE content during the full empirical analysis, greater prominence is given to the practice raised in the literature review chapters.
141