Pathfinder evaluation

The Evaluation strategy of the Pathfinder Project looked at three different perceptions of embedding ePDP:

The tutor

The student experience The mentoring relationship

There were two separate reports, one internal and one external, although both reports addressed the following questions:

1. What are the facilitating and inhibiting factors of implementing and embedding ePDP into level 1 modules?

2. What scaffolded activities do staff put in place to support PDP via an ePortfolio system, why, and how are the outcomes evidenced? 3. How far does the mentoring model, developed through retreats and

a focused immersion model, represent a (cost) effective strategy for staff development for the university?

4. What are student perceptions of ePDP? What are the characteristics of student ePDP experiences?

5. What impact has there been on the academic practice of those staff involved in the project?

The external contribution to the evaluation of the Project was based on a round of interviews with Associate Deans for Learning and Teaching focussed primarily upon the third evaluation question; how far the retreat and mentoring model developed by the Project through retreats might represent a (cost) effective

strategy for staff development for the University in this (and other) areas. Such a focus also sought to illuminate issues of more general concern in the institution in relation to moving e-practice beyond pockets of expertise into more widespread use.

The following data was collected and used in both reports:

606 student questionnaires.

81 staff questionnaires (consisting of interim, end of project and retreat specific questionnaires)

3 hours of video of staff feedback presentations. 2 hours audio of staff final interviews.

External evaluation of telephone interviews with Associate Deans within the project schools

I have had access to all data bar the telephone conversations, for my doctorate and have used relevant data to support the SSM process. The majority of the data has fed into Analysis 2 and 3 in the finding out phase, which has been organic and evolving throughout my project. The data has had impact on Chapter 5 Project findings and analysis: 138 and Chapter 6 Conclusions and recommendations: 192. A key success criteria for the project was the wider and more capable use of an ePortfolio system, specifically the software PebblePad©. To this effect I designed all the retreat activities using the software in the way I had observed staff using the tool with their students. This included creating ‘scaffolded templates’ used in the taught curriculum for students to download, personalise and own. The templates would be used throughout the module for both formative and summative assessment. For the Pathfinder Project, a collaborative web folio template (Figure 27: 131) was created. It required participants to offer reflections on mentoring in their own module teaching contexts. Permission to see these reflections was given to individual academic school groups, members of ILE and the external evaluator.

The scaffolded collaborative web folio has acted like a chart matrix (Megginson et al 2006) to ask questions about and compare the activities and their connections from the concept model in the real world environment.

Figure 27 Screen grab: school collaborative web folio template

The questions asked are open to individual and group reflections which are then shared with other participants in the project. For example, (Figure 28: 131), this reflection was shared by a discipline-based mentor.

Figure 28 Screen grab: staff reflection

By offering materials, including the collaborative web folio, through a Gateway, (a term used in the PebblePad© software), I wanted to recreate the student experience for staff (Figure 29: 132). I had observed that students would often be given instructions to access material from a Gateway, personalise it and resubmit is for both peer and tutor assessment and comment but that sometimes these actions had never been done by the tutor themselves.

Figure 29 Screen grab: Pathfinder resources gateway

I had also hoped that using on-line tools for collaboration would help with the mentoring process. In particular I hoped that it would support the mentees doing things for themselves, helping the mentee to develop their own wisdom, for them to steer their own personal development and to aid questioning and social exchange that would emphasize learning. I envisaged the on-line ‘tools' as supporting face-to-face interactions via the retreats as well as between the retreats when both the mentors and mentees were back working in their own discipline-based cultures. In addition, the on-line ‘tools’ could allow more fluidity between all participants rather than keeping to strict discipline–based cohorts. At the first retreat participants felt that they needed to work with people who understood their subject cultures. However, that quickly changed with the realisation that subject knowledge was of less importance than the pedagogic approaches used and the knowledge of how to get the software to do what you wanted. During the first retreat one mentor had to drop out of the project leaving their mentees with no discipline mentor. A mentor from an unrelated discipline stepped in to provide support which demonstrated to the rest of the group that shared discipline knowledge was not important. This supports the concept that in developmental mentoring the mentor needs to be more experienced in issues relevant to the mentees’ learning needs.

Rather than take an approach that taught software skills I wanted to put members of staff in a situation where they could experience what they were asking their students to do and to note the instructions that they needed to be given to fulfil a task effectively. For many participants this was not something

they had done before as in some schools ‘education technologists’ operated the software so the knowledge of its use stayed with them not the tutors. Simple omissions in a process could mean that a student could not submit their work online or that by setting up a ‘Gateway’ in a particular way it would automatically close by a given date. If tutors knew this then they could be in far more control of the learning and teaching that they were asking their students to engage with. Most members of staff had never posted a response to a ‘Gateway’ (Figure 30: 133). Their experience was likely to be that they had set up the activity but not necessarily followed it through as though they were a student. Getting them to do this within the Pathfinder Project raised valuable issues for them such as how they titled their modules and organised their groups.

Figure 30 Screen grab: assets shared in a gateway

The use of the software to mimic the student experience was vital as we were able to notice key issues that I had previously observed in the classroom. (see section 4.1.2.2 Critical incidents: 106).The ILE Pathfinder team along with the mentors were able to develop software functionality for tutors within their own context. We were able to use the same language and processes that were used with students but hopefully demonstrate effective use of PebblePad© as well as enhancing the capability of participants. It also stopped the staff development sessions become software training or ‘button pushing’ exercises.

4.4 Defining action for improvement

As a result of the Pathfinder Project I had been able to test, discuss and debate whether a concept of development mentoring would motivate staff to engage in PDP. The results of those exchanges raised key issues for defining the action for improvement. Based on the evaluation data gathered within the project and from discussions with all the participants during the last retreat I had modelled two pictures, Figure 31: 135 and Figure 32: 136. I visualized what I perceived staff felt were the positive and negative issues related to cultural desirability and cultural feasibility. Checkland and Poulter (2006) discuss cultural feasibility before cultural desirability, yet I found within my own work situation that discussions turned those two areas around; desirability first, then feasibility. I believe this was caused by the structure of the institution in which departments are cascade-managed by a Dean, supported by two–four Associate Deans who are in turn supported by divisional heads, then subject Co-ordinators. This means that most staff who are involved in delivering student learning, at the ‘chalk-face’ have little or no responsibility or control over resource allocation. They therefore make do with the best that any situation can offer them.