Despite the commitments given in the introductory workshop, teachers were quite reluctant to participate in online professional dialogue around the three identified areas of action research focus during the first semester. Only 16 of the 24 teachers offered posts and the total number of posts throughout the semester was only 36. Moreover,
most commentaries were brief and not clearly related to the questions posed in the research. Instead they were largely anecdotal accounts of incidents or problems in practice. This participation was far less than anticipated in the overall research design and in the action research model itself. Program leaders speculated that this low engagement partly related to teachers working remotely and not feeling either connection with peers or the collective ambition of the original action research plan.
Ongoing feedback from teachers during the semester also suggested some frustration and/or unfamiliarity with the technology, yet improved instructions and online encouragement to participate failed to improve responses. However, subsequent analysis of the end-of-semester workshop suggested this was instead more a result of unfamiliarity with blogging, a reluctance to publicly speculate on student learning and limited confidence in having a useful perspective to offer. Whether this was
characteristic of this specific teaching group, or a more flawed assumption in the research model is a matter that became more apparent in further semesters (this will be returned to later in the chapter). However, this limited response in the first semester meant only a small amount of useful data was generated that could be meaningfully considered.
Toward the end of semester, students were asked to respond to the series of qualitative evaluative questions formulated in the introductory seminar. The survey was
administered by the researcher and offered to students using the online Survey Monkey tool, with an initial email and two follow up emails all carrying an embedded link to the survey. The survey was completely anonymously. By the conclusion of the survey period, 102 responses to the qualitative questionnaire were submitted across the four subjects of the Graduate Certificate program. This represented an acceptable response rate of around 30% and was broadly similar to the response rates for previous
quantitative surveys for this group. However, unlike previous quantitative evaluations, the open-ended questionnaire generated a considerable amount of data - in excess of 20 000 words of student feedback on their learning. Although responses ranged
significantly from great detail to superficial overview, much of the data was rich in form and usefully related to the action research questions that framed the feedback. By coincidence, the MARA had also decided to conduct their standard cross-university quantitative survey during this same semester. In effect, this meant students were asked to complete both a qualitative and quantitative survey for the same subjects. The
outcomes of this survey, which generated 118 responses to a series of standard quantitative student evaluation questions using a Likert-type scale, were also made available to the researcher. This data provided a useful comparator in later analysis of the outcomes of the qualitative data.
Consistent with the methods described in Chapter Five, the extensive data emerging from the qualitative student survey was thematically coded and analysed, along with the more modest data generated via the teacher blogs. As Glense (2006) observes, thematic analysis is the most widely accepted means of data analysis in the sociological tradition. It allows the researcher to effectively segregate qualitative data into clusters for further description and systematic analysis. To develop the analysis in this and the second case study, thematic coding was emergent in form. For instance, as a result of some difficulties in category coding, a second taxonomy was developed to assist in analysis.
This second layer employed the categories proposed by Cresswell (2005): ordinary themes, unexpected themes, hard-to-classify themesand finally, major and minor themes. In
considering student and teacher responses, the themes were allowed to naturally emerge from the data without preconceived expectations of clusters, though the emergent themes were also broadly framed in the language and context of the program itself. This provided utility for developing a report for the action research team, as well as providing a valuable data set for the broader research intent of considering the developmental potential of student opinion. Based on this data analysis, anEvaluation and Course Development Reportwas produced by the researcher for consideration by the action research team. The full report is included at Appendix Four. In summary, this report sought to illuminate the key thematic outcomes emerging from the data and the broad program and course development issues these outcomes implied. In summary, the outcomes of this analysis of the data was that:
a) a significant majority of students were broadly satisfied with their learning experience in the Graduate Certificate program
b) there were clear indications that as the program and its learning approaches were maturing and that student learning was improving
c) the efforts of teachers to facilitate the program was generally highly regarded and valued, with a large number of students singling out teachers for high-level acclaim
d) most students thought that flexible access to online resources, forums, quizzes and live classrooms was a major positive in the design of the program
e) several elements of the programs were highly regarded as contributing to learning (most notably face-to-face intensives, discussion forums, assignments and quizzes)
f) there were widespread reservations about the value, credibility and relevance of the mandatory MARA exams as a form of assessment (which was clearly shared by program teachers), that was seen seriously inhibiting the ability of the
program to broaden and innovate in the learning approaches it could adopt
g) the onerous time limitations on subjects was a source of considerable student and teacher anxiety and frustration, especially around the ability of students to absorb and reflect to the level required for both assessment and later practice
h) there was also considerable anxiety over the reliability and accessibility of the multiple technologies being used by the program
i) there was some apparent tension between lawyers and non-lawyers in the student cohort, particularly around inequitable levels of participation in discussion questions (from those with legal training) and unrealistic entry-level legal knowledge expectations (from non-lawyers).
Consistent with the design of the study outlined in Chapter Five, the report also offered stimulus questions that could be usefully considered in order to potentially develop the program and student learning. These questions, broadly developed around the issues emerging from student responses, are reported in Table 6.4.
Table 6.4: Course Development Issues
a) How can forms of assessment (and the exams specifically) more reliably and validly assess the knowledge, skills and capabilities that are taught in the program and required for practice as a Migration Agent?
b) How can the limited teaching periods be further enhanced to allow students to sense they are sufficiently prepared for assessment and later practice?
c) How can the online learning technologies used in the subjects be more effectively harnessed to enhance the student learning experience?
d) Can we create a greater sense of a community of practice between students within the subjects as a means of allowing greater self-direction, more equitable online participation and peer support?
e) Are there strategies to engender clearer student expectations and related teacher-student protocols that would increase student certainty around subjects and the program more generally?
f) What changes may create the foundation for an even more positive learning environment for students to enhance their overall experience in the qualification?
Consistent with the agreed action research model, a two-day, post-program workshop was convened immediately at the end of the semester. This workshop included 21 of the 24 teachers engaged in the action research. The key focus of the workshop was the data and conclusions drawn in this firstEvaluation and Course Development Report. In introducing the data to the workshop, the researcher encouraged participants to critically reflect on both their personal and shared experiences during the semester, and to
identify potential program development options from this debate. Further, consistent with the underpinning CHAT foundations of the action research model, participants were encouraged to consider the complex, and at times contradictory, expectations of pedagogical practice the report raised. As a result, a novel focus of this collaborative dialogue was the discussion of the tensions emerging in the feedback outcomes and
what would be classed in CHAT terms as their expansive learning potential. The broad primary tensions that were identified and debated included the tension between:
pedagogical expectations of professional self direction in learning andthe pragmatic student drive for expedient completion of specified learning and assessment
activities
exploratory engagement in professional practice discourses andthe largely rigid demands of required professional–vocational competencies at the completion of the program
the rich and collaborative learning engagement possible via simulated learning technologies andthe individualised nature of study which used inter-subjective professional contexts of judgment
differing formal and informal pedagogical approaches designed by teachers and educational designers in the program, from simulation in a virtual environment, to scaffolded building of professional capability to assessment of student responses against ‘real’ interpretations of professional practice.
This formed the foundation for workshop dialogue, which also explored a range of course development options (and possible enabling pedagogies) for enhancing student learning. Further details of how these tensions were understood by participants - and the range of responses developed in response - are outlined in detail in Table 6.5.
138
Table 6.5: Issues, tensions and potential options identified in evaluation process
Identified Issues
(derived from teacher/student data)
Primary Tensions
(identified by the researcher)
Potential Course Development Options (debated in post-program workshop)
Considerable student frustration around the dissonance between learning experiences and summative exam based assessment: how can forms of assessment (and the exams specifically) more reliably and validly assess the knowledge, skills and capabilities that are taught in the program and required for practice?
Breadth of student engagement in learning design/explicit practice focus
versus
professional accreditation demands/assessment reliability- validity across cohorts
General Assessment: Increased number of practice-based assessment activities, assessment progressively timed during subjects, assessment of contributions to discussion or client management, increased use of ‘informal’ or formative assessment. Exams:More scaffolding around likely questions, issuing of non- assessable practice exams, access to previous exams, generation of a more positive climate around the exam context, design of student intercommunication online space around assessment to facilitate peer support.
Significant student workload in teaching periods
inhibiting required levels of preparation and engagement: how can the limited teaching periods be further enhanced to allow students to sense they are sufficiently prepared for assessment and later practice?
Intensive-blended teaching model assumes strong learner self direction and engagement
versus
Students part time combining demanding work and study, often adopting a necessarily pragmatic approach
Earlier release of learning materials/activities to allow early start, inclusion of podcasts on key issues that can be downloaded to portable media devices for more flexible engagement, content review to ensure alignment of learning materials/ activities with both needs of practice and assessment, reshaping student expectations of commitment in blended learning program, introduction of re-occurring cases throughout subjects to increase research efficiency, teacher professional development to further improve the effectiveness of teaching, communication and assessment practices.
Student disorientation in navigating online program site and methods of using the site effectively: how can the online learning technologies used in the subjects be more effectively harnessed to enhance the student learning experience?
Imperative to create a rich and engaging online site that allows the use of multiple technologies and high levels of self-direction
versus
Limited student exposure to both the online learning platform and use of Web 2.0 technologies, low tolerance for ambiguity-disorientation
Creation of an online ‘road map’ for students that includes key guides on technologies and the expectations in subjects of their use, some improved consistency across the subjects around expectations of students online and these communicated consistently, creation of frequently asked questions site for students online, simplification of the strategies for use of online blog tool, establishing email alerts to students of additions and changes across subjects, further professional development for teachers on the effective pedagogical use of learning technologies.
139 Identified Issues
(derived from teacher/student data)
Primary Tensions
(identified by the researcher)
Potential Course Development Options (debated in post-program workshop)
Student concern about inequitable workload and different levels of pre-existing expertise being offered in collaborative work: how can we create a greater sense of a community of practice between students within the subjects as a means of allowing greater self direction, more equitable online participation and peer support?
The rich affordances of online technologies to allow ongoing peer collaboration and sharing of perceptions and practice across differing domains of practice
versus
The individualistic nature of online engagement and subsequent assessment, the personal connection with local professional contexts and related expectations
Establish special interest spaces online for students with different needs (i.e. para-professionals, students currently in professional environments, overseas/remote students etc.), introduce/increase assessment around online contributions, create scaffolding resources online for students who sense a deficit in particular aspects of their knowledge or skills, more systematic introduction of online environment in face-to-face intensives, additional professional development for program teachers in facilitating and sustaining online engagement.
Are there strategies to engender clearer student expectations (and related teacher-student protocols) and greater levels of flexibility whilst ensuring students retain a sense of direction in their engagement: how do we increase student certainty and satisfaction around the program?
Imperative to improve student sense of navigating the program, enhance the utility/scaffolding of its flexible dimensions/transparency of assessment versus
Limitations in teacher capabilities (both physical and technical), maintenance of the pedagogical paradigm of self-direction and restrictive accreditation standards that curtail levels of possible transparency
Development of a more defined framework of expectations for students in orientation, introduction of an online road map, establishing a range of reasonable response times for student enquiries and assessment across the program, introduction of more standards forms of feedbacks via program wide templates, move toward assessment rubrics for non-exam assessment, strategies to increase transparency in approaches to assessment, open access to learning resources, enhanced scaffolding where students need further support, more flexible learning resources via podcast and other web based technologies, advocacy of changes around exam based assessment.
As a result of the collaborative academic dialogue, in the final session of the workshop formalised a series of specific response strategies (in the form of a Course Development Plan) for implementation in the following semester. These would also frame the further deliberations of the action research teams in the following semester. In addition, the Course Development Plan(included as Appendix Five) anticipated a series of related professional development initiatives to support these enhancements. It also envisaged longer-term educational design projects that could be productive to further improve program effectiveness. The primary course development elements of the Plan included:
a) encouraging stronger and earlier student engagement with learning materials
b) enhanced sharing of online discussion stimulus activities
c) establish clearer expectations around teacher and student responsibilities
d) more active forms of collaborative engagement with student blogging responses
e) building a key point of assessable continuity throughout the subjects of the course
f) ensuring online modules are available well in advance of face-to-face sessions with students to improve integration of the subject components
g) creating a stronger online student community across all sites
Specific academic development initiatives were planned around the improved use of online classroom technologies, design of practical cases and more effective facilitation of blogging.