Research Methods

Design-based research can incorporate both qualitative and quantitative methods, and the credibility of the research can be increased by utilizing a mixture of methods. This flexible

methodology may also require changes in data collection methods as the research progresses, and “methods vary during different phases as new needs and issues emerge and the focus of the

research evolves” (Wang & Hannafin, 2005, p.8). Data for this present research was gathered from a variety of sources to strengthen the validity of the findings, and several data collection methods were employed. These methods and the data collected are detailed in the following sections.

3.5.1 Interviews

Interviews took place in each meso-cycle of the research. A semi-structured interview format was chosen to allow the flexibility for participants to provide the data to create the rich description which would better inform the research. Seidman (as cited in Dilley, 2004) explained that “interviewing allows us to put behavior in context and provides access to understanding their [participants’] action” (p.129). It is the context provided by the interview data which decreases the likelihood of misinterpreting the actions observed in other data. While the data available in the online environment provided a record of the participants’ actions online, the likelihood of its accurate interpretation was increased by the information gathered in interviews.

In Phase One individual interviews were held with the centre manager and with three of the four tutors with special responsibilities; the lead tutors for the At-Base and the Community groups and the Resource tutor. The Workplace lead tutor was unable to attend an individual interview, but joined one of the five group interviews which were conducted. Fourteen other participants took part in group interviews focussed on the work teams that they belonged to: At-Base, Workplace,

Community or Resource Development (called Writers). Interviews with participants, averaging 30 minutes in length, were conducted at the ALEC facilities between 4 and 18 October 2010, early in term three of the teaching year. One participant had retired from teaching during this phase of the research, and was not interviewed.

The interviews were video recorded to help distinguish between speakers. After transcription, recordings were converted to audio files for storage and the original video recordings were deleted. Copies of interview transcripts were made available to the participants by email for member

checking, to allow participants to acknowledge the accuracy of their transcript. The NVivo qualitative analysis software package (QSR International Pty Ltd) was used to store and organise the interview transcripts.

The interviews and the collection of data from the site were all initially planned to take place within a single week late in September 2010. The researcher felt that the interviews might influence the participants’ behaviour within the site in a way similar to the Hawthorne effect, which Brown (1992) states “refers to the fact that any intervention tends to have positive effects merely because of the attention of the experimental team on the subject’s welfare” (p.163). However, due to the

disruption caused by a major earthquake, the interviews instead took place over a two week period in October. During the intervening weeks participants’ online behaviour changed because of

also spoke about the Resource professional development day and associated discussions so often in their interviews, a decision was made to include data from October as part of this first phase. In the first meso-cycle of Phase Two interviews were held with all remaining participants. Individual interviews were conducted with the centre manager, the office administrator and with the At-Base, Community and Workplace lead tutors. Organising the intended group interviews with the eight other participants became very problematic and so interviews were conducted with three groups of two tutors, and two further individual interviews. Site data was used to group the participants according to the amount and type of activity they had engaged in within the online environment. Every attempt was made to interview participants with similar levels of involvement, to gather information on the individual contexts of participants who were exhibiting similar online behaviours. However this was not always possible because of participant availability. The participant interview groupings were:

Group A: Two tutors who had high average site access and had participated actively, one Community and one Workplace

Group B: Two Community tutors with low average site access and no active participation Group C: Two At-Base tutors

Individual interviews with two further Community tutors who were not able to attend group interviews, both had high average site access one actively participated, one did not actively participate

These interviews, averaging 30 minutes in length, were conducted between 19 July and 10 August 2011, over the transition from term two to term three of the teaching year. The Resource tutor and two other tutors were unable to attend any interviews. A further three participants were not interviewed; one Workplace tutor had resigned during this time, and the disestablishment of the Resource Development Group had ended the employment of the two tutors who worked solely in that team.

The interview schedule was reviewed by fellow doctoral candidates and improved through their feedback. The interviews were video recorded to help in identifying the speakers. After transcription the recordings were converted to audio files for storage and the originals were deleted. Transcripts of the interviews were made available privately to the participants within the Research Space of the online environment, and a tool for feedback was put in place to allow participants to acknowledge the accuracy of their transcript online.

To increase the opportunities for the interviews to capture each participant’s personal perspective and context, all interviews conducted as the research moved into the online classroom focussed phases were individual semi-structured interviews. Interviews were conducted with the three At- Base tutors who had developed online classrooms for their clients, at the end of Phase Two meso- cycle two on 16th November 2011. This interview gathered information on their progress, their view of the development of an online classroom for their clients, and an indication of ways in which they had felt supported in the first six months of this online aspect of e-learning in their teaching practice.

A final set of interviews was conducted just prior to the researcher withdrawing from the field of study. This second set of individual semi-structured interviews conducted with the three At-Base tutors gathered information on their experiences of developments over the second six months of using an online classroom for e-learning, as well as their perspective on teaching within this shared online community environment. Interviews were conducted with three other participants, two community tutors and one workplace tutor, who had begun development of online classrooms but had not completed them, nor given their students access to them during this evaluation meso-cycle. The ALEC manager and the office administrator were also interviewed to gather information on their unique perspectives on the developments within the organisation. These interviews were conducted on 16 May and 23 May 2012.The transcripts of the interviews were emailed to the participants to be member checked for accuracy.

3.5.2 Data from the online environment

The online environment was created using a Moodle learning management system. This provided the researcher with access to several sets of data. An overview of the data from the online

environment which informed the research can be seen in Table 2. The interface provided three main sources of data: screen shots taken of the developed environment, participant posts within the forums, and information provided by the participants within site tools such as ‘feedback’ and ‘choice’ questions. Behind the interface, information on participants’ activities was available in the form of site data logs. These logs recorded information whenever a participant accessed an area of the site, or a resource or activity within an area. Reports of logs were generated in several formats, providing the ability to view data on total participant logins to the site over the length of the research; to record participant activity which took place in the different areas of the site; and to follow each participant’s activities within the site. The data of individuals’ activity was also used to group the participants, according to the amount and type of use they made of the Moodle site, for group interviews at the end of Phase Two meso-cycle two.

Table 2: Overview of data gathered from the online environment which informed the research. (The shaded areas indicated the data which was collected.)

Phase One 6/10 – 11/10 Phase Two.1 11/10- 6/11 Phase Two.2 5/11 – 11/11 Phase Three 11/11 – 7/12 Site login records

“Tutors Only” logs Forum activity Forum posts

Individual participant logs Research Space responses Demo classroom logs Online classroom logs

Statistics and logs from the Moodle site were downloaded as spreadsheets and processed using SAP® Crystal Reports software to aggregate the information into weekly data sets, which made the volume of data more manageable. This data was then exported into Microsoft Excel® spreadsheets. Using the spreadsheet tools, graphs were created which displayed the information in a variety of ways to highlight particular findings. The tool Social Networking Applied Pedagogical Principles (SNAPP) was used to gather data from the participant activity within the forums in the site. Bakharia, Heathcote, and Dawson (2009) explain that the resulting “social network visualisations provide a snapshot of who is communicating with whom and to what level” (p.1). This data was extracted from SNAPP, pseudonyms were substituted, and the data was re-displayed graphically using NetDraw (Borgatti, 2002) to show the participation and relations within each forum.

Data was gathered from the ‘Research Space’; an area added to the online environment in February 2011 as both a means for the researcher to communicate with participants and a space to share the outputs of the research. The researcher explained

I have added a research space in the ‘Working Groups’ category. I’m using the ‘groups’ functionality to allow tutors to see the transcripts of just their interviews, and I’ve added a forum for them to give me feedback. I’m going to display the outputs of my research here so that they will have access to them, and I can use this space to show the tutors some of the functionality that will be available to them within their own teaching spaces e.g. Groups, choice activities, and displaying web content like Prezi. (Design narrative, 8/2/11) While a forum was made available to all participants within the Research Space to allow them to provide feedback and comments to the researcher, the only post made in this forum was a general comment on the use of e-learning which was made at the end of Phase Three. Three other

opportunities for participants to contribute information were provided within the Research Space. Two single question surveys six months apart, using the Moodle ‘Choice’ tool, gauged tutors’

perceived level of adoption of the online environment. A final survey was conducted in the Research Space using a Moodle ‘Feedback’ tool. The survey enabled participants to give their feedback on the idea of using a Moodle classroom for e-learning for their clients. This survey was completed by all but one of the tutors who had participated in any phase of the research.

3.5.3 Design narrative

The design of the online environment and the story of its development were an integral part of the research. The design narrative, written in a diary form by the researcher in her role as designer, tells the story of the intervention, its design and implementation within the ALEC context. “Design-based researchers must not only document their perspective or starting point, but must also document any plausibly relevant interventional strategies used not only by participants observed, but also by the researcher herself or himself” (Hoadley, 2004, p.205). This design narrative recorded the events and circumstances surrounding changes, both successes and failures, helping to distinguish intentional changes from accidental ones. It was most informative in revealing the smaller cycles of design, enactment and evaluation which took place within the design and development of the site during the research. Throughout the research phases the design narrative continued to provide contextual

information on the development of the intervention and the designer/developer’s interactions with the organisation. This included the resources provided for online classroom development as well as the individual tutor’s development of their online classroom. The design narrative also helped to “meet the challenge of replicability by adequately describing research contexts” (Bell, et al., 2004, p.79). By providing a unique designer perspective on events, the narrative helped to determine whether the data generated by other methods was adequate in capturing the results of the intervention implementation.

In recognising that there are some limitations related to the multiple roles this researcher plays as the sole intervention designer, developer, technical support person, and researcher, the design narrative helped to clarify the designer/developer view as distinct from the other aspects of this researcher’s roles. As Bell, Hoadley and Linn (2004) suggested, a design narrative “can help make explicit some of the implicit knowledge the designer or research partnerships used to understand and implement the intervention” (p.79). Included in the design narrative were the ‘why’ as well as the ‘what’, information which may support generalisation and an understanding of whether similar decisions would be appropriate in other contexts. The addition of this design narrative enhanced the understanding of the research context, increasing the opportunity for others to consider how the results of this study might inform their own context, and the possibility of replicable results in a very similar context.