Data analysis

Data analysis was undertaken on the learner reflective Blogs and tutor observations critiqued in section 4.4.1 and 4.4.2 in Chapter 4. In study 1 a

Cohen Kappa inter-reliability test (Fleiss, 1981) was undertaken on the positive and negative comments derived from the learner reflective Blogs. This was intended to consider objectivity in the data analysis. The inter-rater reliability technique rates show how different coders produce the same results when the same body of material is examined (Silverman, 2000). This was not taken forward into study 2 as this was deemed unsuitable given the second study focus was to gain an understanding of the perception of learner experience to a subjective and interpretive stance appropriate to answering the research questions as critiqued in section 4.2 in Chapter 4. It was not intended to persuade others of the objectivity of this study through science, logic and evidence (Golden-Biddle and Locke, 1997). To increase the worthiness of the research Stake, (2000) suggests using multiple perspectives to triangulate data from different sources. Triangulation is critiqued in section 4.3.2 in Chapter 4.

This section describes the process of analysis of the learner reflective Blogs; this is followed by a section presenting the analysis of the tutor observations.

The sixteen groups of learners comprising six members each were required to complete learner reflective Blogs as an assessed task. The Blogs were analysed and coded based on specific topics raised in the Blogs and in their open nature.

Content analysis as described in section 4.4.3 in Chapter 4 was undertaken on the reflective Blogs. The unit for analysis in this study was a phrase that represented a student quote and documented in their own words, for

example “to have connections”. The categories derived from the data are P- O (People-Oriented) and T-O (Task-Oriented). A sample of People-Oriented and Task-Oriented comments is shown in Table 5.1 and Table 5.2 respectively.

Student No.

People Oriented

1 all working together 1 we share information 2 All members try 100% 2 to get to know each other 3 Interaction between the group 3 the ability of work together

3 each member of the group to be responsible, reliable 3 great way of forming a team

3 discussing ideas

3 you can trust each other and rely on that person 3 may form strong friendships

4 everyone had different skills that they brought 4 group members respective of each other‘s needs 4 they still made an effort to work around any obstacles

4 We all have jobs to do and rules and regulations to adhere to 5 when all members of the group participate

5 by putting forward their ideas

5 contributing by answering questions that have been asked 5 Individually do their own research on a topic

5 knowing what is being said in the group 5 turning up for meetings

5 knowing who the group members are

Table 5.1: List of extracts from reflective Blogs show people-oriented learners

Student

No. Task- Oriented

1 to work together on tasks 1 to fulfil requirements 2 to complete tasks 2 to review all the tasks 2 to complete the set tasks

2 to make sure we all turned up to meeting 2 to make sure we understood what took place 3 to work together on assignment

3 to discuss and analyse

3 to produce relevant issues as well as results 3 to communicate well and efficiently about tasks 3 to contribute various ideas and opinions on tasks

3 it generates ideas as a group

3 to communicate with other members on the assignment 4 to get together to complete a task

Table 5.2: List of extracts from reflective Blogs show task-oriented learners

To measure learner attitude in the reflective Blogs, a sample of two categories were chosen P (a positive response), N (a Negative response). A sample of positive and negative comments is shown in Table 5.3 and Table 5.4 respectively.

Student

No. Positive Comments

1 work on tasks at any time

1 within the comforts of their own homes 1 find it easier to work remotely as they are shy

1 group could continue with the work if they were not in a meeting 2 each member could add and remove the content in their own time

2 face to face could lead to members going off topic compared to online is very unlikely to happen

2 it reduces travel time and expenses

2 reduces the amount of information a third less words per unit

3 times where certain individuals would not be able to meet up online enables the whole group to interact with each other

3 allows fellow group members to share information and ideas

3 allows the individual to analyse and study all data that has been put up 3 data that has been put up clearly and easily unlike face to face where may

not understand a fellow members handwriting

3 allows the group to understand all the information issues and all topics discussed

3 clear and easy to follow information on screen

3 access from almost anywhere, using mobile phones, laptops 4 great for group members to come together

4 communicate and collaborate by posting documents and tasks

Table 5.3: List of extracts from reflective Blogs show positive learner comments

Student

No. Negative Comments

1 other members misinterpret what has been written

1 members can get annoyed if they rely on someone for work 1 ideas can be lost and replies can be days later

2 not being able to see the other person

2 material may look different in different browsers

2 more than one person could reply to a topic which lead to confusion 2 you could be the only one online so help is limited

2

parallel

3 certain members of group may "sit back" and rely on fellow members 3 extra work load causing stress and tension amongst group

4 we all prefer to meet face to face that way you get a feel for what the person is like

4 a lack of physical communication

5 can lead to misunderstanding within the group

5 when arrangements are made not everyone will be available 5 all group members will not have access to the internet

5 if you do not save regular as you are doing your work it does not save and shows an error message meaning we had to start over again

6 technology can easily breakdown therefore it will be difficult to communicate ideas and so slows everything down

11 there can be misunderstandings where the plain text we see online is taken "out-of-context"

12 lack of true response, facial expression

13 no visual audio feedback people may take things the wrong way

Table 5.4: List of extracts from reflective Blogs show negative learner comments

Student attitude was rated and given a number between one and ten, using a Likert type scale where one represented poor, five average and ten excellent. This measure was based on my past experience (Doolan, 2004; Doolan and Barker, 2005). This data analysis was carried out by my own hand in order to get a feel for the data and to explore the true meaning as

discussed in section 4.4.3. The category codes were written on the student text and then the categories and derived quotes from the student text were typed into Excel. A sample of attitudinal measures is shown in Table 5.5. Table 5.5 shows the group number, the student anonymity number, the number of positive and negative comments derived from the analysis of the learner‘s reflective Blog. This was indicated on the text within the Blog by using a P which represented a positive comment, conversely N represented a negative comment.

Group No Anon No Total Positive Total Negative

1 S1 11 9 1 S2 21 5 1 S3 13 11 1 S4 15 2 1 S5 16 9 1 S6 16 9

Table 5.5: Learner attitude ratings derived from reflective Blogs

5.4.3.1 Inter-rater reliability

To check for researcher bias in selecting positive and negative comments an inter-rater reliability test Cohen Kappa (Fleiss, 1980) was performed. It was intended to provide a level of confidence in the data. The two categories P and N were chosen as the rating of student attitude was dependent on these,

hence it was important to ensure that the data derived from the content analysis was reliable. The process was as follows:

Using the Microsoft Excel spread sheet application (Microsoft Corporation, 2010) every positive comment was given a number between one and four and sorted by number, a total of twenty-three positive comments were selected (all number four). Every negative comment was given a number from one to four and sorted by number, a total of thirteen negative comments were selected (all number four). These thirty-six student quotes were merged and mixed in a file and sent for independent analysis. The independent researcher was asked to tick the relevant column P for Positive, N for Negative and DK for don‘t know.

In order to address inter coder and intra coder reliability, the above mentioned strategy was followed individually and then carried out by an independent researcher. The results in the form of an agreement table are shown in Table 5.6.

Values highlighted in bold in the table show agreement between the two individual researchers. The table shows how the researchers agreed on positive and negative comments elicited from the students work.

R es ea rche r 2 Researcher 1 Pos Neg DK Pos 20 1 0 Neg 0 10 0 DK 3 3 1

Table 5.6: Agreement matrix

The reliability of these categorisations is tested through Cohen‘s Kappa (k) statistic where agreement between the values of two raters (rating the same student quotes) is measured. The result is shown Table 5.7

Value Asymp. Std. Errora Approx. Tb Approx. Sig.

Measure of Agreement Kappa N of Valid Cases

.746 36

.094 5.863 .000

a. Not assuming the null hypothesis

b. using the asymptotic standard error assuming the null hypothesis

Table 5.7: Cohen's Kappa measure of agreement

As Table 5.7 illustrates, the Measure of Agreement k is above .7 which shows a reliable agreement (Vogt, 1999 in Beecham, 2003) thus the value in the table .746 shows that the problem classification is reliable in this study. The significant value < 0.01 shows that this did not occur by chance alone.

The full test is provided in 7.9Appendix D. The next section presents the data analysis undertaken on the tutor observations and reflections.