No difference in marks at the test was found between the control group and the participants equipped with an annotation tool (see section “Measures between groups”). But what about meta-cognitive benefits? In order to tackle this dimen- sion and to test Hypothesis 4, five independent experts, specialised in self- regulated learning and reflection-related issues, categorised, in an online card sorting tool (www.websort.net), the 137 participants’ descriptions of learning experience, broken down into 257 units of meaning. The experts freely catego- rised these items by creating as many groups as they liked. Each item ought to be placed in one category only and each category was to contain units that were similar in meaning to each other. The experts returned 50 categories in total. In order to spot convergences and to identify a reasonable and manageable number of clusters, they were submitted to an average linkage cluster analysis algorithm (Börner, Glahn, Stoyanov, Kalz, & Specht, 2010) which performed a
Annotations as reflection amplifiers | 89 clustering based on the average distance between all pairs of objects (one mem- ber of the pair ought to be from a different cluster). Eight clusters emerged from this process (Table 5.4).
In a search for increased parsimony, the eight clusters were related to three components of the Butler and Winne’s model of cognitive system in self- regulated learning (1995): Knowledge and beliefs (Clusters 1, 2, and 3, light grey in Table 5.4), Tactics and strategies (Cluster 4, 5, and 6, medium grey) and Products (Cluster 7 and 8, dark grey).
Table 5.4. Distribution of the learning experience descriptions in the eight
clusters emerging from an expert mapping procedure and relations of these clusters to the Butler and Winne’s model (arrows on the right)
Cluster % of descrip-tions Dominant theme of the learning experi-ence description 1 22% Opinion about course components (struc-
ture, content, navigation, test) with refer- ence to prior learning experience.
2 15% Opinion about the visual illustrations ofthe course. 3 3% Opinion about the difficulty level of the
course.
4 17% Application of personal study strategies in the course
5 13% Application of the strategy “Students set the test”
6 4% Evaluation of own learning activity in the course against the score at the test. 7 19% Expression of satisfaction about the course
completion
8 7% Expression of satisfaction about what has been learnt
A chi-square test revealed that the descriptions stemming from the control group fed very significantly more the category “Knowledge and beliefs” while the students having used the annotation tool tended to provide accounts focused on “Tactics and strategies” and “Products”, χ²(2, N = 137) = 21.712, p < .001.
Discussion
The goal of this study was to ascertain whether frequent and local digital anno- tations used as RAs during the study: (a) could be beneficial to the learning per- formance without extending time on task (Hypothesis 1), (b) would influence the mark on its own or in association with other reflective enactments (Hy- pothesis 2), (c) would yield contrasted results depending upon the use of a free
Knowledge and Beliefs Tactics And Strategies Products
or a structured note-taking technique (Hypothesis 3), and (d) would induce a different narrative tone to the account of the learning experience (Hypothesis 4). The first hypothesis is not confirmed: RAs do expand time on task without de- livering benefit for learning achievement: the control group gets the same mark while using less time. From a strict performance-oriented viewpoint, frequent and local annotations are counter-productive. These results should however be nuanced by the analyses carried out within the treatment group. When applied to the 103 participants making use of the annotation tool, performance-related analyses show a somewhat differentiated picture, as recapped in Table 5.5. This helps contextualising note-taking practice. Again, to interpret Table 5.5, the dif- ference between the absolute amount of annotation and the annotation rate must be kept in mind, along with the difference between learning efficacy and effi- ciency.
Table 5.5. Annotation behaviour matrix (treatment group)
Annotations Learning efficacy (mark at the test) Learning efficiency (speed of learning) Amount A. Positive effect if above average B. Adverse effects
Rate C. Positive effect only in combination with other reflective enactment rates
D. Optimum
Cell A: it is legitimate to encourage online learners not to spare their annotating of the learning material: subjects who made more digital annotations than the average number tended to score better at the test (section “Amount of reflective enactments and mark at the test”).
Cell B: despite the benefits it brings, an above-average quantity of annotations is performed at a price: a lower learning efficiency. It is observed a first time in the comparison with the control group (see section “Indices FinalTest, Time- Spent, LearnEff, and NumberPages”) and confirmed by analysis within the treatment group (see section “Amount of reflective enactments”).
Cell C: the reflection rates provide insights about the way learners balance the primary activity (studying the course) and the secondary reflective activities (annotations, page re-visits, dashboard views). Here, students who write more annotations per unit of time than the average do not get a higher mark. How- ever, combinations of this reflection rate with other reflective enactments (page views, dashboard views) have a significant positive impact on the mark at the test. Students who interlace the first-order learning activity with repeated re-
flective activities perform better than those who practise this crisscrossing at a lower rate. Hypothesis 2 is confirmed: only a compound of reflective activities
can make a difference with regard to performance.
(The qualitative data also seems influenced by combined reflective rates: a sig- nificant effect on student’s sense of control is obtained only from blended re- flective enactments (see section “Control”). On this basis, it can be advanced
Annotations as reflection amplifiers | 91 that the dynamics of reflective commitment to a study task encompasses and interweaves several reflective enactments performed at a certain rhythm). Cell D: raising the annotation rate can serve learning efficiency till a certain point where it starts conflicting with it. The curve in Fig. 5.3 suggests that suboptimal students suffer from a certain reflective passivity that might be counteracted by inviting them to accelerate the frequency of their reflective en- actments on the material. At the other end, highly efficient students make a re- duced use of the annotation tool since they may have developed their own re- flection routines or because at a certain point the type of reflection practised through the annotations gives precedence to other forms of reflection.
As for Hypothesis 3, it is not confirmed: students confronted to a structured an- notation strategy do not outperform their peers who use annotations as they wish. Two explanations can be put forward for this lack of difference. It is pos- sible that free notes and structured notes conveyed onto the learning material the same analytical scrutiny, leading to similar effects. It can also be that the stu- dents did not practise correctly a structured annotation technique they were not familiar with. All things considered, the annotation strategy “Students set the test” was aligned with Hattie’s meta-analyses whose superseding conclusion is that the most powerful cognitive and meta-cognitive effects on learning are in- duced when learners see themselves as their own teachers (2009, p. 238, Fig. II.I). The lack of effect of an exercise entirely oriented in this direction under- lines the difficulty to materialise Hattie’s reversed way of learning and the effort of reflection it entails.
With regard to Hypothesis 4, the cluster analysis brings evidence that a deliber- ate effort to intertwine study practice with structured reflective activities can change the focus of the accounts of learning experience, raising the chance that students take their own learning dynamics as an object of attention (Watkins, 2001) and balance content and process aspects in these descriptions (Ver- poorten, Glahn, Chatti, Westera, & Specht, 2011, p. 279).