Provision of Prompt Feedback

In the monitoring system, feedback can be submitted on a weekly basis by students, monitors, and supervisors. It consists of the student’s (i) overall project progress, (ii) motivation, (iii) report quality, (iv) quantitative report metrics (pages/words), (v) event attendance (in monitoring/supervision meetings). Furthermore, textual feedback can be provided once a week together with these ratings. Weekly feedback was preceded by meetings, that is, monitors, students, and supervisors normally provided it in response to face-to-face interaction. Some monitors also reported that students who could not be present in meetings sent them status updates via email, in which case weekly feedback was provided based on these emails. Features supporting the submission of feedback were online progress and event feedback forms, while features visualising feedback were (i) charts and graphs, (ii) the ranking table, and (iii) the news feed. Email notifications were used for both purposes: they were sent to users in order to remind them to submit weekly feedback and in response to feedback submitted on the system (news feed emails). Closing questionnaire responses (see section 6.1.4.2 on page 158) suggest that the ma- jority of student respondents found both monitor-rated and supervisor-rated progress feedback very helpful for managing their MSc project (29.2% and 37.1%, respectively), though supervisor-rated feedback was perceived more helpful than that of monitors. Sim- ilarly, event attendance feedback, the ranking table, the news feed, and email notifications were perceived helpful by most respondents (31.7%, 27.6%, and 29.4%, respectively). In contrast, most students (27.1%) were undecided regarding the helpfulness and motiva- tional effect of charts visualising progress feedback and the motivational effect of the news

feed. The same applies to general statements (see section6.1.4.4 on page 159): the ma- jority of respondents did not find that the system helped them to communicate problems or keep in contact with their supervisor/monitor during periods of absence. However, 50% of them agreed or fully agreed that the system raised their progress awareness. When it comes to correlations with student performance, progress, and motivation rat- ings, the following results were obtained (see Table6.11 on page 168): (i) the submission of event attendance feedback correlates positively with students’ weekly progress ratings and monitor-rated motivation, (ii) event reminder emails received by students correlate positively with monitor-rated student progress and task state reminder emails correlate negatively with student-rated progress as well as motivation, (iii) there is a correlation between the student use of charts and student performance as well as weekly student pro- gress and motivation ratings, (iv) student use of the news feed correlates positively with monitor and supervisor-rated weekly student progress and motivation, and (v) views of the home page, which contained the ranking table, are positively correlated with student performance, progress, and motivation.

There also exist correlations between subjective student feedback regarding general state- ments in the closing questionnaire, feature use variables, and student performance, pro- gress, as well as motivation variables:

1. The use of the news feed correlates positively (p < 0.05) with perceived helpful- ness of monitoring meetings (rs= 0.201) and agreement with “monitor/supervisor

picked me up on progress reports” (r_s= 0.206), “the system helped to communicate problems” (rs = 0.234), “the system helped to keep in contact with faculty staff”

(rs= 0.232), “the system helped to keep myself organised” (rs = 0.210), and “the

system enhanced MSc project management” (r_s = 0.222).

2. Student views of the home page (containing the ranking table) correlate positively with perceived helpfulness of supervisor-reported progress for project management (rs= 0.231, p < 0.05).

3. Student use of the news feed correlates positively with perceived helpfulness for project management of features event/meeting organisation (r_s= 0.208), progress reports from monitors (rs = 0.204, p < 0.05) and supervisors (rs = 0.211, p <

0.05), the ranking table (rs= 0.348, p < 0.01), and charts (rs= 0.239, p < 0.05).

4. Student use of charts and graphs correlates positively with perceived helpfulness of supervisor progress feedback for project management (r_s = 0.223, p < 0.05), but there are no significant correlations with perceived helpfulness of any of the charts. 5. Students’ perceived helpfulness of the news feed for project management correlates negatively and significantly at the 0.05 level with the dissertation mark (rs =

The promptness of feedback was also analysed using line graphs plotting the number of progress reports submitted on the system by monitors and supervisors against the number of student system logins for each day of the 17-week project period, depicted in Figures6.2.2a and 6.2.2b, respectively. In both cases, student logins are in line with both monitor and supervisor progress reports. In other words, students logged on in response to monitor or supervisor feedback on the system. Students were notified of such activity by means of news feed email notifications sent automatically every 20 minutes. The student response to such emails was relatively fast, that is, in most cases students accessed the system on the same day the progress report was submitted. A Spearman correlation test was also conducted to confirm this relationship. Student logins correlate positively and significantly with monitor progress report submissions (rs= 0.298, p < 0.01), but not with supervisor progress report submissions. Conversely,

both monitor and supervisor logins correlate positively and significantly (p < 0.05) with student progress report submissions (r_s= 0.442 and rs= 0.128, respectively).

In summary, student respondents found progress reports submitted by monitors and supervisors, the ranking table, and email notifications helpful for project management. They also agreed that the system enhanced their progress awareness. However, responses on the helpfulness of charts visualising progress feedback were mixed. Correlation ana- lysis has shown that the submission of event feedback, use of visualisations, and views of the home page positively affected students’ dissertation mark as well as their weekly progress and motivation ratings. Use of the news feed also correlates positively with progress and motivation ratings submitted by monitors and supervisors. Furthermore, students making heavy use of the news feed also agreed that monitoring meetings were helpful, that faculty staff picked them up on feedback submitted on the system, and that the system helped them to communicate problems, keep in contact with faculty staff during their absence, and keep themselves organised. Those students also reported a higher perceived helpfulness of features progress reports from monitors/supervisors and ranking table, although a negative correlation with student performance was also found. Overall, Figure 6.2.2 shows that student response to feedback submitted by monitors and supervisors was relatively prompt – peaks in monitor and supervisor feedback sub- missions are in line with peaks in student system logins. However, correlation tests have also revealed that only the feedback submissions from monitors correlate significantly with student logins, while those of students correlate with both monitor and supervisor logins.