Subjective Student Feedback

As mentioned earlier, students were asked to take part in a closing survey, containing questions about the perceived motivational effect of system features, their helpfulness for MSc project management, and users’ agreement with general statements. What is more, they were asked to rate each of the dashboard charts with regard to its helpfulness for instigating progress awareness. Finally, users were enabled to submit a free-text comment or suggestion as for how to improve the system and make it more useful in the future.

It is common practice to conduct a reliability analysis on any questionnaire used in a study. Reliability analysis attempts to verify that individual questionnaire items remain consistent regarding what is measured by the questionnaire, in other words, if a set of questionnaire items is about the motivational effect of features, then a user who scores high on the set should also score high on randomly chosen individual items within that set. In practice, a method called split-half reliability is used to test this. It splits the data into two equal parts and calculates a score per part for each person who took the survey. If both scores are the same or very similar, one assumes a good scale reliability. This is measured using a correlation test between the two halves; if the correlation is strong this indicates a high reliability. Obviously, this method is dependent on the way of splitting the data, and hence the correlation test result might be biased. Another more common method is Cronbach’s alpha test which is roughly equal to the split-half method. It comes up with an α-value indicating the scale reliability. Values greater than 0.7 for α are acceptable, assuming a high reliability.

Taking a conservative approach, both methods were used on all closing questionnaire items which were to be answered using a 5 or 6-point scale. Since different parts of the questionnaire measure different things, a separate analysis was performed on each survey part. The test results are shown in Table 6.3.

Table 6.3: Survey reliability analysis results Split-Half Method Category Valid Cases Items Cronbach’s α Items per Half r

Motivational effect of features 104 10 0.919 5, 5 0.798

Helpfulness for MSc Project Management

77 11 0.949 6, 5 0.889

Chart helpfulness for progress awareness

104 6 0.966 3, 3 0.893

Agreement with general statements

104 8 0.946 4, 4 0.839

The table shows a very high α-value (α > 0.9) as a result of Cronbach’s alpha test, and a very high correlation coefficient (r > 0.7) between the two halves used in the split-half method. This indicates that all items have a very high scale reliability.

6.1.4.1 Motivational Effect of System Features

The first part of the questionnaire was about the perceived motivational effect of certain system features. Students were asked to submit a rating from 0 to 5 on each questionnaire item, whereby 0 denotes “did not use”, and [1, 5] ranges from “did not motivate at all” to “motivated very much”. There were N = 104 valid cases for each item, that is, 104 students answered all 10 items of this questionnaire part. The results are shown in Table 6.4, whereby the maximum percentage per item is underlined.

Table 6.4: Motivational effect of system features Percentage %

# Feature Mean 0 1 2 3 4 5

1 Dashboard and charts 2.84 5.8 15.4 14.4 27.9 26.9 9.6

2 Ranking table 2.93 5.8 13.5 17.3 26.9 18.3 18.3 3 News feed 2.87 5.8 13.5 17.3 27.9 23.1 12.5 4 Email notifications 3.43 1.0 9.6 10.6 24.0 33.7 21.2 5 Self-reported progress 3.38 1.9 12.5 7.7 22.1 35.6 20.2 6 Monitor-reported progress 3.54 7.7 6.7 3.8 20.2 28.8 32.7 7 Supervisor-reported progress 3.61 5.8 7.7 2.9 19.2 32.7 31.7

8 Event attendance feedback 3.47 2.9 11.5 6.7 20.2 31.7 26.9

9 Task list 3.16 2.9 14.4 9.6 25.0 32.7 15.4

10 Comments on group pages 2.82 7.7 14.4 12.5 31.7 21.2 12.5

Of all 104 students who provided answers to these items, 32.7% percent think that monitor-rated student progress was motivating, followed by self-reported progress (35.6%), email notifications (33.7%), and supervisor-rated progress as well as the task list on the project page (32.7%). Surprisingly, the ranking table was one of the features perceived

least motivating with only 36.6% of students rating it motivating or very motivating. The least used feature was group page comments, while email notifications were used by most users.

6.1.4.2 Feature Helpfulness for MSc Project Management

In the second questionnaire part, students were asked to rate the helpfulness of features for managing their MSc project. Again, answers could be chosen from a scale ranging from 0 to 5 with 0 denoting “did not use”, 1 indicating “not helpful at all” and 5 meaning “extremely helpful”. There were between 96 and 104 valid cases depending on the item, that is, some students did not provide an answer to some of them. The results are shown in Table 6.5.

Table 6.5: Feature helpfulness for MSc project management Percentage %

# Feature N Mean 0 1 2 3 4 5

1 Dashboard and charts 96 2.87 3.1 15.6 18.8 27.1 24.0 11.5

2 Ranking table 97 2.89 2.1 20.6 14.4 26.8 21.6 14.4

3 News feed 98 2.92 3.1 15.3 18.4 24.5 27.6 11.2

4 Email notifications 102 3.36 0.0 8.8 14.7 27.5 29.4 19.6

5 Monitor-rated progress 96 3.48 3.1 8.3 11.5 20.8 27.1 29.2

6 Supervisor-rated progress 97 3.78 2.1 7.2 4.1 20.6 28.9 37.1

7 Event attendance feedback 100 3.62 1.0 9.0 6.0 24.0 31.0 29.0

8 Task list 101 3.30 2.0 6.9 14.9 26.7 34.7 14.9

9 Event organisation 104 3.38 1.9 5.8 10.6 32.7 31.7 17.3

10 Data export 104 2.87 7.7 14.4 11.5 29.8 23.1 13.5

11 Comments on group pages 96 3.17 3.1 10.4 15.6 27.1 25.0 18.8

Features perceived most helpful for MSc project management were monitor-rated and supervisor-rated student progress feedback with 29.2% and 37.1% of students assigning the highest helpfulness rating, respectively. Again, the ranking table was not perceived useful; 35% of respondents rated this feature less helpful or not helpful at all. Finally, a majority of respondents were undecided regarding the helpfulness of the data export feature.

6.1.4.3 Chart Helpfulness for Progress Awareness

The third part was about perceived helpfulness of various charts displayed on the system in terms of if and to what extend they managed to support students’ awareness for their own progress compared to others. There were N = 104 valid responses for all 6 items in this category, whereby answers could be chosen from a 6-point scale ranging from 0 to 5 with 0 denoting “did not use”, 1 indicating “not helpful at all”, and 5 “extremely helpful”. Table 6.6shows the result.

Table 6.6: Chart helpfulness for progress awareness Percentage %

# Chart Mean 0 1 2 3 4 5

1 Weekly ratings histogram 2.73 5.8 17.3 18.3 26.9 20.2 11.5

2 Weekly ratings over time 2.68 9.6 14.4 18.3 24.0 23.1 10.6

3 Event attendance over time 2.74 8.7 14.4 17.3 26.0 21.2 12.5

4 Report metrics 2.72 11.5 10.6 18.3 22.1 28.8 8.7

5 Milestone task progress 2.83 7.7 11.5 17.3 26.9 26.9 9.6

6 Task progress 2.84 7.7 11.5 19.2 24.0 26.0 11.5

Proportionally, the report metrics, milestone task progress, and task progress charts were rated most helpful for increasing progress awareness with 28.8%, 26.9%, and 26.0% of users assigning the second-highest rating to these charts, respectively. For all other charts, users were mostly neutral regarding their helpfulness. The least used chart was the report metrics graph plotting weekly report quantity metrics, possibly because most users did not give accurate feedback on these metrics and generally started writing relatively late into their project.

6.1.4.4 Agreement with General Statements

In the final part of the questionnaire, a number of statements concerning student use of the monitoring system and MSc project management in general was provided. Students could rate each of these statements on a 5-point Likert scale from 1 to 5, whereby 1 denotes “do not agree at all” and 5 “fully agree”. A total of 104 student provided valid responses to these items. Furthermore, students were asked to give feedback on the frequency of meetings with their monitor. This could be done on a scale from 0 to 5, namely “N/A” (0), “never” (1), “when I needed to” (2), “about every third week” (3), “about every second week” (4), “every week” (5). Students’ feedback to general statements is listed in Table 6.7.

Feedback on the general helpfulness of the system and the monitoring scheme was mixed. The majority of students found that monitoring meetings helped them, however, they were neutral with regard to the monitoring system enhancing their project management, the helpfulness of the system for keeping them organised, communicating problems, and keeping contact with their monitor and/or supervisor. Most students also found that their monitor and/or supervisor picked them up on progress reports submitted on the system. Furthermore, they agreed on the system raising their progress awareness and enabling them to compare themselves with their peers. On the other hand, peer-to-peer communication was not something they found facilitated by the system, and in fact no special features for this purpose were provided.

Regarding the frequency of monitoring meetings, the majority of students (54.8%) in- dicated that they met with their monitor every week, followed by “when I needed to”

Table 6.7: Agreement with general statements

Percentage %

# Statement Mean 1 2 3 4 5

1 Found monitoring meetings helpful 3.74 12.5 3.8 19.2 26.0 38.5

2 Monitor/supervisor picked me up on my progress reports

3.41 13.5 9.6 23.1 29.8 24.0

3 System raised my progress awareness 3.34 13.5 12.5 24.0 26.9 23.1 4 System helped me keep myself

organised

3.13 17.3 10.6 29.8 26.0 16.3 5 System helped me communicate

problems to my monitor/supervisor

2.93 26.0 10.6 26.0 19.2 18.3 6 System helped me keep in contact with

monitor/supervisor during absence

2.85 26.0 12.5 27.9 18.3 15.4 7 System enabled me to contact other

students

2.63 32.7 14.4 24.0 15.4 13.5 8 System enhanced my MSc project

management

3.09 13.5 19.2 28.8 22.1 16.3

(14.4%) which is equivalent with attending meetings in irregular intervals, and “about every second week” (9.6%). Only 3.8% of students admitted they never met their mon- itor.

6.1.4.5 General Feedback

Part of the general feedback students submitted was whether any other online or off- line tools besides the MSc Monitoring System were used for MSc project management. Furthermore, they could submit a free-text comment on how to improve the system and make it more useful. In total, there were 63 valid responses to either part of the general feedback category, this is about 57% of all 110 users participating in the survey.

The most popular response (17.4%) was the use of Microsoft Project and a paper-based log book for project management, closely followed by other paper-based tools (12.7%) and Google Calendar (11.1%). Less used tools were Microsoft Outlook (6.3%), Microsoft OneNote (4.8%), Microsoft Visio, Apple iCal, and Notepad (3.2% each). Moodle, private Wikis, Subversion, Evernote, and Microsoft Word were rarely used with only 1.6% of respondents mentioning them in their answers.

Regarding suggestions for system improvement, 10 out of 63 respondents (15.9%) expli- citly wanted the system to be continued, compared to 5 respondents (7.9%) who did not find the system useful at all and recommended it to be discontinued. Main user critique was that both system navigation and user interface need improvement (11.1% and 9.5%, respectively). Furthermore, 6.3% of respondents suggested to provide more interactive tools enabling user-to-user communication on the platform and improved charts/graphs; one user even requested an online chat feature. A few users (3.2%) complained about the number of emails they received from the system and recommended to reduce this

to an absolute minimum. Online storage space for personal notes and files as well as compulsory system use for all participants were also suggested.

6.1.4.6 Feedback from Semi-Structured Informal Interviews

As mentioned earlier (see section5.4.3 on page 125), semi-structured informal interviews were carried out about two to three weeks into the MSc projects in three randomly se- lected monitoring groups. All interviews took place at the end of the weekly monitoring meeting and were aimed at collecting general feedback concerning the monitoring scheme and meetings, on the monitoring system features and their perceived usefulness, sugges- tions for improving the system or additional features desired, and answering questions students had regarding features and their use.

Student feedback received in the interviews is very much in line with general comments submitted in the closing questionnaire. The most common problem encountered was that students did not know about the purpose of certain features and how to use them as part of their project monitoring and/or supervision. This was despite of the system being presented to students at the beginning of their project. This clearly indicates that some features were not self-explanatory enough. Some students also attended several group meetings, especially in groups where two or more meetings were offered per week, and could not find this reflected on the system. Also, there were a few students who complained about monitors/supervisors not remembering what their project was about. Although the system provided means of indicating project title and description on the project page, many students were not aware of this feature and rarely used it.

There were also comments on particular system features. For example, one student sug- gested to include more detailed context information in system email reminders. Others requested more comparative charts and graphs with drill-down functionality, and they would also like the graphs to be bigger and more readable. In terms of the number of emails, students were divided: while some complained about it being to high, others explicitly requested it to be increased. The latter group of students also admitted that for them email notifications were the main incentive to use the system and that they used direct links in these emails to access the system.