Data collection

There were three ways of data collection. Data mining of the questions, question- naires for quantitative data, and focus group interviews for qualitative data collection.

Quality of help seeking

The first way was data mining of the web-application, this data was used to analyze the quality of question asking during programming tutorials . The students could choose to ask a question or choose to be put in the sign-off queue. When they choose to ask a question, the students needed to specify; the week of the tutorial of which they had a question, the exercise they were doing, and type out their ques- tions. Or if other students had already asked questions in that category the student could choose to ask their own question or pick one of the previously asked ques- tions. All the questions were saved in a database. After the evaluation the questions were collected from the database. The questions were categorized in general ques- tions, specific questions and errors, see section 2.2.1 for the definitions of these categories. Next to the category of the question the amount of questions in the re- search week were compared with the amount of questions during a regular week. This last step was done to analyze the acceptance of the new features.

Acceptability and effectiveness of the prototype

To evaluate the acceptance of the web application, the opinion of the participants about the new features and the usability of the site. There were two questionnaires, one for the students and a different version for the TAs, because both participants

had a different role and a different features in the site. The questionnaires started with some demographic questions about the participant. They were then asked if they had worked with version 1.0 before the test week. Participants who had no experience with version 1.0 did not have to answer the questions that compared the two versions.

After the comparison of the versions questions about the features of version 2.0 were asked. Most of the questions used statements and a Likert scale. The Likert scale is a scale where the respondent indicates their degree of agreement with the statement [78]. The scale is a bipolar scale with total agreement on the one side and total disagreement on the other [78]. For this study a five point scale was chosen, because research about the completion of surveys indicated that five point scales were more likely to be completed. Qualtrics the tool that was used for the questionnaires advises a three point scale, because most participants make surveys on their mobile phone and the more points the scale has, the harder it is to distinguish them on a phone [79]. However, psychometric scale users advocate using more categories [78]. The questions were grouped in topics, to reduce the cognitive burden on the subjects [80]. To get more insight in why subjects chose an answer several questions asked the subject to explain their answer in an open ended question. Even though this answer format is more cognitively demanding, this format was chosen because it results in more detailed and informative responses [80]. The survey questions of ”Evaluating Discussion Boards ... Collaborative Learning Tool” [70] were used to be able to compare results with the results from that research.

For the questions about usability the questions of Web Accessibility Initiative (WAI) were used [81]. Their questions ask the participant to give a rating of the look and feel of the website. For instance how attractive the site is, and if the participant feels that the site was design with him/her in mind [81]. For these questions the likert scale, contained four categories to force the participant to pick a side.

Distribution The TA questionnaire was distributed via email, however at the last session a TAs was asked if they had found the questionnaire and it turned out that not all the TAs had received the link. Responding to this the survey link was then distributed via WhatsApp by that TA. During a session about 20 TAs where in the classrooms, some of the TAs did not work all the sessions. The TA questionnaire got 23 respondents.

The distribution of the student survey started at the beginning of the second session. The survey for the students was distributed via a link that was put in a header announcement in the site. Though, after the session only 28 of the 300 students of the course had responded. Wherefore, the module coordinator was mailed if he could spread the link to the survey on the Canvas. After sharing the link

87

in an announcement on Canvas 144 students started the survey and 102 completed the questionnaire.

Focus group interviews

The method of these focus group interviews was the same as the method that is described in section3.1.1. The difference was the purpose statement for this group which was; To get the opinion of the subjects about version 2.0. Because the pur- pose of this interview was different the questions were also different. The moderator guide for these interviews can be found in appendixK.

Tools

Needs more explanation, about the functions and used for... • MaxQDA, coding of transcripts

• Qualtrics, online questionnaire tool, professional appearance are taken more serious [80]