Questionnaires were implemented to perform a quantitative evaluation and are discussed in this chapter. The components of the evaluation plan for the questionnaires are based on the evaluation plan described by Stone et al. (2005) and are discussed in the following subsections.
11.1. Purpose
The questionnaire had multiple purposes. In fact, there could be considered to be two different questionnaires. One was constructed for students in secondary vocational education schools and the other was constructed for parents of such students.
The first and most important purpose of the questionnaire was for the parents to evaluate the requirements and designs. These products of the generative interviews were based mostly on the view of a limited amount of parents. They were of course constructed with the diverse demography of parents with students in secondary vocational education, as discussed in Part One, in mind. Instead of solely relying on the parents that joined the generative sessions to provide a view on parental involvement and the aforementioned application, the questionnaire provided a way to collect information from more parents.
For the same reasons as mentioned above parents were also asked in the questionnaire about their view on parental involvement in secondary vocational education. This was an important part of the questionnaire. It provides a way to validate the representation of the parents that participated in the generative interviews and to answer research question 1. It also allowed to potentially gain more insights that could be applied in the evaluation of the application.
The questionnaire made for students provided a way to potentially and partially answer research question 3b as students are an actor to enabling parental involvement. The students were asked their view on parental involvement. They were also asked about their view on an application that allowed their parents to gain insights in their progress at school. They could state their own requirements for the application. After all, they are in control of their data, especially if they are over 18, as described in requirement R12.
The questionnaire by itself is not completely evaluative. Both the parents and the students did not only evaluate requirements and designs. They were also asked about their view on parental involvement. However, their view is used to evaluate and validate previous findings and the potential effectiveness on the requirements, designs and prototype.
11.2. Procedure
The first thing all participants were greeted with when opening the website of the questionnaires was the welcome screen. A screenshot of this screen can be found in Figure 11.1. This screen explained why the questionnaire was performed and what would happen with the information the participants filled in. The participant had to press a button to begin.
Both questionnaires were located on a single website. They shared several aspects at the beginning and end of the questionnaire with unique questions in between. The website chose the correct path depending on what the role of the participant was in secondary vocational education. The possible paths through the questionnaire are illustrated in Figure 11.2. A description within the paratheses denotes whether it was meant specifically for parents or students.
Figure 11.2: Structure and possible paths through the website.
Each of the blocks illustrated in Figure 11.2 denotes a set of questions with a specific topic. These are described in the following Chapters in more detail.
If the next topic was not self-explanatory, an explanation was shown. This explanation was displayed mandatory for a few seconds before participants could continue. This was done to force the participant to read the information displayed.
The questionnaire was publicly accessible through the internet as an online website and would take participants around 10 to 15 minutes to complete. The website was compatible and tested for tablets as well, because most parents that participated in the generative interviews, as discussed in Chapter 6, stated they make use of them. The questionnaire was online for 2 months before closing down. Participants could win a prize of gift certificates valued 50 euros. The prize was a way of attracting parents and students to fill in the questionnaire.
Participants were gathered in several ways, similar to the generative interviews. A Facebook post was placed with a link to the website. Family and friends were asked if they knew someone who would want to participate. A flyer was printed out that was shared among interested people. Parents participating in the evaluation session discussed in Chapter 14 were also asked to share the flyers with other parents.
11.3. Questions
The questionnaires had several questions in common. These are discussed in this section. All other questions within the questionnaires were specific for either students or parents and will be discussed in Chapter 12 and Chapter 13.
Figure 11.3: Screenshot of the questions about personal details.
Welcome Personal Outside Involvement (parent) Involvement (student) Requirements (parent) Requirements (student) Final Submission Designs (parent) Notifications (parent)
All participants were first asked for their personal information. Note that no questions were asked that could reveal their identity. These questions asked for their gender, age and education level. To determine which specific questions should be applied on the participant, a question asking how the participant is connected to secondary vocational education was included. A screenshot of these questions can be found in Figure 11.3.
Most questions within the questionnaires consisted of statements where participants could indicate to what degree they agree or disagree with the statement. They could do so by selecting a value from a 7-point semantic differential scale as illustrated in Figure 11.4. The internal value for the scale ranges from -3 for completely disagreeing to 3 for completely agreeing.
These values are important to mention as the mean and standard deviation calculated from the results are based on these values. The reason these values were chosen was that a value of 0 then indicates participants were neutral or had no opinion about a specific subject. This way it was easier to conclude from the mean and standard deviation what the general opinion was and whether participants agreed with each other.
Figure 11.4: Example of semantic differential scales used in the questionnaires.
At the end of the questionnaires, participants were asked for final comments or remarks. Participants could fill in anything they would want.
11.4. Constraints
There were some constraints that limited the effectiveness of the questionnaires. These are discussed in this section.
The questionnaires resulted in mostly quantitative data. The richness of this data is very poor. Reasons why a participant chose a certain answer cannot always be given.
Time was a constraint. The questionnaire could not take too much time to fill in as potential participants could then leave the questionnaire without finishing it. The amount of time it was online was limited as well.
The questionnaire for the parents contained several designs that participating parents were asked to judge. However, these designs are not interactive. The designs were animated to reflect the interaction, but actual interaction did not occur. This could have meant that participant might not fully understand how the design would work when implemented.
The geographical spread of both the parents and the students could influence the outcomes of the questionnaire, as mentioned in Part One. The location of the school heavily influences the amount of parental involvement. The location of the participants could not be reliably tested without breaking the anonymous character of the questionnaire.