Classical test analysis reading tests
Table 8
Pre-testing item statistics
Mean Std. Deviation N Pers 1 .80 .45 5 Pers 4 .60 .55 5 Pers 5 .80 .48 5 Pers 6 .60 .55 5 Pers 7 .80 .45 5 Leven 2 .80 .45 5 Studie 2 .80 .45 5 Studie 3 .80 .45 5 Studie 4 .40 .55 5 Spier 7 .80 .45 5 Comp 8 .80 .45 5 Studie 5 .20 .45 5 Werk 2 .80 .45 5 Werk 3 .80 .45 5 Werk 4 .80 .45 5 Comp 6 .60 .55 5 Table 9
Pre-testing 1 reliability statistics
Cronbach’s Alpha Cronbach’s Alpha based on standardized items
N of items
Evaluation of the workshop Effectief studeren and the online course Snellezen – M.D. Otten
57
Table 10
Pre-test item statistics
Scale mean if item deleted Scale variance if item deleted
Corrected item-total correlation
Cronbach’s alpha if item deleted Pers 1 10.40 8.30 ,85 ,63 Pers 4 10.60 7.80 ,85 ,62 Pers 5 10.40 8.30 ,85 ,63 Pers 6 10.60 7.80 ,85 ,62 Pers 7 10.40 8.30 ,85 ,63 Leven 2 10.40 8.30 ,85 ,63 Studie 2 10.40 11.80 -,42 ,76 Studie 3 10.40 8.30 ,85 ,63 Studie 4 10.80 8.70 ,53 ,67 Studie 5 11.00 11.50 -,33 ,76 Spier 7 10.40 10.30 ,07 ,72 Werk 2 10.40 10.30 ,07 ,72 Werk 3 10.40 12.30 -,57 ,78 Werk 4 10.40 8.30 ,85 ,63 Comp 6 10.60 11.30 -,24 ,76 Comp 8 10.40 11.80 -,42 ,76
The p-values show that the reading test was easy because most values are 0.80 (or 1) according to Table 8. The cronbach’s alpha is .71 according to Table 9, that is already sufficient if you want to do research (Kottner et al., 2011).
Table 10 shows scale mean and variance if item deleted, corrected item-total correlation and cronbach’s alpha if item deleted. Items of the text “Zó haalt u de pers” and “Levensverwachting” have a high corrected-item-total (0.8<), which is positive. The text of “Succesvol studeren?” and “Voor computer klinkt ‘nog één’ en ‘nog geen’ hetzelfde” have both two negative items. Question 3 of the text “Meer verdien? Harder werken!” Has the lowest corrected-item-total correlation and would also improve the cronbach’s alpha the most if it would be deleted (to 0.78). Participants of the pre-testing also stated that this question could not be answered if you are allowed to read the text only once and you could not remember the numbers at the start of each paragraph. That is why decided to deleted question 3 of the text “Meer verdien? Harder werken!”.
Questions 2 and 5 of the text “Succesvol studeren?” scored the lowest at the corrected-item total. Feedback of the participants did not affirm that question 2 was wrong but confirmed that question 5 was bad because it was not clear what was meant by the last paragraph (the layout was confusing). Thus question 5 would be deleted.
Questions 6 and 8 of the text “Voor computer klinkt ‘nog één’ en ‘nog geen’ hetzelfde” also scored low on the corrected-item total. Participants also offered feedback that the word “inleiding” was confused and it was recommended to remove by several people. Only one person criticized question
Evaluation of the workshop Effectief studeren and the online course Snellezen – M.D. Otten
58 8 (did not like the addition “zoals verwacht”). That is why is decided to only remove question 6 because several sources affirmed that this was a bad question.
The classical analysis was done after removing the three items. The cronbach’s alpha started at 0.71 and after removing the 3 items, the cronbach’s alpha was 0,85 according to Table 9 and 11. Cronbach’s alpha of 0.6, 0.7 or 0.8 are usually already acceptable as minimum standards for research purposes (Kottner et al., 2011).
In concrete terms, the classical test analysis means for the readings tests the following adjustments:
Question 3 of the text “Meer verdien? Harder werken!” is deleted. However, every text need enough questions. Besides that, many participants indicated this text as “worst” to read because this text included a lot of details. It is decided to remove the whole text.
The headings of the text “Succesvol studeren?” were confusing according to question 5 of this text. The headings “motivatie” en “houding” were removed to better signal what the paragraphs are. Question 6 of the text “Voor computer klinkt ‘nog één’ en ‘nog geen’ hetzelfde” is the word “inleiding” replaced for “tekst”. The adapted question is “wat wordt in de tekst gezegd over het verschil tussen mensen en computers bij spraakherkenning?”.
Other adjustments based on the feedback of the participants of the pre-testing:
A participant that graduated from the master Educational Science and Technology, also recommended to have texts of the same length (similar kind of texts are the best). The text “Zó haalt u de pers” is in ratio very long. The paragraphs about no questions are asked, were removed to decrease the amount of words of the text. Many participants of the pre-testing indicated that question 4 was a bad question because another answer seems also possible, thus this question and associated paragraph were deleted. Comparing to the other texts, this text has one question to much, but based on the classical test analysis and the feedback, no question stands out to be removed. That is why the longest paragraph has been removed to be closer to the average amount of words of each text (in this case “tip 2”).
Furthermore, “too many” texts were test in order that some bad reading test could be discarded. Based on the classical test analysis, the text “Meer verdien? Harder werken!” was dismissed. The design of the research is that in the pre-test two texts are read and in the post-test 1 two texts. Thus still one text should be deleted. Texts should be equal as possible. The length of texts plays a role because it is easier to remember the content of a short text (Smith & Ragan, 2005). That is why the text “Levensverwachting” has been excluded because this text has significantly fewer words comparing to the other texts (401 words instead of 700-750 words).
Furthermore, the procedure for administering the reading test of the online course have been more specified. Before the participants starts to read the text, they have to write down what the time is.
Table 11
Pre-testing 2 reliability statistics
Cronbach’s Alpha Cronbach’s Alpha based on standardized items
N of items
Evaluation of the workshop Effectief studeren and the online course Snellezen – M.D. Otten
59 Subsequently, they read the texts while the computer tracks the time they stay at the page. Afterwards, they are asked to write down what the time is after they read the text. Participants need to believe that they will be assessed on their time. Some participants will maybe be surprised after the first text that also will be asked some reading comprehension questions. It would be interesting to analyse if their reading speed will change when they read the second text.
Feedback participants pre-testing other instruments
Below a summary is written of the feedback of the participants about the other instruments.
Memorytest
Change “biologisch wasmiddel” to only “wasmiddel” because it is unclear if it would be approved if you only remembered “biologisch” or “wasmiddel”. Besides that some participants actually mentioned that they could easier remember those two words because it was different.
Questionnaires
One of the participants noticed that people prefer short questions and that questions are asked in one format. For instance, the question “Heb je dyslexie?” has been changed to “Dyslexie: ja/nee”. The format for answering questions has been set as “ja/nee, omdat …..”.
Several participants noticed that their name was asked whether the informed consent stated that the data would be anonymous. That is why is chosen to write down the explanation of the question in the following phrase behind that question: “Dit is om deze vragenlijst te kunnen koppelen aan je andere vragenlijsten en testen. Alle gegevens worden strikt vertrouwelijk behandeld en anoniem gerapporteerd.”.
Furthermore, the question about the SMART goal is deleted in the pre-test of the online course
Snellezen because people did not yet formulated their goal. Moreover, another question is added to
the post-test 1 of the online course Snellezen to gain insights if the results of this research are comparable to the results of the online course’s tests (“Welke waarden heb je opgeschreven in je voortgangstabel van de online cursus Snellezen?”).
Interview
The book “Leren Communiceren” of Steehouder, Jansen, Maat, van der Staak, de Vet, Witteveen and Woudstra (2006) was recommended to become inspired for follow-up questions. Besides that, guidelines for a structure of an interview are adopted.
Observation scheme
The observation scheme is translated back to the Dutch version because the research is conducted in Dutch but reported in the thesis in English. Besides that, some examples are added to clarify what each topic contains.
Informed consent
For the workshop Effectief studeren and the online course Snellezen are seperated information brochures written. For practical reasons, the participants of the online course have to tick a box to confirm the agreements instead of signing the informed consent.
Evaluation of the workshop Effectief studeren and the online course Snellezen – M.D. Otten
60