Third phase

The third phase was conducted within two weeks of the second phase. In this phase, one-to­ one interviews following the procedure of T ARSUM, which typically took 25 minutes to complete, were conducted to provide a second measure of students' use of strategies. The interviews with each student, were conducted by the researcher and four trained interviewers who were all postgraduate psychology students. The interviewers were familiarized with the procedure, notation (for recording oral reading behaviours), and the strategy types (see · Appendix B, section B.3 .4, Table B2, for definitions of each strategy type). Detailed verbal

explanations were provided to the interviewers and any points of confusion were clarified. The interviewers then listened to a recording of one of the interview sessions (which was pilot tested with a student not included in the study) and were asked to transcribe the session onto

the interview coding forms. Points of confusion and disagreement were discussed and clarified.

8.4.3.1 Training session for using the bleep

Students came to the interview room individually. An attempt was made to initially set them

at ease through brief conversation about recent school activities. Students were then given a short training session to familiarize them with the use of the monitoring device ("bleep"). Instructions:

Today I want to hear you read a passage. While you are reading, you will be using a "Bleep " [Press the bleep button].

This machine is a "Bleep ". I want you to place this on the table close to you and

keep one finger on the button. (Show the child the button. Comment that it's like

a Space Invaders game). While you are reading aloud, I want you to push the button

when something is not right with the sentence. (Give the sentences without the

asterisks to the student. Ask the student to read aloud).

Sentence 1 : One day a boy went out for a walk. On his way, he saw a dog. The dog began to him· chase. The boy started to run away from the dog. The dog boinkrad" angrily at the boy. The boy was frightened of the cat". Luckily, his friend came to help him with wooden stick a big·.

[The button should have been bleeped at the points marked with an asterisk.]

If the student did not bleep at the appropriate points, the student was told where he/she should have bleeped and why. The interviewer then demonstrated this by reading aloud and bleeping at the appropriate points. Students were then asked to try a further two sentences.

Sentence 2: The lady looking· was pet for her daughter. She thought of buying a cute

little puppy. She paid $ 12.00 for the puppy. Her daughter was happy with

168

Sentence 3 : The lady liked to pick flowers. Everyday she croit • flowers for her garden. Sometimes she the· gave flowers to a friend. Her enemy· liked the flowers too.

[The button should have been bleeped at the points marked with an asterisk.]

During each trial the interviewer observed whether or not the button was pressed at the appropriate points. If the button had not been pressed, the student was asked to consider the sentence again and indicate where a bleep should have been made. Students were also asked what was wrong at each point. The interviewer ensured that students knew when to press the bleep before proceeding to th� next step.

8.4.3.2 Interview

Following the training session, for each student, the interviewer placed the appropriate version of the passage in front of the student. The interviewer had a similar passage to refer to while the student was reading. Students were then instructed to stop at designated points indicated by an asterisk on the passage, where they were asked questions designed to reflect their comprehension and use of strategies.

Instructions:

I want you to read this passage out loud I will be tape recording our voices as I will not be able to remember everything you say. Please don 't be nervous.

The passage will have some asterisks like this (show the student) at the end of some sentences. I would like you to stop at these points as I will be asking you some questions. If you come to a word that you don 't know try your best. I cannot help you. Remember this is not a test, so you can look back at the passage when you are answering the questions, if you wish. Don 't forget to use the bleep when you come to something that is wrong. Please keep your finger on the button while you are reading. Right let 's start. [Switch on tape recorder].

While the student read each segment of the passage, the interviewer indicated on his/her copy of the passage the points where bleeps, errors, successful and unsuccessful self-corrections were made by the student. Each time the student bleeped for a segment of text read, the

interviewer would ascertain the student's reasons and record them on the interview coding form.

While the student answered the prescribed questions, the interviewer scored the student's answer to the comprehension question and recorded the presence of each strategy (by placing a tick in the list given) articulated by the student. If the strategy mentioned was not listed, it was described in the "other" category. Students were instructed to continue reading the next segment after answering the questions at each designated point. If they failed to stop at the designated points, they were reminded to do so.

Encouragement was given for effort made and if a student was fully aware that he/she had done poorly on a question, the interviewer would say, "Many students find this one hard, don't worry". However, no indication was given as to whether their answers to the comprehension questions were right or wrong so as not to invalidate subsequent attempts to evaluate their responses to the comprehension questions. If the student's answers were unclear, the interviewer would request more detail with prompts such as "How do you mean?'' or "Tell me more about it".

For each cognition and metacognition question, the interviewers recorded the presence and categorisation of each strategy as it occurred while questioning the student. All interview data were coded and scored by the researcher to obtain a total score each for comprehension, cognitive strategy use, and metacognitive strategy use based on the transcriptions and tape recordings.

8.4.3.3 Reliability

Fifty percent of each interviewer's coding forms (n = 1 98) were randomly subjected to

reliability checks by the researcher. Reliabilities for occurrence and categorisation of

strategies were calculated using the point-by-point percentage agreement method (Cooper,

Heron, & Reward, 1 987). Interrater agreement was calculated on a question-by-question basis

by dividing the number of agreements by the number of agreements plus disagreements, multiplied by 1 00.

Mean interrater agreement for the occurrence and categorisation of cognitive strategies was 99.4% (range 98.2% - 100%) and 90.6% (range 87.0% - 93.5%), respectively. Mean interrater agreement for the occurrence (M = 1 00%) and categorisation (M = 98. 1%, range

1 70

= 96.9% - 1 00%) of the monitoring and planning strategies was higher than for cognitive strategies. Interrater agreement for combined cognitive and metacognitive strategy use (monitoring and planning strategies only) was also calculated. Interrater agreement was 99.7% (range = 99. 1 % - 100%) for the occurrence of a strategy, and 94.4% (range = 93.5% - 95.5%) for categorisation of a strategy. Interrater agreement was also sought for comprehension scores, bleeps, and for oral reading behaviours. Similar to procedures used

in van Kraayenoord ( 1 986), interrater agreement on oral reading behaviours consisted of the

number of uncorrected errors and the number of successful and unsuccessful self-corrections where there was agreement as to the site of the error or self-correction. Mean interrater

agreement was 95.7% (range = 94.9% - 97.0%) for comprehension scores, 1 00% for bleeps,

97.5% (range = 96.4% - 98.3%) for successful self-corrections, 97.6% (range = 97. 1 % - 98.4%) for unsuccessful self-corrections, and 98. 1% (range = 96.9% - 99.0%) for uncorrected errors. The percentage agreements for the oral reading behaviours compare favourably with

reliability in other New Zealand studies (Clay, 1 973 ; van Kraayenoord, 1 986). In addition,

the fmding that more bleeps occurred at expected points (83 .4%) provided a validity check

that students were using the bleep correctly.

Reliability checks for the aforementioned categories were again made using an independent

rater, a postgraduate psychology student who was not one of the original interviewers.

Interrater agreement on the evaluation activities of metacognitive strategy use was also collected. The independent rater received the same training as the other interviewers.

Ten percent of the 396 recordings (n = 40) were randomly selected for the reliability check.

Comparisons of agreement were made on several variables using the reliability formula as

used previously. The percentage agreement between raters is as follows: 99.0% and 93 .9%

for the occurrence and categorisation of cognitive strategies, respectively; 1 00% and 98.3%

for the occurrence and categorisation of metacognitive strategies, respectively. In addition,

interrater agreement for combined strategy use was calculated; 99.5% for occurrence of a

strategy, and 96. 1 % for categorisation of a strategy. Interrater agreement for oral reading

behaviours included: 95.8% for comprehension scores, 98.2% for successful self-corrections,

97.6% for unsuccessful self-corrections, and 99.2% for uncorrected errors.

\

Procedural validity was also assessed by the researcher and the independent rater in terms of

the appropriateness of the interviewer's verbalizations during the taped section of the interview. Interviewer verbalizations were categorized into correct prompts, incorrect prompts and other verbalizations. Incorrect prompts occurred when (a) student' s comprehension answers were prompted or corrected, (b) the interviewer gave some indication that the student

had answered the question correctly, (c) the interviewer prompted the student as to type of strategy that had been used, or (d) the interviewer had prompted the student to bleep at the appropriate points. Other verbalizations referred to student verbalizations, and interviewer

\

verbalizations which were not related to the T ARSUM.

Each verbalisation was categorised and recorded using 1 5-s intervals in a partial-interval recording procedure (Cooper et al., 1 987) for five minutes of 1 0% of the audiotaped sessions (n = 40). The five minute segments were randomly selected from each 25 minute interview session. Rater agreement was calculated on an interval-by-interval basis using point-by-point agreement ratio method (Cooper et al., 1 987). High validity was indicated as 97. 1 % of the interviewer verbalizations were correct, 1 . 9% were incorrect, and 1% were other verbalizations. There was 1 00% interrater agreement on the occurrence of interviewer verbalizations.