7. Main Survey Sampling Tasks
7.7 Sampling Task Twelve: Submitting School Participation Information and Validity Checks to
126. You will need to submit your School Participation instrument (SCP), various other KeyQuest instruments (List of schools, List of students, Student Tracking Form, etc.), and various KeyQuest validity reports to Westat (STF Internal checks regarding student non- participation codes and SEN code combinations, student non-participation codes and student demographic information, and study programme checks; School participation checks; checks on differences between the school frame ENR values and the schools’ actual number of PISA students; checks on CBA and the UH booklet, if either were optional components of the country). See the DMM for further details on the KeyQuest instruments, the validity reports, and data submission.
127. Table 3 summarizes the school participation codes and their meanings. Table 3 also indicates under which circumstances a replacement school should be used.
TABLE 3: PARTICIPATION CODES
Numeric Code for KeyQuest SCP
Code Meaning Use Replacement?
1 Participant No
2 Refusal
Yes
3 Non-participation for other
reasons
Yes
4 School participated but no
students were assessed
Yes
5 No PISA-eligible students No
6 School closure No
7 Replacement school not contacted
Not Applicable 8 School participated but
nonexcluded sampled students were ineligible
No
9 School exclusion No
10 School has no PISA-eligible students but has some national option students
No
11 School ineligible for other reasons (please specify)
No Due one month after the end of the data
Numeric Code for KeyQuest SCP
Code Meaning Use Replacement?
12 National Option school to be sampled separately- not needed in 2012
Not Applicable
128. The school participation information needs to be filled out in the SCP instrument in KeyQuest at the end of all sampling processes and submitted to Westat four weeks after the assessment as part of the ST12 data submission task (See the DMM). The following guidelines explain the use of the participation codes in Table 3 under different circumstances.
• If a sampled school’s participation has been confirmed and at least one age-eligible student who was not excluded was successfully assessed, a “1” should be recorded in the School Participation instrument in KeyQuest to denote a participant.
• If a sampled school is unwilling or unable to participate, then enter “2” for refusal, or “3” for non-participation due to other reasons, for instance, school fire, school flood, teacher's strike, etc. A replacement can be used for such a school.
• If the value “3” is used, the reason for non-participation must be entered in the E3COM comment field available in KQ.
• If a sampled school has agreed to participate, but all sampled students who are age- eligible and not excluded refuse to take the test or are absent, NPMs should treat the school as a nonrespondent, and should assign the code ”4” to this school. If time permits, a replacement can be used for such a school.
• If the sampled school is ineligible, then record a “5” for a school with no PISA- eligible students, “6” for a school closure, and “11” for ineligible for other reasons. • If the value “11” is used, the reason for school ineligibilty must be entered in the
E3COM comment field available in KQ.
• NPMs may consider using code “11” in the case when a school was mistakenly listed on the frame twice and both records were selected into sample. In this case only one record will be kept. The duplicate record will be made ineligible and assigned a participation code “11” so that the school does not have problems with multiple chances of selection and double coverage. Please contact Westat if this happens to help decide which record to keep and which to call ineligible.
• If a sampled school has agreed to participate, but all sampled students who are not excluded are found to be ineligible (e.g., in grade 6 or lower, birthdates recorded incorrectly and not age-eligible, school transfer after being sampled, drop out, etc), NPMs should enter the code ”8” to this school in the SCP Instrument.
• Furthermore, enter code “9” for a school that belongs to one of the school-level exclusion categories recorded on Sampling Task 7b Question 2, or a school where all age-eligible students are otherwise excluded. Note that a school participation status of “9” requires the school-level exclusion type as indicated on ST7b Question 2 to be noted in the E3COM field in the SCP Instrument in KQ, as well as the number of excluded PISA students. Thus, if a sampled school is found to be a school for students with special needs and these schools have been listed as school-level exclusions in Question 2 of ST7b on the line beginning with “1-”, then the school
participation status in SCP will be “9” and “EX-1” should be recorded in the E3COM space along with the number of PISA-eligible students in the school (e.g., EX-1: y students).
• If a school in the sample is found to have all age-eligible students otherwise excluded and no specific school-level exclusion of this type has been previously listed on ST7b (e.g., a school for the blind or more generally a special education school), then use ”9” as the status code as usual. For this additional exclusion type, include an exclusion type number in E3COM, “EX-n”, where n is one greater than the number of school level exclusion types noted on ST7b Question 2. Use the same exclusion type number for other sampled schools that are excluded for the same reason. Record also the number of PISA eligible students in the school (e.g., EX-n: y students). Also provide the description for this exclusion type “n”.
• Consider some “9” excluded school examples. Suppose that on the ST7b Question 2 a country excluded language X schools on the line beginning with “1-” and language B schools on the line beginning with “2-”. During data collection, a sampled school is found to have all 10 PISA students instructed in language X. Such a school needs to have a “9” status code and an “EX-1: 10 students” in the E3COM space.
• Further suppose that another school agrees to participate but at the time of assessment, it is determined that all 35 sampled students have been excluded on the basis of the first two within-school exclusion codes. Such a school needs to have a “9” status code and an “EX-3: 35 students” in the E3COM space, with an added description of exclusion type 3 (i.e. students with SEN=1 or SEN=2).
• Suppose once more that a school agrees to participate but at the time of assessment, it is determined that all 30 sampled students have been excluded on the basis of limited language proficiency. Such a school needs to have a “9” status code and an “EX-4: 30 students” in the E3COM space, with an added description of exclusion type 4 (i.e. SEN=3 school).
• If another school agrees to participate but at the time of assessment, it is determined that all 15 sampled students have been excluded on the basis of the fourth “other” within school exclusion codes, such a school needs to have a “9” status code and an “EX-5: 15 students” in the E3COM space, with an added description of exclusion type 5 (i.e. schools with “other” excluded students).
• If another school agrees to participate but all 20 students are instructed in language Y for which there are no test materials, then such a school needs to have a “9” status code and an “EX-6: 20 students” in the E3COM space, with an added description of exclusion type 6 (i.e. no test materials in language of instruction).
• Finally, if a school agrees to participate but all 5 sampled students are excluded for a variety of reasons, then such a school needs to have a “9” status code and an “EX-7: 5 students” in the E3COM space, with an added description of exclusion type 7 (all students excluded for a variety of reasons).
• If a sampled school’s participation has been confirmed and there are no PISA-eligible students in the school (unexpected) but there are national option students, a “10” should be recorded in the School Participation instrument in KeyQuest to denote a participant for the national option only.
• Since all original sampled and replacement schools need to have a final school participation status, all replacements that were not contacted need to have a “7” status code.
• If a country has any schools that have only National Option students in them, by design, then they need to have a final school participation status code of 12. Only one or two countries might need to use this code.
• If a country has students in a school that fit none of these descriptions, contact Westat for help with assigning a school status code.
129. After all contacts with original sampled schools have been made, NPMs will need to contact replacement schools for those sampled schools that will not participate. Each sampled school that will not participate (“02”, “03”, or “04” status codes only) should be replaced if possible. Replacements should NOT be used for reasons other than these. If the original sampled school is ineligible or excluded, do NOT use any replacement schools for it. Additionally, an original school with any eligible students should not be replaced just because the number of eligible students might be smaller than expected.
130. Second replacement schools should only be used if both the corresponding first replacement school and sampled schools will not participate.
131. As for sampled schools, the participation status of approached replacement schools should be coded as described in paragraph 128.
Appendix A - Treatment of Small Schools
132. For PISA, we identify small schools, subdivided into moderately small schools, and very small schools. A small school is defined as any school whose approximate PISA enrolment falls below the TCS. A very small school is a small school whose PISA enrolment is less than a half the TCS. A moderately small school is a small school whose PISA enrolment is in the range of TCS/2 to TCS. Small schools in the sample can result in a reduced sample size of students for the national sample, below the desired target. Additionally, the sample may contain many small schools, which is an administrative burden. In order to minimise these problems, the small schools in the sampling frame may require special treatment.
133. Information on the importance of small schools will be reported in ST9 and in ST10 on the Small School Analysis sheet in the frame file that is returned to the NPM. As in PISA 2009, no small school strata will be formed. The SSA is done only to enable the final sample size calculation and to determine if very small schools should be undersampled. The small school analysis is first done on the full school frame overall, and then also for each explicit stratum. The calculated sample size for an explicit stratum is the number of schools to be sampled from all schools in the explicit stratum. Note that because schools are sampled randomly, the numbers of large, moderately small, and very small schools actually selected may vary from the numbers calculated in the explicit stratum small school analysis.
134. The following explains the guidelines that Westat and ACER will generally follow to determine the final school sample size and whether or not undersampling of small schools is needed.
135. To determine whether very small schools should be undersampled and whether sample sizes need to be increased because of small schools, the following test will be applied. If none of the following conditions are true, then no undersampling of very small schools is needed and no increase in the school sample size is needed.
• If the percentage of students in very small schools (ENR<TCS/2) is 1 percent or MORE and the percentage of students in moderately small schools (TCS/2<ENR<TCS) is 4 percent or MORE, then very small schools will be undersampled and the school sample size will be increased.
• Otherwise, if the percentage of students in very small schools (ENR<TCS/2) is 1 percent or MORE, then very small schools will be undersampled and the school sample size will be increased.
• Otherwise, if the percentage of students in very small schools (ENR<TCS/2) is LESS than 1 percent, and the percentage of students in moderately small schools (TCS/2<ENR<TCS) is 4 percent or MORE, then there is no undersampling of very small schools but the school sample size will be increased.
• Otherwise, there will be no adjustments to the school sample size.
136. The following discussion gives the approach that will be used by Westat and ACER to determine the minimum sample size of small schools (and of larger schools), that ensures that the resulting student sample is representative, yet gives some control to limit the total sample size of schools in extreme cases. The ensuing discussion, based on an overall small school analysis on the full school frame, assumes that the school sample size needs to be inflated because of very small and moderately small schools, and that very small schools need to be undersampled. The formulae below also require modification when TCS is other than 35. 137. If undersampling of very small schools is required, then the undersampling will be done in
two parts. Schools with enrolment of PISA students equal to zero, one, or two, will be undersampled by a factor of four, and schools with enrolment of PISA students greater than two and less than TCS/2 will be undersampled by a factor of two.
138. The small school analysis is conducted as follows:
a. From the complete sampling frame, find the proportions of total ENR that come from very small schools with zero, one or two PISA students (P1), very small schools with more than two PISA students but fewer than TCS/2 PISA students (P2), moderately small schools (Q), and larger schools (those with ENR of at least TCS) (R). Thus P1+P2+Q+R=1.
b. Calculate the figure L, where
L = 1 + 3(P1)/4 + (P2)/2. Thus L is a positive number, a little more than 1.0.
c. The minimum sample size for larger schools is equal to 150 x R x L, rounded up to the nearest integer.
d. Calculate the mean value of ENR for moderately small schools (MENR), and also for very small schools (V2ENR and V1ENR). Thus MENR is a number in the range of TCS/2 to TCS, V2ENR is a number no greater than TCS/2, and V1ENR is in the range of zero to two.
e. The number of moderately small schools that need to be sampled is given by the formula below and then rounded up:
(5250 x Q x L) / (MENR).
f. The number of very small schools of type P2 that need to be sampled is given by the formula below and then rounded up:
(2625 x P2 x L) / (V2ENR).
g. The number of very small schools of type P1 that need to be sampled is given by the formula below and then rounded up:
(1313 x P1 x L) / (V1ENR).
139. Consider the following example. Suppose that in Country X, the TCS is equal to 35, with 0.1 of the total enrolment of 15-year-olds in moderately small schools, with 0.05 in very small schools of type P2, and 0.05 in very small schools of type P1. Because enrolment in very small schools is 1% or more, undersampling of very small schools is required. Suppose that the average enrolment in moderately small schools is 25 students, while the average enrolment in very small schools of type P2 is 12 students, and in very small schools of type P1 is 1.5 students.
• Thus P2 = 0.05, P1 = 0.05, Q = 0.1, R = 0.8, MENR = 25, V2ENR = 12, and V1ENR = 1.5.
• From b, L = 1.0625.
• Thus, from c, the sample size of larger schools should be at least 150 x 0.8 x 1.0625 = 128. That is, there should be at least 128 of the larger schools sampled.
• From e, the number of moderately small schools required is (5250 x 0.1 x 1.0625) / 25 = 22.3; i.e., 23 schools.
• From f, the number of very small schools required of type P2 is (2625 x 0.05 x 1.0625) / 12 = 11.6; i.e., 12 schools.
• From g, the number of very small schools required of type P1 is (1313 x 0.05 x 1.0625) / 1.5 = 46.5; i.e., 47 schools.
• This gives a total sample size of 128 + 23 + 12 + 47 = 210 schools. The larger schools will yield an initial (i.e., before considering school and student nonresponse) sample of 128 x 35 = 4480 students. The moderately small schools will give an initial sample of approximately 23 x 25 = 575 students. The very small schools of type P1 will give an initial sample size of approximately 12 x 12 = 144 students. The very small schools of type P1 will give an initial sample size of approximately 47 x 1.5 = 71 students. Thus the total initial sample size of students is 4480 + 575 + 144 + 71 =5270. Our sample, modified to handle the small schools, will yield approximately the target student sample size that we seek. Although more schools are being sampled with this modification, many of the 47 P1 sampled schools will have no PISA students and will therefore be ineligible so that little effort needs to be spent on those schools.
140. Why this adapted approach? Several countries have a large number of schools with zero, one, or two PISA students, and in PISA 2009 this initially resulted in a very large school sample size. These countries requested undersampling all very small school by a factor of four to alleviate the large sample size. Undersampling all very small schools by a factor of four however, will usually have adverse effects on sampling variability and should be avoided. Thus, this modification is a compromise so that only the very smallest schools may be undersampled by a factor of four and the other very small schools by the more acceptable factor of two.
141. If a country does not have the need to undersample very small schools, then Westat and ACER will set MOS = TCS for all small schools. If a country needs undersampling of very small schools, then this is accomplished during sample selection by Westat and ACER setting MOS = TCS/2 for very small schools of type P2 (e.g., schools with size smaller than TCS/2 and greater than two), setting MOS = TCS/4 for very small schools of type P1 (e.g., schools with sizes zero, one or two PISA students) and setting MOS = TCS for moderately small schools (e.g., schools with size between TCS/2 and TCS).
ST9: PISA 2012 Main Survey School Sample Selection: Small School Analysis
Westat and ACER will use student enrolment data provided by the NPM on the sampling frame to determine the prevalence of small schools in their school system. Based on these enrolment data, specific sample design decisions may be required if the presence of small schools is substantial. If sampling will not be conducted by Westat and ACER, the analysis of small schools is still needed both for the overall country and also by explicit stratum. A worksheet is provided in Appendix D to assist NPMs with the evaluation procedure.
ST9: PISA 2012 Main Survey School Sample Selection: Small School Analysis – Example
Using a hypothetical scenario for Atlanta’s PISA 2012 MS in Appendix C, there are more than 1 percent eligible students in very small schools (enrolment of 15-year-olds less than 18) and more than 4 percent eligible students in moderately small schools (enrolment of 15-year-olds between 18 and 35). Therefore, the overall analysis indicates Atlanta will need to increase the sample size and undersample very small schools. There are no schools in Atlanta’s school frame with 0, 1, or 2 students.
The example in Appendix C shows the worksheet for the overall Small School Analysis, the worksheet for the first explicit stratum, and the same information in the Excel layout of the Small School Analysis sent to NPMs for ST9.
An additional table shows the Small School Analysis columns with calculations for the needed school sample sizes. The base number of schools/students for individual strata Small School Analysis follow from the initial sample allocation (see Appendix B). Note that the SSA is performed separately for each explicit stratum by the procedure outlined in paragraphs 134 and 138 with minor adjustment for the base number of schools and students.