Chapter 4 Testing non-random assignment:
4.4 Evidence regarding non-random assignment
4.4.2 Statistical testing of Non-random assignment
Even though graphical evidence supports the random assignment assumption of pupil to classrooms in the Chilean school system, we test statistically how con- sistent this hypothesis is for our two selected groups of schools. Similar to the graphical approachers, we make comparisons between real classes and the two type of counterfactuals based on the previous year school marks. If there is non- random assignment of student to classes, there should not be statistical differences between real classes and random (RDM) counterfactual classes, while we expect to find statistical difference between real classes and perfectly sorted (SRT) coun- terfactuals.
The comparisons between real classes and their counterfactuals are carried out with two statistical tests: (i) T-test, for measuring mean difference between real classrooms and counterfactuals; and (ii)Kolgomorov-Smirnov (KS) test, for assessing statistical difference between the distributions.5 We test for “Sorting
Evidence” (SoE) within school based on each possible comparison, given: the type of counterfactual, the type of statistical test, and the previous school performance measures which are used to create the RDM and SRT counterfactuals. In total, we have 12 independent non-random assignment measures per schools, represented by the Non-Random Assignment (Non-RA) Indexes, from (1) to (12), as it is described in Table 4.16.
Depending on the SoE observed in each school for every type of comparison,
the Non-RA) Index classifies schools into five levels of non-random assignment:
None, Low, Medium, Med-High, and High. Therefore, we sum the number of
schools observed in each category, and analyse their distribution for both groups of school.
We compute the SoE to construct the Non-RA) Indexes for all schools in
Group 1and Group 2. Table 4.17 shows how the SoE is estimated for the Non- RA) Index (1) to (6) in Group 1. Here, the type of counterfactual class used the RDM, which is tested either under t-test or ks-test. For every Index, the maximum possible SoE is 8, meaning there is evidence of non-random assignment in the eight possible comparisons for this group of schools. Similarly, for Group 2, we present in Table 4.18 the maximum SoE that can be reached in this group
5The Kolmorov-Smirnov test is considered as the most appropriate test for comparing dis-
tributions (Gibbons and Chakraborti(2011)). The criterion comparison is stricter than a t-test as we are now comparing the whole distribution instead of just the mean.
when we compare real classes with RDM counterfactuals. Here, we have four more possible comparisons, therefore the maximum SoE score is12.
Table 4.16: Non-random assignment (Non-RA) Indexes per school
Type%of% Counterfactual% Class Type%of%Statistic% Test Performance% measures %Non6RA%Indexes% GPA Non$RA'Index'(1) Language Non$RA'Index'(2) Maths Non$RA'Index'(3) GPA Non$RA'Index'(4) Language Non$RA'Index'(5) Maths Non$RA'Index'(6) GPA Non$RA'Index'(7) Language Non$RA'Index'(8) Maths Non$RA'Index'(9) GPA Non$RA'Index'(10) Language Non$RA'Index'(11) Maths Non$RA'Index'(12) Random2(RDM) T7test KS7test Perfectly2Sorted2 (SRT) T7test KS7test
Note: (i)In total, we have 12 independent Non-Random Assignment measures (from 12 Non-RA Indexes) per group of schools (Group 1, Group 2)(ii)There are two categories of artificially created counterfactual classes:
Random (RDM) and Perfectly Sorted counterfactual (SRT).(iii)SRT counterfactual can be created
based on GPA, Language, or Maths school Marks. (iv)To compare real classes with counterfactual classes we apply two statistics test: T-test and KS-test.
Table 4.17: Potential cases of non-random assignment tested with RDM counterfactuals
Group 1 Real%Class Counterfactual%Class Hypothesis%test
Sorting% Evidence%
(SoE) Condition
4A 4A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B 4A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5A 5A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B 5A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
Maximum%SoE%Score% 8
Non$RA'Indexes'(1)'$'(6)
Notes: (i)For schools in Group 1, the potential maximum evidence of non-random assignment is8.(ii)Every real class within school is compared with the two random counterfactual classes per grade(A,B RDM).(iii) We use two statistical tests: T-test to compare means, and KS-test to compare distributions. (iv)We claim there issorting evidence (SoE)in a particular comparison when the Null Hypothesis (Ho) of No differences between the classes is rejected at a 5% significance level (for both t-test and ks-test).(v)In the Hypothesis test column the Ho in brackets refers to the ks-test.
Table 4.18: Potential cases of non-random assignment tested with RDM counterfactuals
Group 2
Real%Class Counterfactual%Class Hypothesis%test
Sorting% Evidence%
(SoE)
Condition
4A 4A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B 4A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4C 4A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
4B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5A 5A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B 5A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5C 5A#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
5B#RDM Ho:*Mean*difference*=*0 1 If%t=tests%(ks=test)%rejects%Ho
(Ho:*Diff.*in*distribution*=*0) 0 If*t#tests*(ks#test)*does*not*reject*Ho
Maximum%SoE%Score% 12
Non$RA'Indexes'(1)'$'(6)
Notes: (i)For schools in Group 1, the potential maximum evidence of non-random assignment is12.(ii)Every real class within school is compared with the two random counterfactual classes per grade(A,B RDM).(iii) We use two statistical tests: T-test to compare means, and KS-test to compare distributions. (iv)We claim there issorting evidence (SoE)in a particular comparison when the Null Hypothesis (Ho) of No differences between the classes is rejected at a 5% significance level (for both t-test and ks-test).(v)In the Hypothesis test column the Ho in brackets refers to the ks-test.
The hypothesis testing suggests there is SoE when the hypothesis null (Ho) of mean difference between real and counterfactual classes equals to zero is rejected at a 5% significance level, when we apply the t-test. Similarly, it also suggests there is SoE when the Ho of difference in distributions between classes equals to zero is rejected at a 5% significance level, when using the ks-test.
The second round of comparisons, which corresponds to the construction of
the Non-RA) Indexes (7) to (12), uses as counterfactual classes the SRT classes.
The SRT counterfactuals are defined as High Performance (HP) and Low Perfor- mance (LP) based on the previous school marks. Similar to the comparisons shown above for the RDM counterfactuals, in this case we also separate the estimation of SoE and the construction Non-RA) Index by group of schools.
In Table 4.19, we present all possible cases to test for SoE in Group 1
based on the comparisons between real classes and the SRT counterfactuals. The difference with respect to the RDM comparison (Table 4.17) is with respect to the hypothesis test, as in this case we account for SoE when we DO NOT reject the null hypothesis of mean differences equal to zero or distribution differences equal
to zero, when we test with t-test and ks-test, respectively.
Table 4.19: Potential cases of non-random assignment tested with SRT counterfactuals
Group 1
Real%Class Counterfactual%Class Hypothesis%test
Sorting% Evidence%
(SoE) Condition
4A 4A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho 4B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4B 4A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho 4B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5A 5A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho 5B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5B 5A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho 5B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
Maximum%SoE%Score% 8
Non$RA'Indexes'(7)'$'(12)
Notes: (i)For schools in Group 1, the potential maximum evidence of non-random assignment is8.(ii)Every real class within school is compared with the two perfectly sorted counterfactual classes per grade(A,B SRT). (iii)We use two statistical tests: T-test to compare means, and KS-test to compare distributions. (iv) We claim there issorting evidence (SoE)in a particular comparison when the Null Hypothesis (Ho) of No differences between the classes is NOT rejected at a 5% significance level (for both t-test and ks-test).(v)In the Hypothesis test column the Ho in brackets refers to the ks-test.
Table 4.20 shows all the combinations to compare real classes with SRT counterfactuals inGroup 2. As we described earlier for the RDM comparison in Table 4.18, in this group of school the maximum SoE score which can be reached per schools is12. All testing made for theNon-RAIndexes (7) to (12) are looking for NON rejection of the Ho which states non mean difference when we apply the
Table 4.20: Potential cases of non-random assignment tested with SRT counterfactuals
Group 2
Real%Class Counterfactual%Class Hypothesis%test
Sorting% Evidence%
(SoE)
Condition
4A 4A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4B 4A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4C 4A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
4B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5A 5A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5B 5A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5C 5A#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho
(Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
5B#SRT Ho:*Mean*difference*=*0 0 If*t#tests*(ks#test)*rejects*Ho (Ho:*Diff.*in*distribution*=*0) 1 If%t=tests%(ks=test)%does%not%reject%Ho
Maximum%SoE%Score% 12
Non$RA'Indexes'(7)'$'(12)
Notes: (i)For schools in Group 1, the potential maximum evidence of non-random assignment is12.(ii)Every real class within school is compared with the two perfectly sorted counterfactual classes per grade(A,B SRT). (iii)We use two statistical tests: T-test to compare means, and KS-test to compare distributions. (iv) We claim there issorting evidence (SoE)in a particular comparison when the Null Hypothesis (Ho) of No differences between the classes is NOT rejected at a 5% significance level (for both t-test and ks-test).(v)In the Hypothesis test column the Ho in brackets refers to the ks-test.
Aggregating all (Non-RA) Indexes at school group level, we are able to count how many schools are classified in each category in every group. Depending of the distribution of schools among the categories, we could suggest whether there
is Low, Medium or High evidence of non-random assignment of pupil to classes
within schools.
The Non-RA Indexes (from (1) to (12)) take the following correspondence
functional form when the comparisons between real classes and counterfactual classes are run in a school s from Group 1 (G1). Schools from this group can have a maximum score of 8 in each Non-RA Index, as it is total possible combinations to compare between real classes and the counterfactuals.
N on−RAs(G1) = None if SoEs(G1) = 0 Low if SoEs(G1) ={1,2} Medium if SoEs(G1) ={3,4} Med-High if SoEs(G1) ={5,6} High if SoEs(G1) ={7,8} (4.1)
The following functional form corresponds when the comparisons are ap- plied in a school s from Group 2 (G2), where the maximum SoE score is 12
given the possible combinations to compare real and counterfactual classes.
N on−RAs(G2) = None if SoEs(G2) = 0 Low if SoEs(G2) ={1,2,3} Medium if SoEs(G2) ={4,5,6} Med-High if SoEs(G2) ={7,8,9} High if SoEs(G2) ={10,11,12} (4.2)
Both groups of school require that none of the comparisons provides SoE in order to be classified as a school with None evidence of non-random assignment. If schools from Group 1 present 1 or 2 cases of SoE, they are classify as schools withLow evidence of non-random assignment, while schools fromGroup 2require from 1 to 3 cases to be in the same group. The same logic is used to identify schools
from Medium to High levels of SoE. Therefore, we produce 5 different categories
of school regarding the non-random assignment evidence of pupil to classrooms. In the following subsection, we present the results of the means comparison
t-test, and the non-parametric estimate of the difference in distributions ks-test. We aggregate the results at the school group level.