Comparing Means in Two Populations

Comparing Means in Two Populations

Overview

• The previous section discussed hypothesis testing when sampling from a single population (either a single mean or two means from the same population).

• Now we’ll consider how to compare sample means from two populations.

• Towards the end of the course, we’ll discuss comparing means from more than two populations.

• When we’re comparing the means from two independent samples we usually ask: “How does one sample mean compare with the other?”

• However, focusing just on comparing the means can be premature.

• It’s safer to first consider the variability of each sample, the pattern of any outliers, the shape of the distributions.

• Then it may be safe to assume a normal distributions;

but not always.

• So, we’ll also discuss approaches to answering these questions when we’re not comfortable with the

assumption of normality, or when this assumption is just not defendable.

Two Sample Means

• Cities and counties: Returning to the Alabama SOL pass-rates, is there a difference between cities and counties?

• Recall last time we looked at a difference across years in the same population.

• Here, we want to look at the difference in one year between two populations: city high schools and county high schools.

Phase 1: State the Question,

• 1. Evaluate and describe the data

• Begin by looking at the data. Where did the data come from? What are the observed statistics?

• The source of this data is Alabama Department of Education.

• The first step in any data analysis is evaluating and describing the data.

• These first steps are also called preliminary analysis, to distinguish them from the definitive (or outcome)

analysis.

Preliminary Analyses

• The goal of a preliminary analysis is to describe and inform.

• To give a description of the data.

• Keep in mind that, the goal of a definitive analysis is decision making or hypothesis testing.

Preliminary Analysis

• What are the observed statistics?

• Use the Fit Y by X platform to look at a graphical and tabular summary of the data, as in the next figure.

• Note: Previously, in Step 1, we used the

Distribution of Y reports to identify and fix errors and to further understand the data.

• You can use the Distn of Y but you would have to run it twice – here you can see everything in one place.

• There are three components to this figure: The dot plots, the box plots, and the quantiles table.

Let’s look at each.

The Dot plot

• A dot plot shows the continuous Y-variable’s (Algebra I 2000 pass rate) values along the vertical axis and the nominal X-variable’s (City Yes or No) values along the horizontal axis.

• So we see the two groups along the horizontal axis; City

= “No” and City = “Yes”. The width of the groups is

proportional to the sample size of each group; there are more “No” (non-cities) values so it is drawn wider. This follows the “your eye goes to ink” rule.

• Groups with larger samples are more informative than groups with smaller samples so the larger n group is drawn bigger.

Dot plots

• Along the vertical axis, we see the 10th grade, year 2000 Algebra I SOL pass-rate; One dot for each high school.

Values range from 0% passing to 100% passing.

• The horizontal spreading of the values is done so that you can see each school’s scores better (called “Jittered Points”). The amount of horizontal spread is random; so, don’t try to interpret the scores for points farther to the right or left than scores closer to the center (horizontally).

• Of course, the vertical values are interpretable; that is, a school at the top has a higher pass-rate than a school at the bottom.

Box Plots

• These side-by-side box plots describe the shape of the distributions within each group.

• These plots do not assume normality so we use them to begin to answer the question, “Is each group normally distributed?”

• In the box plots we can easily see whether the values are symmetric about the median.

• Look for these “warning flags” that the data is not normal:

Box Plots

• Is the distance between the median and the 75%tile

different than the distance between the median and the 25%tile?

• Is the upper whisker-bar (actually, the 90%tile) more distant from the median than the lower whisker-bar (the 10%tile)?

• Are the high-extreme tail-values more distant from the median than the low-extreme tail-values?

• These informal, graphic assessments don’t raise any warning flags for these data.

• The dotted horizontal line represents the mean value for all schools (not considering the group).

Quantiles Report

• If a more detailed comparison of values is

needed, the numerical values plotted in the box plots are shown in the Quantiles Report.

Quantiles Report

• For instance, in the City = “No” group, the

distance from the median to the 75%tile (~49 vs 65, 16 points) is about the same as the distance between the median and the 25%tile (~49 vs 34, about 15 points).

• But, as in the Distribution platform, the preferred way to answer the question, “Is each group

normally distributed?” is with a normal quantile plot.

Normal Quantile Plots

• Actually, the more proper phrasing of the question is: “Within each group, is each group normally distributed?”

• That is, it may be that if we were to lump both groups together, the data would

appear non-normal.

• We must take group membership into account when making this assessment.

Interpretation

• Follow the same interpretation of the

normal quantile plot as we discussed with a single mean.

– Are each group of black dots along a straight line?

– In the SOL data, these two sets of points follow the lines fairly well, with some

departure in the tails.

Normality

• So, we now have enough information to answer the question, “Within each group, is each group normally distributed?”

– If the answer is “Yes” or “Probably” then we can

proceed with parametric tests to compare the means.

The Central Limit Theorem can apply if the sample size is “large.” The rule of thumb is if the total n is at least 30 (n1 + n2 ≥ 30).

– If the answer to the normality question is “No” or “I doubt it” then we’ll use nonparametric methods to answer our question.

Preliminary analysis, showing means

• If the data is normally distributed then means and SDs make sense.

• If these distributional assumptions are unwarranted, then we should consider nonparametric methods.

• Thus, the next thing to do in our preliminary analysis may be to get rid of the box plots and quantile plot and to show the means and

standard deviations calculated within each group.

Means

• This figure shows the means, here connected with a line, and a short dashed bar that is one standard-error “error bars.”

• The long dashed lines above and below the means are one standard deviation away from their respective mean.

• The means and standard errors and deviations can be shown by selecting Means and Std Dev from the main Options menu. From the Display Options sub-menu in the Options menu, select the options necessary.

Means and SDs Report

• We can use it to describe the following:

– the number of observations in each group, – the means of each group,

– the standard deviation within each group.

• Recall that the SE is not a “descriptive statistic”

for the data, it is used for inference about the mean.

• JMP includes the SE here because it is used to form confidence intervals about the mean.

• Note: You can change the number of decimal places displayed in any JMP report:

– Double-click a number in the report. A dialog will appear.

– Change the number of decimal places.

Summary: Preliminary Analysis

• So far, what have we learned about the data?

• We have not found any errors in the data.

• We’re comfortable with the assumption of normality within each group.

• We’ve obtained descriptive statistics for each of the group we’re comparing.

Preliminary Results

• Also, at this point, we can look at the two means and make a guess, “is there a difference

between cities and counties?”

• City schools seem to be about 12 points below non-city schools, and with SEs < 3, this seems like a “big” difference.

• Recall that the t-statistic is the ratio of the difference to a standard error.

• The ratio of 12 to 3 is bigger than 2.

2. Review assumptions

• As always there are three questions to consider:

• Is the process used in this study likely to yield data that is representative of each of the two populations?

– Yes, it is the population

• Is observation in the two samples independent of the others?

– Yes.

• Is the sample size sufficient?

– Yes, both groups are “large” and we’re comfortable with normality for both groups.

Bottom line

• We have to be comfortable that the first two assumptions are met before we can proceed at all.

• If we’re comfortable with the normality

assumption, then we proceed, as below.

• Later, we’ll discuss what to do when normality can not be safely assumed.

3. State the question—in the form of hypotheses

• Let’s refer to the two groups as “1” and “2” for notational purposes. Using these as subscripts, there are three possible null hypotheses:

1. The null hypothesis is µ1 ≤ µ2, 2. The null hypothesis is µ1 ≥ µ2 , or

3. The null hypothesis is a fixed value, µ1 = µ2.

• And the alternative hypothesis is the opposite of the null.

Phase 2: Decide How to Answer the Question

• 4. Decide on a summary statistic that reflects the question

• Recall the general test statistic:

summary statistic - hypothesized paramter test statistic =

standard error of the summary statistic

summary statistic - hypothesized paramter test statistic =

standard error of the summary statistic

Difference Score

• In this situation (as in comparing paired means), we are going to use the difference score as our summary statistic:

The relevant statistic is y₁ − y₂ or the observed difference of the two means.

• The hypothesized parameter is easy: µ₁− µ₂ = , since under any of the three 0 null hypotheses a difference of 0 would result in “failing to reject” the null

hypothesis.

• What about the standard error? There are two possibilities for the standard error of y₁−y₂. The two possibilities depend upon the two standard deviations within each group.

o Are they the same?

o Or do the two groups have different standard deviations?

Standard Error

Same SD

• If the standard deviations (or variances) within the two populations are equal than the standard error of the difference is easy.

• We just “average” the two estimated standard deviations and obtained a pooled estimate.

• The variance of the mean difference is the sum of the standard errors of each mean

Same SD

2 2

1 2

1 2

n n

σ σ

+

The t-statistic

We’ll use the t-test to compare the two sample means and, using a pooled estimate for the variance called s²_p, we calculate:

1 2

2 2

1 2

p p

y y t

s s

n n

= −

+

Estimating σ

• The pooled variance estimate is a weighted average of the two individual-group variances:

(

)

(

)

2

1 2

1 1

p 2

n s n s

s n n

− + −

= + −

Under the equal variance assumption, we

calculate the p-value using df = n1 + n2 – 2.

Unequal SD

• If the variances are not equal, the calculation is more complicated:

1 2

2 2

1 2

1 2

y y t

s s

n n

′ = −

+

t-prime

• Note that the separate variance estimates are used in this “t prime” statistic, not the pooled estimated for variance.

• Further, the df is not a simple function of just n1 and n2.

• The details of these calculations are not important.

• What we need to know is how to proceed using JMP.

Deciding on the correct t-test

• Which test should we use?

• We may not need to choose; if the two sample sizes are equal (n1 = n2) the two methods give identical results.

• It’s even pretty close if the n’s are slightly different.

• If one n is more than 1.5 times the other (in the SOL case, n1 = 306 and n2 = 93, which is over 3 times as large), you’ll have to decide which t-test to use.

Decision

• Decide whether the standard deviations are different.

– Use the equal variance t-test if they are the same, or

– Use the unequal variance t-test if they are different.

• Or, you could decide not to decide; use the unequal variance t-test. It’s more

conservative.

Determining Equal SDs

• There are three ways to make this decision.

1. Inspect the two standard deviation estimates.

2. Use the normal quantile plot.

3. Test for equal standard deviations.

Inspect the SDs

• Refer to the Means and Std Deviations report.

• Look at the two standard deviations, in this case 22.1 and 19.9.

• Form the ratio of the largest to the smallest.

• If the ratio is larger than about 3, then the two SD’s may be unequal (in our case, the ratio is 3.3).

• For a better answer to the question, see the normal quantile plot.

Normal Quantile Plot, SDs

• If the two standard deviations are equal then the slopes for the two lines in the normal quantile

plot will be the same (the lines will be parallel).

• In our case the lines have roughly the same slope.

• So, for the SOL data the assumption of equal variability seems safe.

Questionable Parallel?

• If the slopes of the two lines are in that “gray area”

between clearly parallel and clearly not parallel, what do we do?

• There are four possibilities:

1. Ignore the problem and be risky: use the equal variance t-test.

2. Ignore the problem and be conservative: use the unequal variance t-test.

3. Make a formal test of unequal variability in the two groups.

4. Compare the means using nonparametric methods.

What if not Parallel?

• Here, the data appear to be reasonably

normal (this isn’t the question) but the lines are not parallel – they start out close and

end far apart.

• Here, not only do the variances appear to be unequal but

normality is also in questions.

• We’ll look at this case later.

Test for equal SDs

• In the case of the first figure, where we’ll be using a t-test, but we’re not sure which one, JMP provides a way to test for equal variance in the main options menu.

Choosing between the tests of equal variance

• Of the five tests: O’Brien’s, the Brown-Forsythe test, Levene’s test, Bartlett’s test, and the F-test;

• the last three are out of date and are not recommended.

• There’s not much difference between O’Brien’s and Brown-Forsythe.

• Brown-Forsythe is more robust (resistant to outlying observations), so we’ll use this result.

• What are we testing?

Variance test

• The null-hypothesis for these tests are,

“the variances are equal.”

• So, if the Prob>F value for the Brown-

Forsythe test is < 0.05, then you will reject the null hypothesis (universal decision

rule) and conclude that the groups have unequal variances.

Reject Equal SDs?

• The report also shows the result for the t-test to compare the two means, allowing the standard deviations to be unequal.

• This is the “unequal variance t-test.” Here is a written summary of the results using this

method:

• “The two groups were compared using an unequal variance t-test and found to be

significantly different (t = 7.1, df = 217.4, p-value

< 0.0001). School districts in cities had lower scores ….”

Unequal test df?

• Notice the degrees of freedom – it isn’t a whole number.

• That is because it is based on a “weighted”

contribution of each sample (with unequal ns) to the standard error estimate.

• You can round the number off but ONLY to one decimal place, do not round to a whole number.

Step 5: Random Variation

• Recall the rough interpretation that t’s

larger than 2 are likely not due to chance.

• For either type of t

6. State a decision rule

• The universal decision rule

• Reject H0: if p-value < a.

Phase 3: Answer the Question

• 7. Calculate the statistic

• There are three possible statistics that may be appropriate:

1. an equal variance t-test,

2. an unequal variance t-test, or

3. the nonparametric Wilcoxon rank-sum test.

Equal variance

• If the equal variance assumption is reasonable, then the standard t-test is appropriate.

• Note: When reporting a t-test it’s assumed that, unless you specify otherwise, it’s the equal-variance t-test.

• The next figure shows the means diamonds in the dot plot, the t-test report, and the means for a oneway

ANOVA (Analysis Of VAriance) report.

• We’ll cover oneway ANOVA later in the course. When there are only two groups, the t-test and ANOVA give identical results.

In JMP

• To compare the two means using an equal variance t- test in JMP:

– Choose Means/Anova/Pooled t from the main options menu.

• This adds the Oneway ANOVA report and means diamonds.

• The t-value, df, and p-value are shown in the t-Test report.

• However, only the two-tailed p-value is reported (under Prob>|t|).

• If the null-hypothesis specified that we were testing for equality, then this is the p-value we want.

One-tail p-values

• First, which group did JMP use for y1 and which for y2? JMP uses the order of the X-variable:

• If the X-variable is character, JMP alphabetically sorts the values and whichever comes first is y1.

• If the X-variable is numeric, JMP uses the smallest value of the X-variable as y1.

• If your alternative was HA: µ₁ > µ₂ and the t-test value is positive, then the one- tailed p-value is half the two-tail Prob>|t| in the report.

• If the t-test value was negative then you’ve observed a difference in the opposite direction from that expected. The p-value is one minus half the Prob>|t| in the report.

• If your alternative was HA: µ₁ < µ₂ and the t-test value is negative, then the one- tailed p-value is half the two-tail Prob>|t| in the report.

• If the t-test value was positive then you’ve observed a difference in the opposite direction from that expected. The p-value is one minus half the Prob>|t| in the report.

Unequal variance

• If the variances are not equal or if we just want a more conservative test then see the bottom

portion of the Tests that the Variances are Equal report.

• The unequal-variances t-test is listed as the Welch Anova.

• Report the t-value, df and p-value, as in the equal variance case.

Nonparametric comparison of the medians

• If normality isn’t reasonable, then you can use a nonparametric test.

• You will use a test that compares the medians between the two groups.

• The nonparametric test is based solely on the ranks of the values of the Y-variable.

• In JMP, choose

– Nonparametric > Wilcoxon test.

Wilcoxon

• The Wilcoxon rank-sum test (also called the Mann-Whitney test) ranks all the Y-values (in both groups) and then compares the sum of the ranks in each group (the groups are specified by the X-variable).

• If the median of the first group is, in fact equal to the median of the second group, then the sum of the ranks should be equal for equal sample

sizes

Reporting Wilcoxon

• When reporting the results of a nonparametric test, it’s usual to only report the p-value.

• In the above report, there are two p-values, one for the z-test and one using a chi-square value.

• The p-values will rarely be different.

• For large samples report the p-value from the normal approximation.

• For smaller samples, use the chi-square

• For really small samples you should probably consult a statistician to help you obtain exact p-values.

Steps 8 & 9

• 8. Make a statistical decision

– Using all three tests, the two groups are different. All p-values are < 0.05.

• 9. State the substantive conclusion

– The schools in cites have significantly lower mean pass rates (35.9% vs 48.8%) and

significantly different median pass rates (36.0% vs 49.2%)

Phase 4: Communicate the Answer to the Question

• 10. Document our understanding with text, tables, or figures

• The year 2000 Alabama SOL pass-rates in 10th grade Algebra I were divided into two groups according to whether the school was a city or county high school.

There were n = 306 schools within city school-districts and n = 92 in county school districts. The observed

average pass rates within city schools was 35.9% (SD = 19.9) and pass rates outside of cities were 48.8% (SD = 22.1). Using a two-tailed t-test, we conclude that the observed means are significantly different (t = 5.0, df = 396, p-value < 0.0001). From this we conclude that city schools have a significantly lower pass rate compared to county schools. The 95% confidence interval about the mean difference is between 7.8% and 17.9%.

Less text, replaced by information in a table

• Alternatively, it may be more

straightforward to include many of the

numbers in a table and state your results in text.

• So instead of the above paragraph, you could do this:

• The summary results for the year 2000 Alabama SOL pass rate percentages in 10th grade algebra I are shown in Table 1. Schools were divided into cities if their

district name contained “City” and were otherwise

classified as a county school district. From these results we conclude that schools in cities had a pass-rate that was significantly lower than the pass-rates compared to county schools. The 95% confidence interval about the mean difference is between 7.8% and 17.9%.

Table 1. 2000 Alabama SOL Pass-Rates in 10^th Grade Algebra I For Schools in Cities and in Counties

Location

Number of Schools

Pass Rate

(SD) SE 95% CI

City 92 35.9 (19.90) 2.23 31.5 – 40.4 County 306 48.8 (22.05) 1.62 51.2 – 46.4

Difference 12.9^* 2.57 7.8 – 17.9

*t = 5.0, df = 396, p-value < 0.0001

Less text, replaced by information in a figure

• As another alternative, it may be more informative to describe the results in a figure.

• Instead of the above, you could do this:

• The summary results for the year 2000 Alabama SOL pass rate percentages in 10th grade

algebra I are shown in Figure 12. Schools were divided into city and county high schools. From these results we conclude that schools in cities had a pass-rate that was significantly lower than the pass-rates compared to county schools(t = 5.0, df = 396, p-value < 0.0001). The 95%

confidence interval about the mean difference is between 7.8% and 17.9%.

0 10 20 30 40 50 60 70 80 90 100

0 10 20 30 40 50 60 70 80 90 100

County High Schools City High Schools

mean = 48.8, SD = 22.05 mean = 35.9, SD = 19.90

Figure 12. 2000 Alabama SOL Pass-Rates in 10^th Grade Algebra I For Schools in Cities and in Counties

Summary: Two Independent Means

• Briefly, here is how to proceed when comparing the means obtained from two independent samples.

• Describe the two groups and the values in each group.

What summary statistics are appropriate? Are there missing values? (why?)

• Assess the normality assumption. If normality is not

warranted, then do a nonparametric test to compare the medians.

• If normality is warranted, then assess the equal variance assumption.

Summary (cont)

• Report confidence intervals on each of the means if normality is reasonable.

• Perform the appropriate statistical test: equal variance t-test, unequal variance t-test, or the Wilcoxon rank-sum test. Determine the p-value that corresponds to your hypothesis.

• Reject or fail to reject? State your substantive conclusion.

Summary (cont)

• Additional note: Say you conclude that the groups have different means. How do you describe what the different means are?

• If you’ve followed the above recipe, you are in one of three situations:

1. Normality is reasonable and the variances are equal 2. Normality is reasonable and the variances are

unequal

3. Normality is not warranted

Summary (cont)

• Normality is reasonable and the variances are equal, use the equal variance t-test.

• The write up reads “… the means are

significantly different (t = x.xx, df =xxx, p-value = 0.xxxx).”

• Also give a table of means, SEs and 95%CIs just like the “Means for Oneway Anova”.

Summary (cont)

• The variances are unequal, use unequal- variance t-test.

• The write up reads “… the means are

significantly different (unequal variance t = x.xx, df =xxx, p-value = 0.xxxx).”

• Also give a table of means, SEs and 95%CIs

just like the “Means and Std. Deviations” report.

• Note: the means are the same.

• The SEs and CIs are different.

Summary (cont)

• Normality is unreasonable, use Wilcoxon’s test.

• The write up reads “… the medians are

significantly different (by Wilcoxon’s signed-rank test, p-value = 0.xxxx).”

• Also give a table of medians and IQRs.

• There is a way to put 95%CIs on these

estimates but not using any easily available software.

Always

• Report a measure of the center and spread.

• For all three tests, report the p-value and make a decision based upon your

hypothesis.

• Your final statement should summarize the results in terms of the experiment (no

statistics),