Prospective analysis helps answer the question, “Will I detect the group differences I am looking for given my proposed sample size, estimate of within-group variance, and alpha level?” In a prospective power analysis, an estimates of the group means and sample sizes in a data table and an estimate of the within-group standard deviation (σ) are required in the Power Details dialog.
The Sample Size, Power command in the DOE menu determine how large a sample is needed to be reasonably likely that an experiment or sample will yield a significant result, given that he true effect size is at least a certain size.
The Sample Size, Power platform handles the following cases: ¨ Testing one sample's mean is different from a hypothesized value. ¨ Testing two samples have the same mean
¨ Testing that there are differences in the means among k samples.
The Power and Sample Size facility assumes that there are equal numbers of units in each group.
You can also apply this facility to more general experimental designs, if they are balanced, and a number-of-parameters adjustment is specified.
Chapter 10
Contents
Prospective Power Analysis ... 137 Launch the Sample Size and Power facility ... 137 Single-Sample Mean ... 139 Two-Sample Means ... 141 k-Sample Means ... 142
10 Power
Prospective Power Analysis
The following five values have an important relationship in a statistical test on means: ¨ Alpha is the significance level that prevents declaring a zero effect significant more
than alpha portion of the time.
¨ Error Standard Deviation is the unexplained random variation around the means.
¨ Sample Size is how many experimental units (runs, or samples) are involved in the
experiment.
¨ Power is the probability of declaring a significant result.
¨ Effect Size is how different the means are from each other or from the hypothesized
value.
The Sample Size and Power facility in JMP helps estimate in advance either the sample size needed, power expected, or the effect size expected in the experimental situation where there is a single mean comparison, a two sample comparison, or when comparing k sample means.
The Sample Size, Power
command is on the DOE main menu (or toolbar), or on the
DOE tab page of the JMP Starter. When you launch this facility, the dialog shown here appears with a button
selection for three experimental situations. Each of these selections then displays its own dialog that prompts for estimated parameter values and the desired computation.
Launch the Sample Size and Power Facility
After you click either One Sample Mean, Two Sample Means, or k Sample Means in the initial dialog (shown above), the next dialog asks for the anticipated experimental values. The values you enter depend on your initial choice. As an example, consider the two- sample situation.
The Two Sample Means choice in the initial power dialog always requires values for Alpha and the error standard deviation (Error Std Dev), as shown here, and one or two of the other three values: Difference to detect, Sample Size, and Power. The power facility then calculates the missing item. If there are two unspecified fields, the power facility constructs a plot that shows the relationship between those two values: ¨ power as a function of sample size,
given specific effect size
¨ power as a function of effect size, given a sample size ¨ effect size as a function of sample size, for a given power.
The Sample Size dialog asks for the values depending the first choice of design:
Alpha
is the significance level, usually .05. This implies willingness to accept (if the true difference between groups is zero) that 5% (alpha) of the time a significant difference will be incorrectly declared.
Error Std Deviation
is the true residual error. Even though the true error is not known, the power
calculations are an exercise in probability that calculates what might happen if the true values were as specified.
Extra Params
is only for multi-factor designs. Leave this field zero in simple cases. In a multi-factor balanced design, in addition to fitting the means described in the situation, there are other factors with the extra parameters that can be specified here. For example, in a three-factor two-level design with all three two-factor interactions, the number of extra parameters is five—two parameters for the extra main effects, and three parameters for the interactions. In practice, it isn’t very important what values you enter here unless the experiment is in a range where there is very few degrees of freedom for error.
10 Power
Difference to Detect
is the smallest detectable difference (how small a difference you want to be able to declare statistically significant). For single sample problems this is the difference between the hypothesized value and the true value.
Sample Size
is the total number of observations (runs, experimental units, or samples). Sample size is not the number per group, but the total over all groups. Computed sample size numbers can have fractional values, which you need to adjust to real units. This is usually done by increasing the estimated sample size to the smallest number evenly divisible by the number of groups.
Power
is the probability of getting a statistic that will be declared statistically significant. Bigger power is better, but the cost is higher in sample size. Power is equal to alpha when the specified effect size is zero. You should go for powers of at least .90 or .95 if you can afford it. If an experiment requires considerable effort, plan so that the experimental design has the power to detect a sizable effect, when there is one.
Continue
evaluates at the entered values.
Backup
means go back to the previous dialog.
Single-Sample Mean
Suppose there is a single sample and the goal is to detect a difference of 2 where the error variance is .9, as shown in the left-hand dialog in Figure 10.1 To calculate the power when the sample size is 10, leave Power missing in the dialog and click Continue. The dialog on the right in Figure 10.1 shows the power is calculated to be .99998, rounding to 1.
Figure 10.1 A One-Sample Example
To see a plot of the relationship of power and sample size, leave both Sample Size and
Power missing and click Continue.
Double click on the horizontal axis to get any desired scale. The right-hand graph in
Figure 10.2 shows a range of sample sizes for which the power varies from about 0.2 to
.95. Change the range of the curve by changing the range of the horizontal axis. For example, the plot on the right in Figure 10.2 has the horizontal axis scaled from 1 to 8, which gives a more typical looking power curve.
Figure 10.2 A One-Sample Example
10 Power When only Sample Size, is specified (Figure 10.3) and Difference to Detect and Power
are left blank, a plot of power by difference appears.
Figure 10.3 Plot of Power by Difference to Detect for a Given Sample Size
Two-Sample Means
The dialogs work similarly for two samples; the Difference to Detect is the difference between two means. Suppose the error variance is .9 (as before), the desired detectable difference is 1, and the sample size is 16.
Leave Power blank and click Continue to see the power calculation, 0.5433, as shown in the dialog on the left in Figure 10.4. This is considerably lower than in the single sample because each mean has only half the sample size. The comparison is between two random samples instead of one.
To increase the power requires a larger sample. To find out how large, click Backup on the Power Calculation dialog. Leave Sample Size and Power both blank and examine the plot shown on the right in Figure 10.4. The crosshair tool estimates that a sample size of about 35 is needed to obtain a power of 0.9.
Figure 10.4 Plot of Power by Difference to Detect for a Given Sample Size
k-Sample Means
The k-sample situation can examine up to 10 kinds of means. The next example considers a situation where 4 levels of means are expected to be about 10 to 13, and the Error Std Dev
is 0.9. When a sample size of 16 is entered the power calculation is 0.95.
As before, if you leave both Sample Size and Power are left blank, the power facility produces the power curve shown on the right in Figure 10.5. This confirms that a sample size of 16 looks acceptable.
Notice that the difference in means is 2.236, calculated as square root of the sum of squared deviations from the grand mean. In this case it is the square root of
10 Power
References
References
Atkinson, A. C. and Donev, A. N. Optimum Experimental Designs Clarendon Press, Oxford (1992) p. 148.
Bose, R.C., (1947) "Mathematical Theory of the Symmetrical Factorial Design"
Sankhya: The Indian Journal of Statistics, Vol 8, Part 2, pp. 107-166.
Box, G.E.P. and Meyer, R.D. (1986), “An analysis of Unreplicated Fractional Factorials,”
Technometrics 28, 11–18.
Box, G.E.P. and Draper, N.R. (1987), Empirical Model–Building and Response Surfaces, New York: John Wiley and Sons.
Box, G.E.P. (1988), “Signal–to–Noise Ratio, Performance Criteria, and Transformations,” Technometrics 30, 1–40.
Box, G.E.P., Hunter,W.G., and Hunter, J.S. (1978), Statistics for Experimenters, New York: John Wiley and Sons, Inc.
Byrne, D.M. and Taguchi, G. (1986), ASQC 40th Anniversary Quality Control Congress
Transactions, Milwaukee, WI: American Society of Quality Control, 168–177.
Chen, J., Sun, D.X., and Wu, C.F.J. (1993), “A Catalogue of Two-level and Three-Level Fractional Factorial Designs with Small Runs,” International Statistical Review, 61, 1, p131-145, International Statistical Institute.
Cochran, W.G. and Cox, G.M. (1957), Experimental Designs, Second Edition, New York: John Wiley and Sons.
Cornell, J.A. (1990), Experiments with Mixtures, Second Edition New York: John Wiley & Sons.
Daniel, C. (1959), "Use of Half–normal Plots in Interpreting Factorial Two–level Experiments," Technometrics, 1, 311–314.
Daniel C. and Wood, F. (1980), Fitting Equations to Data, Revised Edition, New York: John Wiley and Sons, Inc.
Derringer, D. and Suich, R. (1980), “Simultaneous Optimization of Several Response Variables”, Journal of Quality Technology, Oct 1980, 12:4, 214–219.
Haaland, P.D. (1989), Experimental Design in Biotechnology, New York: Marcel Dekker, Inc.
Hahn, G. J., Meeker, W.Q., and Feder, P. I., (1976), “The Evaluation and Comparison of Experimental Designs for Fitting Regression Relationships,” Journal of Quality
Technology, Vol. 8, #3, pp. 140-157.
John, P.W.M. (1972), Statistical Design and Analysis of Experiments, New York: Macmillan Publishing Company, Inc.
Johnson, M.E. and Nachtsheim, C.J. (1983), “Some Guidelines for Constructing Exact D–Optimal Designs on Convex Design Spaces,” Technometrics 25, 271–277. Jones, Bradley (1991), “An Interactive Graph For Exploring Multidimensional Respnse
Surfaces,” 1991 Joint Statistical Meetings, Atlanta, Georgia
Khuri, A.I. and Cornell J.A. (1987) Response Surfaces: Design and Analysis, New York: Marcel Dekker.
Lenth, R.V. (1989), "Quick and Easy Analysis of Unreplicated Fractional Factorials,"
Technometrics, 31, 469–473.
Mahalanobis, P.C. (1947), "Sankhya," The Indian Journal of Statistics, Vol 8, Part 2, April.
Myers, R.H. (1976) Response Surface Methodology, Boston: Allyn and Bacon. Meyers, R.H. (1988), Response Surface Methodology, Virginia Polytechnic and State
University.
Meyer, R.K. and and Nachtsheim, C.J. (1995), The Coordinate Exhange Algorithm for Constructing Exact Optimal Designs,” Technometrics , Vol 37, pp. 60-69. Meyer, R.D., Steinberg, D.M., and Box, G.(1996), Follow-up Designs to Resolve
Confounding in Multifactor Experiments, Technometrics , Vol. 38, #4, p307. Mitchell, T.J. (1974), “An algorithm for the Construction of D-Optimal Experimental
Designs,” Technometrics , 16:2, pp.203-210.
Piepel, G.F. (1988), "Programs for Generating Extreme Vertices and Centroids of Linearly Constrained Experimental Regions," Journal of Quality Technology 20:2, 125-139.
Plackett, R.L. and Burman, J.P. (1947), “The Design of Optimum Multifactorial Experiments,” Biometrika, 33, 305–325.
Sheffe, H. (1958) Experiments with Mixtures, JRSS B 20, 344-360.
Snee, R.D. and Marquardt, D.W. (1974), “Extreme Vertices Designs for Linear Mixture Models,” Technometrics, 16, 391–408.
Snee, R.D., and Marquardt D. (1975), "Extreme vertices designs for linear mixture models", Technometrics 16 399-408.
Snee, R.D. (1975), “Experimental Designs for Quadratic Models in Constrained Mixture Spaces,” Technometrics, 17:2, 149–159.
Snee, R.D. (1979), “Experimental Designs for Mixture Systems with Multicomponent Constraints,” Commun. Statistics, A8(4), 303–326.
References Snee, Ronald D. (1985)Computer Aided Design of Experiments - Some Practical
Experiences, Journal of Quality Technology , Vol 17. No. 4 October 1985 p.231.
Snee, R.D. and Marquardt, D.W. (1974), “Extreme Vertices Designs for Linear Mixture Models,” Technometrics, 16, 391–408.
Snee, R.D. and Marquardt D.W. (1975), “Extreme vertices designs for linear mixture models," Technometrics 16 399-408.
St John, R.C. and Draper, N.R. (1975), “D-Optimality for Regression Designs: A Review,” Technometrics, 17 pp 15-23.
Taguchi, G. (1976), “An Introduction to Quality Control”, Nagoya, Japan: Central Japan Qualiy Control Association.
Indes
Index
A
ABCD, mixture design 105 actual-by-predicted plot 68
add center points, augment design 125 aliasing of effects 60, 63
analysis example
augmented design 130-134 mixture design 116
response surface design 78-84 screening design 67-68 augment design 121-134
add center points 121, 125 analysis example 130-134 block factor 126 D-optimal 121, 126 data table 125, 129 foldover design 121, 125 interface 123, 126-128
Model Specification dialog 130 random number seed 129 replicate design 121, 124 stepwise regression 130 axial points, RSM 69, 73 axial scaling options 73 backup button 61
B
Big Class.jmp sample data 43 BounceData.jmp sample data 76 BounceFactor.jmp sample data 76 BounceResponse.jmp sample data 76 Box-Behnken, RSM 69, 71, 76-78 Box-Cox transformation 67
Byrne Taguchi Data.jmp sample data 99 Byrne Taguchi Factors.jmp sample data 100
C
canonical curvature, RSM 80 center points 63, 69, 71
central composite design, RSM 69, 74 coded design 60, 61
column property (data table) 14 constraints, loading and saving 14 contour profiler
response su rface design 83 screening design 67 covariate factors 43 cube plot 67
cubic model, custom design 26 Cubic Model.jsl sample script 27 custom design 17, 33-51
all two-factor interactions 31 all two-factor interactions involving
only one factor 30 cubic model 26 data table 22
design generation panel 20-21 dialog 19-23
factor constraints 48 factors, defining 19
fixed covariate factors 43-45 flexible block sizes 36-38 internal details 32 JSL scripting example 27 main effects only 28
mixture with nonmixture factors 47 model panel 20
modify design interactively 23 number of runs 21
prediction variance profiler 24 quadratic model 24
random number seed 15
RSM with categorical factors 38-42 screening design examples 28-31
D D-Optimal augmentation 126-129 data table 11, 13 augmented design 125, 129 custom design 22 design role 14
extreme vertices mixture design 112 full factorial design 90
pattern variable 63 replicates 63
response surface design 72 run script command 75 screening design 61, 63
simplex centroid mixture design 110 simplex lattice mixture design 111 simulated response 63
table property 75 Taguchi arrays 101
variable constraint state (DOE) 14 design choices
mixture design 107 response surface design 77 screening designs 10, 60 Taguchi arrays 101 design output options 60-61
desirability trace, prediction variance profiler 81
Diamond Constraints.jmp sample data 48 DOE Example 1.jmp sample data 11 DOE main menu 3-6
Augment Design 5, 121-134 Custom Design 4, 17-32, 33-51 Full Factorial Design 5, 85-95
Mixture Design 5, 105-120 Response Surface Design 4, 69-84 Sample Size, Power 6, 135-143 Screening Design 4, 53-68 Taguchi Arrays 5, 97-104
Donev Mixture Factors.jmp sample data 45
E
effect sparcity 53, 56
extreme vertices mixture design 105, 112
F
factors 13
constraints 48-51 entering into dialog 9 generators 61 profiling 67
saving and loading 13 factors panel
custom design 19 screening design 58 Taguchi design 100
foldover design, augment design 125 full factorial design 85, 93-95
5-factor example 88 analysis example 91 data table 90 dialog 87
load responses and factors 88 prediction variance profiler 94 sample size 85
stepwise regression 91
I
inner array, Taguchi arrays 97 interaction plot 67
J
Index
L
L18, L36 screening designs 57 loading constraints 15
loading factors and responses 66, 76, 88
M
Main menu, DOE 3 mixture design 105-120 analysis example 116 constrained factors 113-115 data table 110, 111 design choices 107 dialog 107 extreme vertices 105, 112-113 factor constraints 51
prediction variance profiler 119 response surface reports 117 simplex centroid 105, 107, 109-110 simplex lattice 105, 110
ternary plot 115, 120 Model Specification dialog 11, 65
augmented design 130 full factorial design 93 response surface model 75 stepwise regression 132 Taguchi arrays 103
N
non-estimable effect 56
O
orthogonal axial scaling 73 orthogonal design 55, 57, 71 outer array, Taguchi arrays 97
P
pattern variable 63, 68, 74 Plasticizer.jmp sample data 118 power analysis 135-143
alpha 137, 138
difference to detect 139
effect size 137
error standard deviation 137 extra parms 138
k-sample means 142 plotting 140, 142, 143 power 137, 138, 139 single sample 139
standard error deviation 137 two-sample means 141 prediction variance profiler augmented design 134 custom design 24 desirability function 94 desirability trace 81 full factorial design 94 mixture model analysis 47 prediction trace 81
response surface design 81-82 Taguchi arrays 103
prospective power analysis 6, 135-143
Q
quadratic model, custom design 24
R
random number seed 15, 129 Reactor 32 Runs.jmp, sample data 88 Reactor 8 Runs.jmp sample data 67, 123,
126
Reactor Augment Data.jmp sample data 130 Reactor Factors.jmp sample data 88 Reactor Response.jmp sample data 88 replicate design, augment design 124 replicates 60, 63
resolution 56
response surface design 69-84 3-d geormentric view 78 analysis example 78-84 analysis reports 79 axial points 69
axial scaling 72 Box-Behnken 69, 71, 76-78 canonical curvature 80 categorical factors 38-42 central composite 69, 74 contour profiler 83 data table 72 design choices 77 dialog 71 factor constraints 48
load responses and factors 76 Model Specification dialog orthogonal 71
pattern variable 74 plotting 82-84
prediction variance profiler 81 run script command 75 simulate response 74 solution 80
star points 69 uniform precision 71
response surface reports, mixture design 117 responses 8-12, 13
entering into dialog 8 saving and loading 13 simulate 15
rotatable axial scaling 73 run order 60 S sample data Big Class 43 BounceData 76 BounceFactor 76 BounceResponse 76 Byrne Taguchi Data 99 Cubic Model.jsl 27 Diamond Constraints 48 DOE Example 1 11
Donev Mixture factors 45 Plasticizer 118
Reactor 32 Runs 88
Reactor 8 Runs 67, 123, 126 Reactor Augment Data 130 Reactor Factors 88 Reactor Response 88
Sample Size, Power command 135-143 sample size, prospective 6
saving constraints 15
saving factors and responses 66 scaled estimates report 68 screening design 53-68 aliasing of effects 62 analysis example 67 center points 63 coded design 61 Cotter Design 57 data table 64 design choices 60 dialog 7, 58 example 58-65 factor generators 61 factors panel 58 L18, L36 mixed-level designs 57 loading factors and responses 66 mixed-level designs 57
Model Specification dialog 65 non-estimable effect 56 orthogonal 55 output options 60, 63 Plackett-Burman design 56 replicates 63 resolution 56 response panel 58
saving factors and responses 66 simulate response 61, 63 two-level fractional factorial 55
Index two-level full factorial 55
types 55-57
signal-to-noise ratio, Taguchi arrays 97 signal-to-noise ratio, Taguchiarrays 99 simplex centroid, mixture design 105, 109 simplex lattice, mixture design 105, 111 simulate responses 15, 61, 63, 74 single sample power analysis 139 star points, RSM 69
stepwise regression augmented design 130 full factorial design 91
T Taguchi arrays 97-104 contour profiler 103 data table 101 design choices 101 desirability function 103 example 99-102 inner array 97 outer array 97 signal-to-noise ratio 97-99 ternary plot, mixture design 115, 120
U
uniform precision, RSM 71 utility functions 12
free technical support for one year. As a registered JMP user, you can report a problem by calling 919-677-8008 between 9 a.m. and 5 p.m. Eastern Time Monday through Friday, by email at: [email protected], or by submitting a form via the Technical Support link from the JMP web site at: www.JMPdiscovery.com.
To register your software via the web, simply connect to www.JMPdiscovery.com and follow the necessary instructions given on the Registration link. As an alternative, you can complete the JMP registration card located with the software and either mail or fax it to our Registration Database Coordinator.
When you want to learn more...
Statistical discovery is not just about having good tools, but knowing how to use them. SAS Institute's Training Services can guide you through the unique capabilities of JMP
Statistical Discovery Software either on-site or at one of the many training facilities located across the United States.
Learn to master the tools of experimentation, statistical modeling, and graphic visualization to move effortlessly through hundreds of angles of your data. Information on course descriptions, locations, dates, and training options are available in the JMP Training Catalog, which can be obtained by calling 1-919-677-8000, ext. 7321 or sending mail to:
Reading about JMP...
As a supplement to the printed versions, all JMP documentation is included on the software CD as Portable Document Format (PDF) files, which may be viewed and printed using Adobe® Acrobat® Reader. With Version 4, JMP users are now able to quickly locate the answers they need using this online method.
In addition to the set of publications included with the software, JMP Statistical Discovery Software users can order course notes and case studies to supplement their knowledge of data analysis techniques. JMP® Start Statistics, a book that focuses on using JMP to learn about statistics, is also available and serves both the novice as well as the expert. A
complete list of JMP documentation, course notes, and case studies are available in the SAS Institute Publications Catalog, which can be obtained by calling 1-800-727-3228 or sending mail to: [email protected].
For more information on these and other services, please refer to the JMP web site located at: www.JMPdiscovery.com or send email to: [email protected].