Home / DOE / Additional Information / Power and Sample Size Calculation
Power and Sample Size Calculation¶
The dialog titled Power and Sample Size for Quantitative Outputs answers one question: with this design and this many replicates, how likely is the experiment to detect an effect of a given size.
Opening it¶
From the replicates page, with the button Help Me Choose Power/Sample size for Quantitative Outputs. The Design Wizard's version of the same button reads Help Me Choose....
The button is not on every design's replicates page. It appears for the two-level factorial, the three-level factorial, the N-level factorial, Plackett-Burman, and the Taguchi L4, L8, L12, L16 and L32 arrays. CCD, Box-Behnken, D-Optimal, Custom and the other Taguchi arrays do not show it at all.
On the Create Design builder it is greyed out unless the design has at least one quantitative output, since the calculation is for quantitative responses only. The Design Wizard's version is never greyed out, because the wizard reaches this step before it asks about outputs.
It opens with the design's current structure already loaded, so the numbers it reports are for the design as it stands.
What it asks for¶
Two numbers, and the dialog explains both on screen:
| Control | The dialog's own description |
|---|---|
| Effect: | The minimum difference in the output the experiment needs to detect. |
| Standard deviation: | Estimate of the population standard deviation for the output. |
On a blocked design only, the dialog also states how it is blocked, in one of two lines: Design is blocked on replicates. or Design is blocked with aliasing (n blocks). On an unblocked design neither line appears. The distinction matters, because blocking on replicates changes what the replicate count buys.
There is no alpha control. The calculation uses a fixed alpha of 0.05, which you cannot change from this dialog or from the Options dialog. See Type I and Type II Error.
What it gives back¶
A grid of replicate counts with the power at each, three columns wide: Replicates, Power, and a third column that marks one row Recommended.
- The grid is filled in for you as soon as you enter an effect and a standard deviation. It holds about twenty replicate counts, spaced so that the range reaches a power of 0.99, and rows past a power of 0.9999 are dropped rather than repeated.
- The Recommended row is the first one whose power reaches 0.90. It is highlighted in green and selected when the grid appears. That 0.90 is fixed in the software, not a setting.
- Power is shown to four decimal places.
- A row can read NA instead of a number. That means the calculation could not produce a power for that replicate count, and the reasons are listed below.
Calculate adds one extra row for a replicate count you type into the box beside it. The count must be a whole number greater than 1 and no more than 10000, and anything else gives Number of replicates must be a number greater than 1.
Use selected replicates closes the dialog and sets the design's replicate count to the row you have selected in the grid, which is the point of opening it from the replicates page. It takes the selected row, not the value in the Calculate box.
It does nothing at all when no row reached 0.90 power. The button is gated on a Recommended row
existing rather than on your selection, so on a grid where every power is below 0.90 it closes nothing and
says nothing, and double-clicking a row behaves the same way. The replicate count can still be typed
straight into the replicates page. This is recorded in code-flags-for-engineering.md as flag 45.
What it computes¶
Power is computed from the noncentral \(F\) distribution. The critical value is the ordinary \(F\) value that cuts off \(\alpha\) in the upper tail, and the power is the probability that a noncentral \(F\) with the same degrees of freedom exceeds it:
Everything specific to the design is in the numerator degrees of freedom \(df_1\) and the noncentrality parameter \(\lambda\).
Two formulas, chosen by the design:
For a two level design the test is on a single effect, so \(df_1 = 1\), and
where \(r\) is the number of replicates, \(k\) the number of runs at the corners, \(\delta\) the effect to be detected, \(\sigma\) the assumed standard deviation, \(N\) the total observations and \(p\) the number of parameters including the constant.
For a general factorial the test is on the factor with the most levels, so \(df_1\) is one less than that count, and the noncentrality uses the number of observations at each of its levels:
where \(L_{\max}\) is the largest number of levels on any factor and \(n_L\) is the number of observations at each level of that factor, replicates included. The error degrees of freedom is supplied rather than derived.
Note the two forms divide by \(4\) and by \(2\) respectively. They are not the same expression rearranged.
Full details: Power and Sample Size.
Why a row reads NA¶
The dialog shows two validation messages. A standard deviation of zero or less gives Standard deviation must be greater than zero., and an effect that is not a number gives Effect must be a numeric value. Everything else that stops the calculation is caught and turned into NA in that row's Power cell, with no message at all.
These are the conditions that produce an NA. Each is a real limit in the calculation, so the NA is telling you the power for that row could not be computed, not that the design is invalid:
| Condition | Applies to |
|---|---|
| the standard deviation is not greater than zero | both formulas |
| fewer than 1 replicate | two-level designs. The formula itself accepts 1; it is the Calculate box that will not take a value below 2 |
| fewer than 2 runs at a corner | two-level designs |
| the observations do not outnumber the parameters | two-level designs |
| the error degrees of freedom are 0 or fewer | general factorial designs |
| the sample size at the factor with the most levels is 2 or fewer | general factorial designs |
| the noncentrality parameter exceeds 1000, which is the case below | both formulas |
The high-power rows are the ones that go NA¶
The noncentrality parameter must lie between \(0\) and \(1000\). A design with a large effect relative to \(\sigma\) and a large number of runs can produce a \(\lambda\) above that ceiling, and the calculation then returns no value rather than a power just short of \(1\).
When \(\lambda\) is smaller than \(10^{-10}\) the ordinary \(F\) distribution is used instead, since the noncentral form adds nothing at that scale.
This one is worth understanding because it is counterintuitive. A large effect relative to the standard deviation, with a decent number of runs, pushes the noncentrality past the ceiling the calculation accepts. The power in that situation is essentially 1, but the cell reads NA.
So a grid whose lower rows hold numbers and whose upper rows read NA is not broken: the NA rows are the ones where the design is most likely to detect the effect. Read the last numeric row as your answer, and treat the NA rows above it as at least that powerful.
There is a note on this in code-flags-for-engineering.md, flag 27, because NA is indistinguishable
from a genuine failure.