Home / Shared Math Details / Control Chart Math / Varying Subgroup Size Limits
Varying Subgroup Size Limits¶
Most control chart formulas assume every subgroup holds the same number of observations. When the sizes differ, the standard deviation behind the limits changes from point to point, and so do some of the center lines. This page defines what changes, what stays fixed, which points are left blank, and what the Force Straight Limits option does.
Notation¶
| Term | Description |
|---|---|
| \(n_i\) | size of subgroup \(i\) (its count of non-missing observations) |
| \(\hat{\sigma}_{\text{within}}\) | the chart-wide within-subgroup sigma estimate |
| \(\sigma_i\) | the standard deviation of the plotted statistic at subgroup \(i\) |
| \(\text{CL}\) | a center line that does not change from point to point |
| \(\text{CL}_i^{R}, \text{CL}_i^{S}\) | per-point center lines of a range panel and a standard deviation panel |
| \(k\) | the sigma multiplier (number of standard deviations), default \(3\) |
| \(\bar{p}, \bar{u}\) | pooled proportion defective and pooled defect rate |
| \(\sigma_Z\) | Laney overdispersion factor |
| \(n_0\) | the assumed constant size supplied with Force Straight Limits |
| \(d_2, d_3, c_4, c_5\) | control chart constants, defined on Control Chart Constants |
Where unequal sizes come from¶
Unequal sizes are not always a deliberate choice. They arise when:
- The subgroup column holds a genuinely variable inspection or sample size, which is the normal case for a p, np, u, or Laney chart.
- Observations are grouped by an identifier column and the groups are not all the same length. The data are padded out to the widest group, and the short groups end up smaller.
- A single column is split into fixed-size subgroups and the row count is not a multiple of that size, leaving a short subgroup at the end.
- A blank cell sits inside an otherwise full subgroup, or a value in it was removed as an outlier. Either one reduces that subgroup's size by one.
The last two cases are worth watching for, because a chart that was intended to have constant subgroups quietly acquires stepped limits at the end or at one point in the middle.
Per-point control limits¶
Every quantity that divides by a subgroup size, or looks a constant up at a subgroup size, becomes a per-point quantity as soon as the sizes differ. On a chart of subgroup means the standard error uses each subgroup's own size \(n_i\):
The band is narrow where a subgroup is large and wide where it is small, so the limits are drawn as a staircase instead of a straight pair of lines, with one step per change of size. Each point is judged against its own limits, and its zone lines are placed from its own limits. The center line does not move: it stays at the grand mean, which is the mean of the individual observations, so larger subgroups already carry more weight in it.
The within sigma itself is a single chart-wide value. It is estimated once from all of the subgroups, with the size differences already accounted for inside the estimator, and it is not re-estimated per point.
Because the reported center, upper, and lower values are single numbers while the drawn limits are a staircase, the values shown in the chart's summary are those of the last plotted point rather than a value that holds across the chart. Read the drawn band, not the summary line, when the sizes vary.
Dispersion panel center lines¶
The two dispersion panels are built the same way from different constants. The range panel first:
On a range panel the center line is a function of the subgroup size, so the center line steps as well as the limits:
The multipliers in parentheses are the \(D_4\) and \(D_3\) factors evaluated at \(n_i\). The lower limit is floored at zero, because a range cannot be negative; the zone lines are placed from the unfloored lower limit and then floored themselves.
A moving range panel uses this same form with the moving range length \(L\) in place of \(n_i\). That length is a single setting rather than a property of each point, so those three lines come out flat.
And the standard deviation panel:
On a standard deviation panel the center line is a function of the subgroup size, so the center line steps as well as the limits:
The multipliers in parentheses are the \(B_4\) and \(B_3\) factors evaluated at \(n_i\). The lower limit is floored at zero, because a standard deviation cannot be negative; the zone lines are placed from the unfloored lower limit and then floored themselves.
This panel has one extra behavior: its center line steps only while unbiased constants are in use. With the within-sigma unbiasing option cleared the center line is the flat \(\hat{\sigma}_{\text{within}}\) at every point, while the limits still step, so the band becomes visibly asymmetric about a straight center line.
Which lines move and which stay flat¶
For the variable charts:
| Panel | Center line | Half width of the band |
|---|---|---|
| Xbar (on an Xbar-R or Xbar-S chart) | flat at the grand mean | \(k\,\hat{\sigma}_{\text{within}}/\sqrt{n_i}\), steps with size |
| Range | \(d_2(n_i)\,\hat{\sigma}_{\text{within}}\), steps with size | multiplier evaluated at \(n_i\), steps with size |
| Standard deviation | \(c_4(n_i)\,\hat{\sigma}_{\text{within}}\) with unbiased constants, otherwise flat | multiplier evaluated at \(n_i\), steps with size |
| Individuals and moving range panels of an I-MR-R/S chart | flat | flat: these panels treat one subgroup mean as one observation, so no size enters |
| EWMA | flat at the process mean | grows point by point; each point's variance term adds a contribution divided by that point's \(n_i\) |
| CUSUM | flat at zero | decision interval \(h\,\hat{\sigma}_{\text{within}}/\sqrt{n_i}\), steps with size |
For the attribute charts, where a varying size is the normal case:
| Chart | Center line | Standard deviation at point \(i\) |
|---|---|---|
| p | flat at \(\bar{p}\) | \(\sqrt{\bar{p}(1 - \bar{p})/n_i}\) |
| np | \(n_i\,\bar{p}\), moves with size | \(\sqrt{n_i\,\bar{p}(1 - \bar{p})}\) |
| c | flat at \(\bar{c}\) | constant: the c chart assumes one fixed inspection size, so nothing varies |
| u | flat at \(\bar{u}\) | \(\sqrt{\bar{u}/n_i}\) |
| Laney P' | flat at \(\bar{p}\) | \(\sqrt{\bar{p}(1 - \bar{p})/n_i}\;\sigma_Z\) |
| Laney U' | flat at \(\bar{u}\) | \(\sqrt{\bar{u}/n_i}\;\sigma_Z\) |
The np chart is the only attribute chart whose center line moves with the size, because it plots a count rather than a proportion or a rate.
Points that cannot be plotted¶
A point is plotted only when its statistic exists:
- A range or a standard deviation needs at least two observations. When \(n_i < 2\) the dispersion point is left unplotted and receives no center line, no limits, and no zones, so the panel shows a break at that position.
- A mean needs at least one observation. A subgroup with no observations at all is unplotted on every panel.
- A subgroup of size one contributes nothing to the within sigma, but it still contributes its value to the grand mean and it still consumes one position of the baseline window used to estimate the limits.
A related restriction applies to the between-subgroup decomposition on an I-MR-R/S chart: it needs one size to be clearly dominant, so the most common subgroup size must account for more than half of the usable subgroups. When the sizes are too mixed for that, the individuals and moving range panels draw without limits. See Process Sigma Estimation.
Force Straight Limits¶
Force Straight Limits replaces the varying size in the limit formula with one assumed size, so the staircase becomes a level pair of lines. It is offered on attribute charts only; the variable charts have no equivalent option.
With Force Straight Limits enabled, a single assumed size \(n_0\) replaces \(n_i\) in the standard deviation used by the limits:
Since \(\sigma_i\) no longer depends on the point, the limits are level. The pooled \(\bar{p}\) or \(\bar{u}\) is still estimated from the real subgroup sizes read from the data, and which points enter that estimate does not change, so the option composes with a baseline window without altering it. When the chart is split into phases, each phase keeps its own \(\bar{p}\) or \(\bar{u}\) while the single assumed size applies to every phase.
One side effect is specific to the np chart: there the assumed size also replaces \(n_i\) in the center line \(n_i\,\bar{p}\), so enabling the option flattens the center line along with the limits. On the u and Laney U' charts the center line is already a flat \(\bar{u}\) and is unaffected.
Four conditions must all hold before the option does anything:
- The chart is an np, u, or Laney U' chart. The control is greyed out on the p, Laney P', and c charts.
- The size comes from a worksheet column. With a single size typed into the dialog there is nothing to straighten, and the control is greyed out.
- The checkbox is ticked.
- An assumed size greater than zero is entered. A blank assumed size is accepted and means the option is off, so a ticked checkbox with a blank size leaves the ordinary stepped limits in place.
Straightened limits are an approximation: a point whose real size is much smaller than the assumed size is judged against limits that are too tight, and a point from a much larger subgroup is judged against limits that are too loose. Use the option when the sizes vary only slightly and the staircase is more distracting than informative.
Used by¶
- Attribute charts: p, np, u, Laney P', and Laney U', for their per-point standard deviation, and the np, u, and Laney U' charts for Force Straight Limits. The c chart is the exception: it assumes a constant inspection size.
- Xbar-R and Xbar-S: the Xbar panel's per-point standard error and the range or standard deviation panel's per-point center line and multipliers. Their math pages are planned.
- I-MR-R/S: the range or standard deviation panel behaves exactly as it does on an Xbar chart, while the first two panels are unaffected by size. Its math page is planned.
- EWMA and CUSUM on subgrouped data: both scale their per-point spread by the subgroup size. Their math pages are planned.
See Also¶
- Control Limits and Zones
- Process Sigma Estimation
- Control Chart Constants
- Moving Ranges
- Shared Math Details
References¶
- Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Wiley.
- Wheeler, D. J., and Chambers, D. S. (1992). Understanding Statistical Process Control (2nd ed.). SPC Press.
- Ryan, T. P. (2011). Statistical Methods for Quality Improvement (3rd ed.). Wiley.
- Duncan, A. J. (1986). Quality Control and Industrial Statistics (5th ed.). Irwin.
- Laney, D. B. (2002). Improved control charts for attributes. Quality Engineering, 14(4), 531-537.