Home / Shared Math Details / Control Chart Math / Process Sigma Estimation

Process Sigma Estimation

This page defines the standard deviation estimators Quantum XL uses for process data: the overall sigma, the within-subgroup sigma (subgrouped data), the within sigma for individual observations, the standard deviation of the subgroup means, and the between-subgroup decomposition. Capability analysis and the variable control charts share these estimators. The dialog group box that selects among them is titled Estimate Sigma.

Notation

Term	Description
\(x_i\)	an individual observation
\(N\)	number of non-missing observations inside the baseline window
\(\bar{x}\)	mean of those observations
\(n_{\text{sub}}\)	number of subgroups inside the baseline window with at least two observations. Written \(n_{\text{sub},1}\) in the Satterthwaite combination below, where subgroups of a single observation are counted as well
\(n_i\)	size of subgroup \(i\) (non-missing values)
\(\bar{x}_i\)	mean of subgroup \(i\)
\(R_i\)	range (maximum minus minimum) of subgroup \(i\)
\(s_i\)	sample standard deviation of subgroup \(i\) (\(n_i - 1\) denominator)
\(L\)	moving range length (default 2, allowed 2 to 100)
\(MR_t\)	moving range: maximum minus minimum of \(L\) consecutive observations
\(MR_{\bar{x},j}\)	moving range of \(L\) consecutive subgroup means
\(m\)	number of moving ranges (or successive differences) inside the baseline window
\(n^{*}\)	modal (most common) subgroup size
\(d_2, d_3, d_4, c_4, c_4'\)	unbiasing constants, defined on Control Chart Constants
\(\hat{\sigma}\)	an estimated process standard deviation
\(\hat{\sigma}_{\bar{x}}\)	estimated standard deviation of the subgroup means
\(\nu\)	degrees of freedom of a sigma estimate
\(k\)	sigma multiplier, the dialog's Number of standard deviations (k):, default \(3\). It is never a count of subgroups on this page

The baseline window

Every sum, count, and mean on this page runs over the baseline window, not over the whole column. The window is set by the Estimate Limits From option:

• All subgroups (the default) makes the window the entire data set, and then \(N\), \(n_{\text{sub}}\), and \(m\) are the totals.
• First N subgroups restricts the window to the first part of the data, and then \(N\), \(n_{\text{sub}}\), and \(m\) count only what falls inside it. On a subgrouped chart the number you enter is a count of subgroups; on an individuals chart it is a count of data points. Fewer than five is refused, and it cannot be combined with a capability (Cpk) request.

Observations you exclude as outliers are treated as missing, so they drop out of these counts as well. A completely empty subgroup is skipped and does not consume a place in the window.

Full details: Baseline and Limit Estimation.

Overall sigma

\[ \hat{\sigma}_{\text{overall}} = \sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

\(N\) is the number of non-missing observations inside the baseline window and \(\bar{x}\) their mean. When Use unbiased constants (overall sigma) is checked, the estimate is divided by \(c_4(N)\):

\[ \hat{\sigma}_{\text{overall}}^{\,\text{unbiased}} = \frac{1}{c_4(N)}\sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

Used by: the overall (Ppk) capability block and the fitted overall PDF curve; the Levey-Jennings control limits, which are built from this sigma and no other.

Within sigma, subgrouped data

The estimator is selected in the dialog (Rbar, Sbar, or pooled standard deviation), and not every chart offers all three; see Which methods each chart offers. Subgroups with fewer than two observations contribute nothing to the within sigma.

Rbar (average range)

\[ \hat{\sigma}_{\text{within}} = \frac{\sum_{i=1}^{n_{\text{sub}}} f_i\, \dfrac{R_i}{d_2(n_i)}}{\sum_{i=1}^{n_{\text{sub}}} f_i}, \qquad f_i = \left(\frac{d_2(n_i)}{d_3(n_i)}\right)^{2} \]

\(R_i\) is the range of subgroup \(i\), \(n_i\) its size, and \(n_{\text{sub}}\) the number of subgroups with at least two observations. Each subgroup's unbiased range estimate \(R_i/d_2(n_i)\) is combined with the minimum-variance weights \(f_i\) (subgroups whose range estimate is more precise get more weight). The unbiased-constants setting has no effect on this estimator: \(d_2\) is always applied.

This is not the textbook \(\overline{R}/d_2\) estimator. When every subgroup has the same size \(n\) all of the weights are equal and the expression collapses to \(\overline{R}/d_2(n)\), the familiar average range over \(d_2\). When the subgroup sizes differ it does not: each subgroup is weighted by the precision of its own range, so a hand calculation of \(\overline{R}/d_2\) from a mixed-size data set will not reproduce the reported value.

Sbar (average standard deviation)

With Use unbiased constants (within sigma) checked (the default):

\[ \hat{\sigma}_{\text{within}} = \frac{\sum_{i=1}^{n_{\text{sub}}} h_i\, \dfrac{s_i}{c_4(n_i)}}{\sum_{i=1}^{n_{\text{sub}}} h_i}, \qquad h_i = \frac{c_4(n_i)^2}{1 - c_4(n_i)^2} \]

\(s_i\) is the sample standard deviation of subgroup \(i\) (with the \(n_i - 1\) denominator) and \(h_i\) the minimum-variance weight. With the unbiased-constants box unchecked, the estimate is the plain average of the subgroup standard deviations:

\[ \hat{\sigma}_{\text{within}} = \frac{1}{n_{\text{sub}}}\sum_{i=1}^{n_{\text{sub}}} s_i \]

The unchecked form applies no \(c_4\) at all and weights every subgroup equally, so with unequal subgroup sizes a subgroup of two observations counts as much as a subgroup of twenty five.

Pooled standard deviation

\[ \hat{\sigma}_{\text{within}} = \sqrt{\frac{\sum_{i=1}^{n_{\text{sub}}} (n_i - 1)\, s_i^2}{\sum_{i=1}^{n_{\text{sub}}} (n_i - 1)}} \]

When Use unbiased constants (within sigma) is checked (the default), the pooled estimate is divided by \(c_4(\nu + 1)\), where \(\nu = \sum_i (n_i - 1)\) is the pooled degrees of freedom:

\[ \hat{\sigma}_{\text{within}}^{\,\text{unbiased}} = \frac{1}{c_4(\nu + 1)}\sqrt{\frac{\sum_{i} (n_i - 1)\, s_i^2}{\nu}} \]

Within sigma, individual observations

The moving ranges these methods use are built from a window of \(L\) consecutive non-missing observations, where \(L\) is the Moving range length in the dialog. Full details of that construction, and of the other moving range series in the product: Moving Ranges.

Average moving range

\[ \hat{\sigma}_{\text{within}} = \frac{\overline{MR}}{d_2(L)}, \qquad \overline{MR} = \frac{1}{m}\sum_{t} MR_t \]

Each moving range \(MR_t\) is the maximum minus the minimum of \(L\) consecutive non-missing observations (\(L\) is the moving range length, default \(2\), in which case \(MR_t = |x_t - x_{t-1}|\)), and \(m\) is the number of moving ranges. A missing value breaks the chain of consecutive observations, so no moving range spans a gap.

When Use Nelson moving-range correction is checked, moving ranges larger than \(D_4(L)\,\overline{MR}\), with \(D_4(L) = 1 + 3\,d_3(L)/d_2(L)\), are removed and the average is recomputed. This repeats until no moving range exceeds the recomputed threshold (at most 50 iterations); a moving range removed in one pass stays removed in every later pass. The multiplier in that threshold is always \(3\), whatever sigma multiplier the chart is drawn with. The correction affects only the sigma estimate, not the moving range values themselves, and it is offered only with the average moving range method. If it removes every moving range, no within sigma can be computed.

Median moving range

\[ \hat{\sigma}_{\text{within}} = \frac{\widetilde{MR}}{d_4(L)} \]

\(\widetilde{MR}\) is the median of the moving ranges (the average of the two middle values when their count is even), and \(d_4(L)\) is the unbiasing constant for the median range of \(L\) standard normal observations. The moving ranges are formed exactly as in the average moving range method.

The moving range chart's center line is not the median you estimated from. That panel centers on \(d_2(L)\,\hat{\sigma}_{\text{within}}\), which equals \(\overline{MR}\) under the average method but equals \(d_2(L)\,\widetilde{MR}/d_4(L)\) under the median method, a slightly different number from the plotted median.

Square root of MSSD

\[ \hat{\sigma}_{\text{within}} = \sqrt{\frac{\sum_{i=1}^{m} (x_{i+1} - x_i)^2}{2\,m}} \]

The sum runs over the \(m\) successive differences of consecutive non-missing observations (a missing value breaks the chain). This is the square root of half the mean squared successive difference (MSSD). When Use unbiased constants (within sigma) is checked (the default), the estimate is divided by the tabulated correction factor \(c_4'(m + 1)\).

This method ignores the moving range length entirely, and the dialog hides that box when it is selected. It is offered only on the Individuals variant of I-MR, and selecting it collapses the chart to a single Individuals panel with no moving range chart.

Sigma of the subgroup means

This quantity is not only an intermediate step toward the between-subgroup sigma. On an I-MR-R/S chart it is what the Individuals panel is drawn from: the limits on that panel are the grand mean plus and minus \(k\,\hat{\sigma}_{\bar{x}}\), and the moving range panel below it centers on \(d_2(L)\,\hat{\sigma}_{\bar{x}}\).

\[ \hat{\sigma}_{\bar{x}} = \frac{\overline{MR}_{\bar{x}}}{d_2(L)}, \qquad \overline{MR}_{\bar{x}} = \frac{1}{m}\sum_{j} MR_{\bar{x},j} \]

Each \(MR_{\bar{x},j}\) is the maximum minus the minimum of \(L\) consecutive subgroup means, and \(\overline{MR}_{\bar{x}}\) is their average. Only subgroups whose size equals the modal (most common) size \(n^{*}\) contribute, and a subgroup of any other size breaks the chain of consecutive means in the same way a missing value breaks the chain on an individuals chart. The modal size must also account for more than half of the subgroups; when it does not, \(\hat{\sigma}_{\bar{x}}\) cannot be computed, and neither can the between or within-and-between sigma.

The aggregate is always the average moving range. The median moving range of the subgroup means cannot be selected: the individuals methods apply to individual observations, and subgrouped data always uses the average here, whichever individuals method a dialog happens to show.

Between and within & between sigma

The between-subgroup standard deviation starts from \(\hat{\sigma}_{\bar{x}}\), the standard deviation of the subgroup means, and removes the part of it that within-subgroup variation already explains:

\[ \hat{\sigma}_{\text{between}} = \sqrt{\max\!\left(0,\; \hat{\sigma}_{\bar{x}}^2 - \frac{\hat{\sigma}_{\text{within}}^2}{n^{*}}\right)} \]

\[ \hat{\sigma}_{\text{within\&between}} = \sqrt{\hat{\sigma}_{\text{within}}^2 + \hat{\sigma}_{\text{between}}^2} \]

The result is floored at zero, so the between sigma is reported as zero whenever the subgroup means vary no more than within-subgroup variation alone would explain. When that floor is reached the within-and-between sigma equals the within sigma exactly.

Which methods each chart offers

Each dialog exposes only the methods that make sense for its chart, and the first one listed is its default.

Chart	Methods offered under Estimate Sigma	Default	Sigma behind the control limits
Xbar-R	Rbar, pooled standard deviation	Rbar	Xbar panel: the within sigma divided by the square root of each subgroup size. Range panel: center line \(d_2(n_i)\,\hat{\sigma}_{\text{within}}\).
Xbar-S	Sbar, pooled standard deviation	Sbar	Xbar panel: the same. Standard deviation panel: center line from the within sigma, scaled by \(c_4(n_i)\) or not depending on the within unbias setting.
I-MR-R/S	Rbar, Sbar, pooled standard deviation	Rbar	Individuals panel: the sigma of the subgroup means. Moving range of means panel: center line \(d_2(L)\,\hat{\sigma}_{\bar{x}}\). Third panel: the within sigma. Choosing Sbar also turns that third panel from a range chart into a standard deviation chart.
I-MR	average moving range, median moving range, and square root of MSSD on the Individuals variant	average moving range	the within sigma
Levey-Jennings	none, the dialog has no Estimate Sigma group	not applicable	the overall sigma, which is the sample standard deviation of all the plotted values. This is the main thing that distinguishes a Levey-Jennings chart from an Individuals chart, which uses a within-subgroup estimate instead.
EWMA	the subgroup methods, or the individuals methods when the data is not subgrouped	Rbar, or average moving range	the within sigma
CUSUM	the subgroup methods, or the individuals methods when the data is not subgrouped	Rbar, or average moving range	the within sigma, divided by the square root of the subgroup size on subgrouped data
Cpk/Histogram (capability with no chart)	the subgroup methods, or the individuals methods when the data is not subgrouped	Rbar, or average moving range	no limits are drawn; all four sigmas feed the capability blocks

Two substitutions happen without a message:

• If the data turns out to be individual observations (for example an identifier column whose values are all distinct, so every subgroup holds one observation), a subgroup method cannot be evaluated and the moving range within sigma is used instead, whatever the radio buttons say.
• The Nelson moving-range correction applies only to the average moving range method. The dialog enables its checkbox only there and clears it when you switch methods.

The unbiased-constants settings

Two checkboxes sit in the Estimate Sigma group, and they have different defaults and different reach.

Setting	Default	What it changes
Use unbiased constants (within sigma)	checked	Sbar: switches the estimator entirely, from the weighted form with \(c_4\) to the plain average of the \(s_i\) with no \(c_4\) at all. Pooled: divides by \(c_4(\nu + 1)\). Square root of MSSD: divides by \(c_4'(m + 1)\). Nothing at all for Rbar, average moving range, and median moving range, where \(d_2\) or \(d_4\) is always applied; the dialog greys the box out for those three.
Use unbiased constants (overall sigma)	unchecked	Divides the overall sigma by \(c_4(N)\), and nothing else. Because \(N\) is usually large this correction is small. It moves a drawn control limit on one chart only, Levey-Jennings, whose limits come from the overall sigma.

The within checkbox also moves a line on the chart, which is easy to miss: the standard deviation panel of an Xbar-S or I-MR-R/S chart centers on \(c_4(n_i)\,\hat{\sigma}_{\text{within}}\) when the box is checked and on \(\hat{\sigma}_{\text{within}}\) when it is not, and that panel's limits are multiples of its center line. So clearing the box moves the S chart's center line and both of its limits, not just the reported estimate. The range panel center, \(d_2(n_i)\,\hat{\sigma}_{\text{within}}\), does not read the box.

On a Levey-Jennings chart both boxes are ignored. Its overall sigma is always the plain square root form with no \(c_4\) correction, whatever the boxes show.

When a correction factor is undefined for the size it would be evaluated at, the correction is skipped and the uncorrected value is reported rather than nothing.

Degrees of freedom

The degrees of freedom feed the chi-square and normal approximations behind the capability confidence intervals.

Sigma estimate	Degrees of freedom \(\nu\)
Overall	\(N - 1\)
Within, pooled standard deviation	\(\sum_i (n_i - 1)\)
Within, Rbar	\(0.9\,n_{\text{sub}}\,(\bar{n} - 1)\) with \(\bar{n}\) the ceiling of the average size of the subgroups with at least two observations
Within, Sbar	\(f_n \sum_i (n_i - 1)\), where \(f_n\) is a tabulated efficiency adjustment (Bissell) looked up at the ceiling of (total observations \(\div\) \(n_{\text{sub}}\)); this average differs from the Rbar one when single-observation subgroups exist
Within, moving range (average or median)	the number of moving ranges \(m\)
Within, square root of MSSD	the number of successive differences \(m\)
Between	none; the between sigma carries no degrees of freedom, so it gets no confidence interval
Within & between	Satterthwaite combination, below

For the within & between sigma the two variance components are combined with the Satterthwaite approximation:

\[ \nu_{wb} = \frac{\left[\dfrac{MS_B + (m_e - 1)\,MS_E}{m_e}\right]^2}{\dfrac{(m_e - 1)^2\,MS_E^2}{m_e^2\,\nu_E} + \dfrac{MS_B^2}{m_e^2\,\nu_B}} \]

where, with \(n_{\text{sub},1}\) the number of subgroups having at least one observation and \(N_1\) the total observations across them (single-observation subgroups count here, unlike everywhere else on this page): \(MS_B = \sum_i n_i(\bar{x}_i - \bar{\bar{x}})^2 / (n_{\text{sub},1} - 1)\) is the between-subgroup mean square, \(MS_E = \sum_i (n_i - 1) s_i^2 / \sum_i (n_i - 1)\) the pooled within mean square (subgroups of size one contribute zero), \(\nu_B = n_{\text{sub},1} - 1\), \(\nu_E = \sum_i (n_i - 1)\), and \(m_e = (N_1^2 - \sum_i n_i^2)\,/\,(N_1 (n_{\text{sub},1} - 1))\) the effective subgroup size (\(m_e = n\) for balanced data). When \(\hat{\sigma}_{\text{between}}\) is floored to zero, the within & between sigma equals the within sigma exactly, so \(\nu_{wb}\) is set to the within degrees of freedom.

Interactions with other options

Split control limits (phases)

Each phase is estimated independently: its own within, overall, between, and within-and-between sigma, and its own degrees of freedom. Nothing is pooled across phases. A First N baseline is applied inside every phase rather than once at the start of the data, and every phase needs at least five points. Manual limits cannot be combined with splits. Full details: Splits and Phases.

Historical (known) values

A supplied mean or sigma replaces the computed one after the data-based calculation, so the estimators above still run and are then overwritten. Two consequences are worth knowing:

• Supplying a within sigma also redefines the between sigma, which is back-solved from the data-based sigma of the subgroup means. Supplying a between sigma instead redefines the sigma of the subgroup means. Entering one number changes two.
• An overridden sigma carries no degrees of freedom, so it gets no confidence interval. A supplied mean does not affect any degrees of freedom.

A single supplied value applies to every phase when splits are active; there is no per-phase historical value. On a Levey-Jennings chart a within or between sigma cannot be supplied at all, since that chart uses neither.

Transformation

A Box-Cox or Johnson transformation is applied before any sigma is computed, so with a transformation active every estimate on this page is a standard deviation of the transformed values, not of the original units. A transformation cannot be combined with historical values or with manual limits, and Levey-Jennings does not accept one. With splits, a single Box-Cox lambda is fitted on the union of the per-phase baseline windows and then applied to every row. The lambda fit follows the chart's own within-sigma structure: a subgroup fit for the subgrouped charts, and a moving-range fit for I-MR, in which case the moving range length changes the fitted lambda. See Box-Cox Transformation.

When no sigma can be computed

• A subgroup size or moving range length outside 2 to 100 is not supported. The unbiasing constants are undefined there, so no within sigma is produced.
• Subgroups of a single observation contribute nothing to any within estimator, but they still count toward the mean, the mean of the subgroup means, and the within-and-between degrees of freedom.
• Data with no variation at all, or with fewer than five usable values in any one phase, is refused with a message instead of a chart.
• When a within sigma cannot be formed even though the data varies (for example every gap breaks the moving range chain, or the Nelson correction removes every moving range), the chart is drawn without a limit band rather than refused. The same happens on an I-MR-R/S chart when no subgroup size accounts for more than half of the subgroups, since then the sigma of the subgroup means does not exist.

Used by

• Cpk/Histogram Math Details: every capability block (Within, Overall, Between, Within & Between) uses the matching sigma estimate and its degrees of freedom.
• Xbar-R and Xbar-S, I-MR, EWMA and CUSUM build their control limits from the within sigma, as do the single-panel variants Xbar, R, S, Individuals and Moving Range.
• I-MR-R/S uses three of these estimates at once: the sigma of the subgroup means on its Individuals and moving range panels, and the within sigma on its third panel.
• Levey-Jennings builds its limits from the overall sigma, which is the main thing separating it from an Individuals chart.
• Z-MR estimates a sigma per part, by one of four modes, and then standardizes against it.

See Also

• Control Chart Constants
• Control Limits and Zones
• Moving Ranges
• Baseline and Limit Estimation
• Splits and Phases
• Varying Subgroup Size Limits
• Shared Math Details

References

1. Wheeler, D. J., and Chambers, D. S. (1992). Understanding Statistical Process Control (2nd ed.). SPC Press.
2. Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Wiley.
3. Bissell, A. F. (1990). How reliable is your capability index? Journal of the Royal Statistical Society: Series C (Applied Statistics), 39(3), 331-340.
4. Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110-114.
5. Cryer, J. D., and Ryan, T. P. (1990). The estimation of sigma for an X chart: MR-bar/d2 or S/c4? Journal of Quality Technology, 22(3), 187-192.
6. Vardeman, S. B. (1999). A brief tutorial on the estimation of the process standard deviation. IIE Transactions, 31(6), 503-507.