Home / Shared Math Details / Control Chart Math / Process Sigma Estimation

Process Sigma Estimation¶

This page defines the standard deviation estimators Quantum XL uses for process data: the overall sigma, the within-subgroup sigma (subgrouped data), the within sigma for individual observations, and the between-subgroup decomposition. Capability analysis and the variable control charts share these estimators.

Notation¶

Term	Description
\(x_i\)	an individual observation
\(N\)	total number of non-missing observations
\(\bar{x}\)	mean of all non-missing observations
\(k\)	number of subgroups with at least two observations
\(n_i\)	size of subgroup \(i\) (non-missing values)
\(\bar{x}_i\)	mean of subgroup \(i\)
\(R_i\)	range (maximum minus minimum) of subgroup \(i\)
\(s_i\)	sample standard deviation of subgroup \(i\) (\(n_i - 1\) denominator)
\(L\)	moving range length (default 2)
\(MR_t\)	moving range: maximum minus minimum of \(L\) consecutive observations
\(m\)	number of moving ranges (or successive differences)
\(n^{*}\)	modal (most common) subgroup size
\(d_2, d_3, d_4, c_4, c_4'\)	unbiasing constants, defined on Control Chart Constants
\(\hat{\sigma}\)	an estimated process standard deviation
\(\nu\)	degrees of freedom of a sigma estimate

Overall sigma¶

\[ \hat{\sigma}_{\text{overall}} = \sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

\(N\) is the number of non-missing observations and \(\bar{x}\) their mean. When Use unbiased constants (overall sigma) is checked, the estimate is divided by \(c_4(N)\):

\[ \hat{\sigma}_{\text{overall}}^{\,\text{unbiased}} = \frac{1}{c_4(N)}\sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

Used by: the overall (Ppk) capability block and the fitted overall PDF curve; control chart estimates that need the long-term sigma.

Within sigma, subgrouped data¶

The estimator is selected in the dialog (Rbar, Sbar, or pooled standard deviation). Subgroups with fewer than two observations contribute nothing to the within sigma.

Rbar (average range)¶

\[ \hat{\sigma}_{\text{within}} = \frac{\sum_{i=1}^{k} f_i\, \dfrac{R_i}{d_2(n_i)}}{\sum_{i=1}^{k} f_i}, \qquad f_i = \left(\frac{d_2(n_i)}{d_3(n_i)}\right)^{2} \]

\(R_i\) is the range of subgroup \(i\), \(n_i\) its size, and \(k\) the number of subgroups with at least two observations. Each subgroup's unbiased range estimate \(R_i/d_2(n_i)\) is combined with the minimum-variance weights \(f_i\) (subgroups whose range estimate is more precise get more weight). The unbiased-constants setting has no effect on this estimator: \(d_2\) is always applied.

Sbar (average standard deviation)¶

With Use unbiased constants (within sigma) checked (the default):

\[ \hat{\sigma}_{\text{within}} = \frac{\sum_{i=1}^{k} h_i\, \dfrac{s_i}{c_4(n_i)}}{\sum_{i=1}^{k} h_i}, \qquad h_i = \frac{c_4(n_i)^2}{1 - c_4(n_i)^2} \]

\(s_i\) is the sample standard deviation of subgroup \(i\) (with the \(n_i - 1\) denominator) and \(h_i\) the minimum-variance weight. With the unbiased-constants box unchecked, the estimate is the plain average of the subgroup standard deviations:

\[ \hat{\sigma}_{\text{within}} = \frac{1}{k}\sum_{i=1}^{k} s_i \]

Pooled standard deviation¶

\[ \hat{\sigma}_{\text{within}} = \sqrt{\frac{\sum_{i=1}^{k} (n_i - 1)\, s_i^2}{\sum_{i=1}^{k} (n_i - 1)}} \]

When Use unbiased constants (within sigma) is checked (the default), the pooled estimate is divided by \(c_4(\nu + 1)\), where \(\nu = \sum_i (n_i - 1)\) is the pooled degrees of freedom:

\[ \hat{\sigma}_{\text{within}}^{\,\text{unbiased}} = \frac{1}{c_4(\nu + 1)}\sqrt{\frac{\sum_{i} (n_i - 1)\, s_i^2}{\nu}} \]

Within sigma, individual observations¶

Average moving range¶

\[ \hat{\sigma}_{\text{within}} = \frac{\overline{MR}}{d_2(L)}, \qquad \overline{MR} = \frac{1}{m}\sum_{t} MR_t \]

Each moving range \(MR_t\) is the maximum minus the minimum of \(L\) consecutive non-missing observations (\(L\) is the moving range length, default \(2\), in which case \(MR_t = |x_t - x_{t-1}|\)), and \(m\) is the number of moving ranges. A missing value breaks the chain of consecutive observations, so no moving range spans a gap.

When Use Nelson moving-range correction is checked, moving ranges larger than \(D_4(L)\,\overline{MR}\), with \(D_4(L) = 1 + 3\,d_3(L)/d_2(L)\), are removed and the average is recomputed. This repeats until no moving range exceeds the recomputed threshold (at most 50 iterations). The correction affects only the sigma estimate, not the moving range values themselves.

Median moving range¶

\[ \hat{\sigma}_{\text{within}} = \frac{\widetilde{MR}}{d_4(L)} \]

\(\widetilde{MR}\) is the median of the moving ranges (the average of the two middle values when their count is even), and \(d_4(L)\) is the unbiasing constant for the median range of \(L\) standard normal observations. The moving ranges are formed exactly as in the average moving range method.

Square root of MSSD¶

\[ \hat{\sigma}_{\text{within}} = \sqrt{\frac{\sum_{i=1}^{m} (x_{i+1} - x_i)^2}{2\,m}} \]

The sum runs over the \(m\) successive differences of consecutive non-missing observations (a missing value breaks the chain). This is the square root of half the mean squared successive difference (MSSD). When Use unbiased constants (within sigma) is checked (the default), the estimate is divided by the tabulated correction factor \(c_4'(m + 1)\).

Between and within & between sigma¶

The between-subgroup standard deviation starts from a moving range of the subgroup means. Only subgroups whose size equals the modal (most common) size \(n^{*}\) are used, and the modal size must account for more than half of the subgroups; a subgroup of any other size breaks the chain of consecutive means. The moving ranges of those means are aggregated with the selected individuals method (average with \(d_2(L)\), or median with \(d_4(L)\)) to give the standard deviation of the subgroup means, \(\hat{\sigma}_{\bar{x}}\). Then:

\[ \hat{\sigma}_{\text{between}} = \sqrt{\max\!\left(0,\; \hat{\sigma}_{\bar{x}}^2 - \frac{\hat{\sigma}_{\text{within}}^2}{n^{*}}\right)} \]

\[ \hat{\sigma}_{\text{within\&between}} = \sqrt{\hat{\sigma}_{\text{within}}^2 + \hat{\sigma}_{\text{between}}^2} \]

Subtracting \(\hat{\sigma}_{\text{within}}^2 / n^{*}\) removes the part of the variation of the subgroup means that is explained by within-subgroup variation; the result is floored at zero.

Degrees of freedom¶

The degrees of freedom feed the chi-square and normal approximations behind the capability confidence intervals.

Sigma estimate	Degrees of freedom \(\nu\)
Overall	\(N - 1\)
Within, pooled standard deviation	\(\sum_i (n_i - 1)\)
Within, Rbar	\(0.9\,k\,(\bar{n} - 1)\) with \(\bar{n}\) the ceiling of the average size of the subgroups with at least two observations
Within, Sbar	\(f_n \sum_i (n_i - 1)\), where \(f_n\) is a tabulated efficiency adjustment (Bissell) looked up at the ceiling of (total observations \(\div\) \(k\)); this average differs from the Rbar one when single-observation subgroups exist
Within, moving range (average or median)	the number of moving ranges \(m\)
Within, square root of MSSD	the number of successive differences \(m\)
Between	none; the between sigma carries no degrees of freedom, so it gets no confidence interval
Within & between	Satterthwaite combination, below

For the within & between sigma the two variance components are combined with the Satterthwaite approximation:

\[ \nu_{wb} = \frac{\left[\dfrac{MS_B + (m_e - 1)\,MS_E}{m_e}\right]^2}{\dfrac{(m_e - 1)^2\,MS_E^2}{m_e^2\,\nu_E} + \dfrac{MS_B^2}{m_e^2\,\nu_B}} \]

where, with \(k_1\) the number of subgroups having at least one observation and \(N_1\) the total observations across them (single-observation subgroups count here, unlike everywhere else on this page): \(MS_B = \sum_i n_i(\bar{x}_i - \bar{\bar{x}})^2 / (k_1 - 1)\) is the between-subgroup mean square, \(MS_E = \sum_i (n_i - 1) s_i^2 / \sum_i (n_i - 1)\) the pooled within mean square (subgroups of size one contribute zero), \(\nu_B = k_1 - 1\), \(\nu_E = \sum_i (n_i - 1)\), and \(m_e = (N_1^2 - \sum_i n_i^2)\,/\,(N_1 (k_1 - 1))\) the effective subgroup size (\(m_e = n\) for balanced data). When \(\hat{\sigma}_{\text{between}}\) is floored to zero, the within & between sigma equals the within sigma exactly, so \(\nu_{wb}\) is set to the within degrees of freedom.

Used by¶

Cpk/Histogram Math Details: every capability block (Within, Overall, Between, Within & Between) uses the matching sigma estimate and its degrees of freedom.
The variable control charts (X-bar R, X-bar S, I-MR) use the same within estimators for their control limits; their math pages are planned.

References¶

Wheeler, D. J., and Chambers, D. S. (1992). Understanding Statistical Process Control (2nd ed.). SPC Press.
Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Wiley.
Bissell, A. F. (1990). How reliable is your capability index? Journal of the Royal Statistical Society: Series C (Applied Statistics), 39(3), 331-340.
Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110-114.
Cryer, J. D., and Ryan, T. P. (1990). The estimation of sigma for an X chart: MR-bar/d2 or S/c4? Journal of Quality Technology, 22(3), 187-192.
Vardeman, S. B. (1999). A brief tutorial on the estimation of the process standard deviation. IIE Transactions, 31(6), 503-507.