Home / Statistical Tools / Analysis Tools / Cpkhistogram / Math Details

Math Details

This page gives the exact formulas Quantum XL uses to build a Cpk/Histogram (the tool behind the Capability Analysis and Histogram ribbon buttons). Each equation lists what it computes and where it appears in the output.

Notation

Term	Description
\(x_i\)	an individual observation
\(N\)	number of observations used for capability (non-missing values, excluding any observations excluded as outliers)
\(\bar{x}\)	process mean estimate (or the historical mean when one is entered)
\(LSL, USL\)	lower and upper specification limits
\(T\)	target value
\(\hat{\sigma}\)	the sigma estimate of the capability block being computed (within, overall, between, or within & between)
\(\nu\)	degrees of freedom of that sigma estimate
\(X_p, X_{pl}, X_{pu}, X_{pk}\)	the capability indices of one block; reported as Cp/Cpl/Cpu/Cpk or Pp/Ppl/Ppu/Ppk depending on which block the standard assigns to each label
\(\alpha\)	\(1 -\) confidence level
\(\Phi, \Phi^{-1}\)	standard normal CDF and its inverse
\(\chi^2_{p,\nu}\)	quantile \(p\) of the chi-square distribution with \(\nu\) degrees of freedom

Mean and sigma estimates

The mean is the average of the \(N\) observations (Welford accumulation). Up to four sigma estimates are computed: overall, within (subgrouped or individuals, by the selected method), between, and within & between. All but the between sigma carry degrees of freedom; the between sigma has none, so its block never reports confidence intervals. The estimators are defined once on the shared page:

Overall:

\[ \hat{\sigma}_{\text{overall}} = \sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

\(N\) is the number of non-missing observations and \(\bar{x}\) their mean. When Use unbiased constants (overall sigma) is checked, the estimate is divided by \(c_4(N)\):

\[ \hat{\sigma}_{\text{overall}}^{\,\text{unbiased}} = \frac{1}{c_4(N)}\sqrt{\frac{\sum_{i}(x_i - \bar{x})^2}{N - 1}} \]

Within (subgrouped data) uses the selected method: Rbar, Sbar, or pooled standard deviation. Within (individual observations) uses average moving range, median moving range, or square root of MSSD. Between and within & between decompose the variation of the subgroup means.

Full details: Process Sigma Estimation.

The Process Capability Standard decides which block each reported label uses: AIAG reports Cpk from the within block and Ppk from the overall block; Boeing D1-9000 reports Cpk from the overall block and suppresses Ppk; Custom lets you assign either label to the within, overall, between, or within & between block. Historical values entered in the dialog replace the corresponding estimates.

Capability indices (Cp, Cpl, Cpu, Cpk / Pp, Ppl, Ppu, Ppk)

For each sigma block, with \(\hat{\sigma}\) that block's sigma:

\[ X_{pl} = \frac{\bar{x} - LSL}{3\hat{\sigma}}, \qquad X_{pu} = \frac{USL - \bar{x}}{3\hat{\sigma}}, \qquad X_p = \frac{USL - LSL}{6\hat{\sigma}}, \qquad X_{pk} = \min(X_{pl}, X_{pu}) \]

These appear in the Process Capability section of the statistics table. \(X_p\) requires both specification limits. With only one limit, \(X_{pk}\) equals the one available one-sided index. A specification limit marked as a boundary is treated as absent for the indices and the expected PPM on that side.

Z scores and Z.Bench

\[ Z_{LSL} = \frac{\bar{x} - LSL}{\hat{\sigma}}, \qquad Z_{USL} = \frac{USL - \bar{x}}{\hat{\sigma}} \]

\[ Z_{\text{Bench}} = \Phi^{-1}\!\left(1 - \frac{PPM_{\text{Total}}}{10^6}\right) \]

\(Z_{\text{Bench}}\) converts the total expected out-of-specification proportion of the block into the equivalent single-tail normal benchmark z; it is computed when the total is strictly between \(0\) and \(10^6\) PPM.

Expected PPM

For each sigma block, the expected parts per million outside each specification limit under the normal model:

\[ PPM_{<LSL} = \Phi\!\left(\frac{LSL - \bar{x}}{\hat{\sigma}}\right)\times 10^6, \qquad PPM_{>USL} = \left[1 - \Phi\!\left(\frac{USL - \bar{x}}{\hat{\sigma}}\right)\right]\times 10^6 \]

\(PPM_{\text{Total}}\) is the sum of the tails that exist (a single tail when only one limit is given). These appear in the PPM (Defects Per Million) section as the Within and Overall rows.

Observed PPM

\[ PPM_{<LSL}^{\text{obs}} = \frac{\#\{x_i < LSL\}}{N_{\text{numeric}}}\times 10^6, \qquad PPM_{>USL}^{\text{obs}} = \frac{\#\{x_i > USL\}}{N_{\text{numeric}}}\times 10^6 \]

The observed row simply counts the data outside each limit. A boundary marking does not suppress the observed counts.

Confidence intervals for the indices

Computed when Show Confidence Interval is checked, using \(\alpha^* = \alpha/2\) for a two-sided interval and \(\alpha^* = \alpha\) for a one-sided bound.

Cp / Pp (chi-square on sigma):

\[ \left[\; X_p\sqrt{\frac{\chi^2_{\alpha^*,\,\nu}}{\nu}}, \quad X_p\sqrt{\frac{\chi^2_{1-\alpha^*,\,\nu}}{\nu}} \;\right] \]

Cpk / Ppk (Bissell's approximation):

\[ X_{pk} \pm z_{1-\alpha^*}\sqrt{\frac{1}{\left(T_\sigma/2\right)^2 N} + \frac{X_{pk}^2}{2\nu}} \]

\(T_\sigma\) is the sigma tolerance from the Process Capability Standard box (\(6\) under AIAG and Boeing D1-9000, so the first term is the familiar \(1/(9N)\)). Each block uses its own \(\nu\). When a historical mean is entered, the Cpk/Ppk interval is not reported (the mean-uncertainty term assumes the mean was estimated from the data), while the Cp/Pp interval, which depends only on sigma, remains.

Cpm

Cpm measures capability against the target using the deviation-from-target spread

\[ \tau = \sqrt{\frac{\sum_{i}(x_i - T)^2}{N - 1}} \]

\[ C_{pm} = \begin{cases} \dfrac{USL - LSL}{6\tau} & \text{both limits, } T \text{ at the midpoint (to within } 10^{-12}\text{)} \\[8pt] \dfrac{\min(T - LSL,\; USL - T)}{3\tau} & \text{both limits, } T \text{ off the midpoint} \\[8pt] \dfrac{USL - T}{3\tau} & \text{USL only} \\[8pt] \dfrac{T - LSL}{3\tau} & \text{LSL only} \end{cases} \]

Cpm requires a target and at least one non-boundary specification limit. Its one-sided lower confidence bound uses an effective degrees of freedom (Boyles):

\[ a = \frac{\bar{x} - T}{\hat{\sigma}_{\text{overall}}}, \qquad \nu_{C_{pm}} = \frac{N\,(1 + a^2)^2}{1 + 2a^2}, \qquad C_{pm}^{\text{lower}} = C_{pm}\sqrt{\frac{\chi^2_{\alpha,\,\nu_{C_{pm}}}}{\nu_{C_{pm}}}} \]

using \(\alpha\) directly (one-sided, regardless of the interval mode).

Goodness of fit

The selected normality test runs on the data (the transformed data when a transformation is active) and its statistic and p-value appear in the Goodness of Fit section.

Anderson-Darling:

\[ A^2 = -n - \frac{1}{n}\sum_{i=1}^{n}(2i - 1)\left[\ln F\!\left(x_{(i)}\right) + \ln\!\left(1 - F\!\left(x_{(n+1-i)}\right)\right)\right] \]

\(F\) is the standard normal CDF applied to the standardized value, \(F(x) = \Phi\!\left((x - \bar{x})/s\right)\), and \(x_{(1)} \le \cdots \le x_{(n)}\) are the \(n\) values being tested, sorted ascending. When Quantum XL tests a sample for normality, \(\bar{x}\) and \(s\) are the sample mean and sample standard deviation (n − 1 denominator). When it tests a fitted distribution (for example in distribution fitting), \(F\) uses the fitted parameters instead of re-estimating them.

The statistic is adjusted for sample size using the Stephens correction:

\[ A^{*2} = A^2\left(1 + \frac{0.75}{n} + \frac{2.25}{n^2}\right) \]

The p-value is a piecewise function of the adjusted statistic:

\[ p = \begin{cases} 0 & A^{*2} > 13 \\[4pt] \exp\!\left(1.2937 - 5.709\,A^{*2} + 0.0186\,(A^{*2})^2\right) & 0.6 \le A^{*2} \le 13 \\[4pt] \exp\!\left(0.9177 - 4.279\,A^{*2} - 1.38\,(A^{*2})^2\right) & 0.34 \le A^{*2} < 0.6 \\[4pt] 1 - \exp\!\left(-8.318 + 42.796\,A^{*2} - 59.938\,(A^{*2})^2\right) & 0.2 \le A^{*2} < 0.34 \\[4pt] 1 - \exp\!\left(-13.436 + 101.14\,A^{*2} - 223.73\,(A^{*2})^2\right) & A^{*2} < 0.2 \end{cases} \]

The reported p-value is clamped to a minimum of \(0.005\).

Full details: Anderson-Darling Test.

Shapiro-Wilk:

\[ W = \frac{\left(\sum_{i=1}^{n} a_i\, x_{(i)}\right)^2}{\sum_{i=1}^{n} (x_i - \bar{x})^2} \]

\(x_{(1)} \le \cdots \le x_{(n)}\) are the values sorted ascending and \(\bar{x}\) is their mean. The constants \(a_i\) are derived from the expected values and covariances of standard normal order statistics; Quantum XL computes them with Royston's Algorithm AS R94. \(W\) is close to \(1\) when the data follow a normal distribution.

The p-value comes from Royston's normalizing transformation. For \(n = 3\) the exact value \(p = \frac{6}{\pi}\left(\arcsin\sqrt{W} - \frac{\pi}{3}\right)\) is used. For \(4 \le n \le 11\) the statistic is transformed with \(y = -\ln\!\big(\gamma - \ln(1 - W)\big)\), and for \(n > 11\) with \(y = \ln(1 - W)\); in both cases \(y\) is standardized with sample-size-dependent mean and standard deviation polynomials, and the p-value is the upper tail of the standard normal distribution for the resulting \(z\).

Full details: Shapiro-Wilk Test.

Kolmogorov-Smirnov:

\[ D^{+} = \max_i\left(F_n(x_{(i)}) - \Phi(z_{(i)})\right), \qquad D^{-} = \max_i\left(\Phi(z_{(i)}) - F_n(x_{(i-1)})\right), \qquad D = \max(D^{+}, D^{-}) \]

\(F_n\) is the empirical cumulative distribution function of the sorted data, and \(z_{(i)} = (x_{(i)} - \bar{x})/s\) standardizes each value with the sample mean and sample standard deviation. \(D\) is the largest vertical distance between the empirical CDF and the fitted normal CDF.

Because the normal parameters are estimated from the same data, the p-value uses the Lilliefors correction rather than the classical Kolmogorov-Smirnov distribution.

Full details: Kolmogorov-Smirnov Test.

Distribution fitting

When distribution fitting is enabled, Quantum XL fits each candidate distribution (by maximum likelihood; the normal uses the sample mean and standard deviation) and reports, per distribution: the corrected Akaike information criterion (AICc), the Bayesian information criterion (BIC), \(-2\ln L\), the adjusted Anderson-Darling statistic against the fitted distribution, its p-value, and the parameter estimates with standard errors and confidence intervals. Lower AICc/BIC and higher p-values indicate better fit. The candidate set is the 14 supported distributions unless specific fit types are chosen. The per-distribution density formulas are on the Distribution Fit pages.

Non-normal capability

When the selected fit distribution is not Normal, capability comes from the fitted distribution instead of the normal model, by the chosen method. Confidence intervals are not defined for either method. The out-of-specification rate is the fitted distribution's tail area on each side:

\[ PPM_{<LSL} = F(LSL)\times 10^6, \qquad PPM_{>USL} = \bar{F}(USL)\times 10^6 \]

where \(F\) is the fitted CDF and \(\bar{F}\) the upper tail probability (computed directly, not as \(1 - F\), to stay accurate at very low defect rates).

Percentile method. With \(P_q = F^{-1}(q)\) the fitted quantiles:

\[ X_{pl} = \frac{P_{0.5} - LSL}{P_{0.5} - P_{0.00135}}, \qquad X_{pu} = \frac{USL - P_{0.5}}{P_{0.99865} - P_{0.5}}, \qquad X_p = \frac{USL - LSL}{P_{0.99865} - P_{0.00135}}, \qquad X_{pk} = \min(X_{pl}, X_{pu}) \]

The \(0.135\%\) and \(99.865\%\) quantiles play the role of \(\bar{x} \pm 3\hat{\sigma}\), and the median plays the role of the mean.

Z-Score method. Each tail area is converted to the equivalent normal z:

\[ z_{LSL} = -\Phi^{-1}\!\big(F(LSL)\big), \qquad z_{USL} = -\Phi^{-1}\!\big(\bar{F}(USL)\big) \]

\[ X_{pl} = \frac{z_{LSL}}{3}, \qquad X_{pu} = \frac{z_{USL}}{3}, \qquad X_p = \frac{z_{LSL} + z_{USL}}{6}, \qquad X_{pk} = \min(X_{pl}, X_{pu}) \]

\(Z_{\text{Bench}}\) is built from the total fitted out-of-specification proportion exactly as in the normal path.

Transformations

With a Box-Cox or Johnson transformation active, the data, specification limits, and target are transformed, all capability math runs in the transformed domain, and the statistics table reports both domains where applicable. A transformation cannot be combined with a non-normal fit distribution or with historical values.

Box-Cox:

\[ y = \begin{cases} x^{\lambda} & \lambda \ne 0 \\[4pt] \ln x & \lambda = 0 \end{cases} \]

Quantum XL uses the simple power form (not the scaled form \((x^{\lambda} - 1)/\lambda\)), so the transformed values are directly comparable to transformed specification limits. The transformation requires every value to be positive, at least 3 non-missing values, and non-zero variation. \(\lambda\) is either supplied by the user or fitted automatically; in both cases it is limited to \([-5, 5]\).

The automatic \(\lambda\) minimizes the spread of the scaled transform

\[ W_i(\lambda) = \begin{cases} \dfrac{(x_i/g)^{\lambda} - 1}{\lambda} & \lambda \ne 0 \\[8pt] \ln(x_i/g) & \lambda = 0 \end{cases} \]

where \(g\) is the geometric mean of the data (dividing by \(g\) makes spreads comparable across \(\lambda\)). The objective depends on the data structure:

• Subgrouped data: minimize the pooled within-subgroup standard deviation of \(W(\lambda)\).
• Individual observations: minimize the moving-range sigma of \(W(\lambda)\).

The minimum is found with Brent's bounded minimization after a coarse scan brackets it.

Specification limits and the target are transformed with the same \(y = x^{\lambda}\) (or \(\ln x\)) rule so capability is evaluated consistently in the transformed domain. A negative \(\lambda\) reverses order, so the transformed LSL and USL swap roles; Quantum XL performs this swap internally and reports results in the user's original orientation.

Full details: Box-Cox Transformation.

Johnson fits the bounded, lognormal, or unbounded Johnson family and transforms with the fitted family; the family forms and parameter fitting are on Johnson Transformation Math Details. The Anderson-Darling statistic of the transformed data is reported with the transformation.

Histogram bins

With automatic binning, the target number of bins is the square root rule

\[ k_{\text{bins}} = \operatorname{round}\!\left(\sqrt{N_{\text{numeric}}}\right) \]

and the bin width is the data range divided by \(k_{\text{bins}}\), rounded to a nearby "nice" round value derived from the axis tick scale. If that width would need more than 240 bins, it is widened to the smallest nice width (from a finer candidate table) that does not. The grid is then snapped so bin edges land on multiples of the width, and extended so the bars, both specification limits, and the PDF curve spans (\(\bar{x} \pm 3.6\,\hat{\sigma}_{\text{overall}}\) and \(\bar{x} \pm 3.6\,\hat{\sigma}_{\text{within}}\), when a curve is drawn) are covered, up to an overall bin cap.

Special cases:

• All values identical (checked first, before any manual setting): a single bin spanning \(\pm 0.5\) around the value.
• Manual bin width is used exactly as entered.
• Manual number of bins \(k\) sets the width to \(\text{range}/k\) (with a tiny upward nudge so the maximum lands in the last bin). Combined with a manual left point, the width becomes \(\big(x_{\max} - \min(\text{left point}, x_{\min})\big)/k\).
• Manual left point anchors the grid at \(\min(\text{left point}, x_{\min})\).
• Integer binning: when every value is an integer, there are at most 11 distinct values, the axis span is between 1 and 11, and no PDF curve is drawn, the bins are unit-wide. With no specification limits and no manual left point they are centered on the integers (anchored at \(x_{\min} - 0.5\)); with a specification limit present, the unit-wide bins align to the standard grid instead.

A value below the left anchor is excluded; a value at or beyond the last edge counts in the last bin; otherwise a value falls in bin \(\lfloor (x - \text{anchor})/\text{width} \rfloor\) (edges chain exactly, so a value on a boundary belongs to the higher bin).

PDF curves

Each displayed probability density curve is evaluated at 500 points across the axis and scaled from density to expected frequency:

\[ y(x) = f(x)\; N_{\text{numeric}}\; w \]

where \(f\) is the normal density with the block's mean and sigma (overall, within, and within & between curves) or the fitted non-normal density, and \(w\) is the bin width. This puts the curve on the same vertical scale as the bars.

Shared Math Details used here

This tool uses shared formulas defined once in Shared Math Details. See those pages for the exact definitions.

Shared concept	Used here for	Reference
Process sigma estimation	the sigma and degrees of freedom of every capability block	Process Sigma Estimation
Control chart constants	\(d_2\), \(d_3\), \(d_4\), \(c_4\), \(c_4'\) inside the sigma estimators	Control Chart Constants
Anderson-Darling test	the default goodness-of-fit method	Anderson-Darling Test
Shapiro-Wilk test	a selectable goodness-of-fit method	Shapiro-Wilk Test
Kolmogorov-Smirnov test	a selectable goodness-of-fit method	Kolmogorov-Smirnov Test
Box-Cox transformation	the Box-Cox option	Box-Cox Transformation
Standard normal distribution	expected PPM, Z scores, Z.Bench, the Cpk interval	Standard Normal Distribution
Chi-square distribution	the Cp/Pp interval and the Cpm lower bound	Chi-Square Distribution

See Also

• Options
• Preparing Your Data
• Johnson Transformation Math Details

References

1. Kane, V. E. (1986). Process capability indices. Journal of Quality Technology, 18(1), 41-52.
2. Bissell, A. F. (1990). How reliable is your capability index? Journal of the Royal Statistical Society: Series C (Applied Statistics), 39(3), 331-340.
3. Chan, L. K., Cheng, S. W., and Spiring, F. A. (1988). A new measure of process capability: Cpm. Journal of Quality Technology, 20(3), 162-175.
4. Boyles, R. A. (1991). The Taguchi capability index. Journal of Quality Technology, 23(1), 17-26.
5. Clements, J. A. (1989). Process capability calculations for non-normal distributions. Quality Progress, 22(9), 95-100.
6. Kotz, S., and Johnson, N. L. (1993). Process Capability Indices. Chapman & Hall.
7. Box, G. E. P., and Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2), 211-252.
8. Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Wiley.