Home / Shared Math Details / Control Chart Math / Baseline and Limit Estimation

Baseline and Limit Estimation

Quantum XL can estimate a control chart's center line and control limits from the first part of the data instead of from all of it. This page defines that baseline window: which quantities are restricted to it, which are always computed from every value, how the window advances past blanks, and how it behaves inside a phase.

The option that selects the window is called Estimate Limits From on the variable charts and Baseline Estimation on the attribute and rare-event charts. Two labels, one idea. The Z-MR chart does not offer it.

Notation

Term	Description
\(N_b\)	the requested baseline count (the number typed in the box)
\(M\)	number of plotted positions available (points, subgroups, or rows)
\(M_s\)	number of plotted positions in section (phase) \(s\)
\(N_b^{\text{eff}}\)	the baseline count actually used
\(\rho_i\)	zero-based rank of position \(i\) among the positions the window counts
\(b_i\)	\(1\) when position \(i\) is in the baseline, \(0\) otherwise
\(\mathcal{B}\)	the baseline: the set of positions with \(b_i = 1\)
\(x_{ij}\)	observation \(j\) of subgroup \(i\)
\(n_i\)	size of subgroup \(i\), or the sample size of attribute row \(i\)
\(d_i\)	defective units in attribute row \(i\)
\(y_i\)	defect count in attribute row \(i\)
\(a_i\)	inspection size of attribute row \(i\)
\(L\)	moving range length (default \(2\))
\(m_{\mathcal{B}}\)	number of moving ranges available inside the baseline
\(k\)	sigma multiplier, default \(3\)
\(\hat{\mu}_{\mathcal{B}}, \hat{\sigma}_{\mathcal{B}}\)	center and sigma estimated from the baseline

What the baseline changes, and what it never changes

The baseline changes what the limits are estimated from. It never changes what is plotted. Every point always appears on the chart. Choosing a baseline of the first 25 subgroups does not hide subgroup 26 onward: it means the center line and the control limits are computed from the first 25 subgroups and then extended across the whole chart, so the later points are judged against the earlier baseline. That is the entire purpose of the setting.

The resulting estimates are applied to every position in the section, including the positions outside the baseline:

\[ \text{CL} = \hat{\mu}_{\mathcal{B}}, \qquad \text{UCL}_i = \hat{\mu}_{\mathcal{B}} + k\,\hat{\sigma}_{\mathcal{B},\,i}, \qquad \text{LCL}_i = \hat{\mu}_{\mathcal{B}} - k\,\hat{\sigma}_{\mathcal{B},\,i}, \qquad \text{for every } i \]

The subscript \(\mathcal{B}\) records that the center and the sigma came from the baseline. Nothing in these formulas depends on \(b_i\), so a position with \(b_i = 0\) is plotted, given limits, and tested by the out-of-control rules exactly like a baseline position.

The baseline window

The requested count is capped at what is available:

\[ N_b^{\text{eff}} = \min\left(N_b,\; M\right) \]

\(N_b\) is the requested baseline count and \(M\) the number of plotted positions available (points on an individuals chart, subgroups on a subgrouped chart, rows on an attribute or rare-event chart). When no baseline count is requested, \(N_b^{\text{eff}} = M\) and the baseline is the whole set.

Membership is then an indicator on each position:

\[ b_i = \begin{cases} 1, & \rho_i < N_b^{\text{eff}} \\[4pt] 0, & \rho_i \ge N_b^{\text{eff}} \end{cases} \]

\(\rho_i\) is the zero-based rank of position \(i\) among the positions the window counts, so \(b_i = 1\) marks the first \(N_b^{\text{eff}}\) counted positions and \(\mathcal{B} = \{\, i : b_i = 1 \,\}\) is the baseline.

How the window advances

The rank \(\rho_i\) counts positions, not rows, and what counts as a position differs between the two chart families.

Family	The window counts	A blank position	An excluded outlier
Variable charts, subgrouped (\(\bar{X}\)-R, \(\bar{X}\)-S, I-MR-R/S, and the subgrouped EWMA and CUSUM)	every subgroup holding at least one value	a fully empty subgroup does not consume a slot, so the window reaches further down the chart	the value is treated as missing, so it does not consume a slot
Variable charts, individuals (I-MR, Levey-Jennings, and the individuals EWMA and CUSUM)	every point that has a value	does not consume a slot, so the window reaches further	does not consume a slot
Attribute and rare-event charts (p, np, c, u, Laney P', Laney U', G, T)	every row by position	consumes a slot: the row is skipped in the estimate but still counted against \(N_b\), so blanks reduce the effective baseline	consumes a slot and is force-excluded from the estimate

The consequence of the last row is worth stating plainly: for the same user-facing setting, a blank inside the window shortens the attribute baseline but does not shorten a variable baseline.

Units on an individuals chart

On an individuals chart the count is a number of data points that have a value, not a number of moving ranges. The moving ranges follow from those points, as shown below.

What is estimated from the baseline

Every estimator that feeds a center line or a control limit replaces its sum over all positions with the same sum restricted to the baseline:

\[ \sum_{i} \quad \longrightarrow \quad \sum_{i \,:\, b_i = 1} \]

The grand mean and the overall sigma become

\[ \bar{\bar{x}} = \frac{\displaystyle\sum_{i\,:\,b_i=1}\;\sum_{j=1}^{n_i} x_{ij}}{\displaystyle\sum_{i\,:\,b_i=1} n_i}, \qquad \hat{\sigma}_{\text{overall}}^{\,2} = \frac{\displaystyle\sum_{i\,:\,b_i=1}\;\sum_{j=1}^{n_i} \left(x_{ij} - \bar{\bar{x}}\right)^{2}}{\displaystyle\left(\sum_{i\,:\,b_i=1} n_i\right) - 1} \]

and each within-subgroup estimator uses only the baseline subgroups' own statistics, so \(\overline{R}\), \(\overline{s}\), the pooled sum of squares, the pooled degrees of freedom, the modal subgroup size, and the between-subgroup terms are all accumulated over \(\{\, i : b_i = 1 \,\}\).

On the attribute charts the same indicator restricts the center-line estimate:

\[ \bar{p} = \frac{\displaystyle\sum_{i\,:\,b_i=1} d_i}{\displaystyle\sum_{i\,:\,b_i=1} n_i}, \qquad \bar{u} = \frac{\displaystyle\sum_{i\,:\,b_i=1} y_i}{\displaystyle\sum_{i\,:\,b_i=1} a_i}, \qquad \bar{c} = \frac{1}{\left|\mathcal{B}\right|}\sum_{i\,:\,b_i=1} y_i \]

\(d_i\) is the number of defective units in subgroup \(i\), \(n_i\) its sample size, \(y_i\) a defect count, and \(a_i\) an inspection size. A baseline row whose count or size is missing or not positive contributes nothing to these sums.

The full split is exhaustive. Restricting the sums to \(\mathcal{B}\) moves these quantities, and only these:

Estimated from the baseline only	Computed from all the data
Grand mean \(\bar{\bar{x}}\)	Every plotted value
Overall sigma and all of its accumulators	Each subgroup's own mean, range, and standard deviation
Mean of the subgroup means	Each subgroup's own size
\(\overline{R}\), \(\overline{s}\), and the pooled sum of squares	Minimum, maximum, and the count \(N\)
Pooled degrees of freedom	Observed parts per million against the specification limits
Between-subgroup components and their degrees of freedom	The plotted moving ranges
Modal subgroup size	The histogram and the distribution fit
The moving range accumulators and the MSSD accumulators	The out-of-control results (the rules run on every point)
The Box-Cox lambda and the Johnson parameters
The attribute overdispersion statistic \(\sigma_Z\) and the z-scores behind its probability plot
The degrees of freedom reported for each sigma
\(\bar{p}\), \(\bar{u}\), and \(\bar{c}\)

Two rows in the left column surprise readers. The transformation is fitted on the baseline, not on the full data set, so the baseline choice defines the transformation that is then applied to every value. The degrees of freedom reported alongside each sigma follow the baseline counts, so a short baseline reports fewer degrees of freedom and widens the capability confidence intervals that read them.

Moving ranges inside the baseline

On an individuals chart the mean uses the baseline points themselves, while the within sigma uses the moving ranges that fall among those points:

\[ m_{\mathcal{B}} \;\le\; N_b^{\text{eff}} - (L - 1) \]

With the default moving range length \(L = 2\), a baseline of \(N_b^{\text{eff}}\) points supplies at most \(N_b^{\text{eff}} - 1\) moving ranges, and fewer when a gap inside the baseline breaks the chain of consecutive values.

So a baseline of 25 points gives the mean 25 values and the within sigma at most 24 moving ranges. When the average moving range method is used with the Nelson correction, only the baseline moving ranges are candidates for trimming, so the baseline choice and the correction compound.

Baseline inside a phase

When the chart is split into phases, the baseline is applied inside each phase:

\[ N_b^{\text{eff}}(s) = \min\left(N_b,\; M_s\right) \]

One requested count \(N_b\) is reused inside every section \(s\) and capped at that section's own length \(M_s\), so each section estimates from its own first \(N_b^{\text{eff}}(s)\) positions and the window never reaches across a section boundary.

There is one requested count for the whole chart and no way to set a different count per phase. The cap \(\min(N_b, M_s)\) is what makes a short phase fall back to using all of its own positions while a long phase still uses \(N_b\). Because the window is rebuilt inside each phase, a baseline can never span a phase boundary.

Minimum counts

The engine enforces a minimum baseline count, and the minimum differs by family. A value below the minimum is raised to it silently.

Charts	Option label	Minimum \(N_b\)
\(\bar{X}\)-R, \(\bar{X}\)-S, I-MR, I-MR-R/S, EWMA, CUSUM, Levey-Jennings	Estimate Limits From	5
p, np, c, u, Laney P', Laney U'	Baseline Estimation	2
G, T	Baseline Estimation	3
Z-MR	not offered	not applicable

Edge cases

Case	Behavior
The baseline option is selected but the count is left blank	Equivalent to using all positions. No baseline restriction is applied.
\(N_b\) below the family minimum	Raised to the minimum silently.
\(N_b\) at or above the number of positions	All positions are used. On the variable charts this follows from \(\min(N_b, M)\); on the rare-event charts the restriction is switched off entirely; on the attribute charts the exclusion index simply falls past the last row.
\(N_b\) larger than one phase but not another	Each phase is treated independently: the short phase uses all of its positions, the longer phase uses \(N_b\).
A blank position inside the window	Variable charts: does not consume a slot, so the window reaches further. Attribute and rare-event charts: consumes a slot, because the mask is positional.
An excluded outlier inside the window	Variable charts: the value is removed before the statistics are accumulated, so it does not consume a slot. Attribute and rare-event charts: it consumes a slot and is force-excluded from the estimate.
A phase with fewer than five valid values (variable charts)	The whole analysis fails with a minimum-data error. The check is made per phase, so one short phase stops the chart.
A baseline that leaves fewer than two moving ranges	The within sigma has too little information to be meaningful even though the call may succeed. Check the reported degrees of freedom.

Interaction with supplied and manual limits

A historical value replaces the estimate after the estimate has been made, so the baseline no longer affects whatever was supplied. Supply the mean and the center line comes from the supplied value while the baseline still governs any sigma that was not supplied, and still governs the transformation fit. Supply a sigma and that sigma's degrees of freedom are reported as not available.

Manual limits behave differently by family. On the variable charts a manually entered band replaces the drawn limits, while the baseline still governs the mean and sigma reported in the summary table and used by the process statistics. On the attribute charts the whole center-line estimate is skipped under manual limits, so the baseline has no effect at all.

Capability and the baseline cannot be combined

On the variable charts that offer a capability analysis, requesting Cpk or Ppk and requesting a baseline are mutually exclusive: a frozen baseline leaves the capability basis ambiguous. Entering a specification limit reverts the selection to all subgroups, and the engine rejects the combination if it ever reaches it.

Used by

• Attribute charts: p, np, c, u, Laney P', and Laney U'.
• Rare-event charts: G and T.
• The variable charts (\(\bar{X}\)-R, \(\bar{X}\)-S, I-MR, I-MR-R/S, EWMA, CUSUM, Levey-Jennings) apply the same window to their mean and sigma estimates; their math pages are planned.

See Also

• Process Sigma Estimation
• Control Limits and Zones
• Splits and Phases
• Laney Overdispersion
• Shared Math Details

References

1. Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Wiley.
2. Wheeler, D. J., and Chambers, D. S. (1992). Understanding Statistical Process Control (2nd ed.). SPC Press.
3. Woodall, W. H. (2000). Controversies and contradictions in statistical process control. Journal of Quality Technology, 32(4), 341-350.
4. Chakraborti, S., Human, S. W., and Graham, M. A. (2009). Phase I statistical process control charts: an overview and some results. Quality Engineering, 21(1), 52-62.