Home / Shared Math Details / DOE / Sums of Squares
Sums of Squares¶
The ANOVA table on a regression sheet reports two sums of squares per term, Seq SS and Adj SS, and they answer different questions. This page defines both, the test built on each, the pure error and lack of fit block below them, and the variance inflation factor beside the coefficients.
Notation¶
| Term | Description |
|---|---|
| \(SSR(\cdot)\) | regression sum of squares of the model listed |
| \(SS_{\text{seq}}\), \(SS_{\text{adj}}\) | sequential and adjusted sum of squares of a term |
| \(MS_{\text{error}}\) | residual mean square |
| \(df_{\text{reg}}\), \(df_{\text{error}}\) | regression and residual degrees of freedom |
| \(n\) | number of runs used |
| \(n_i\) | number of responses in repeat group \(i\) |
| \(s_i^2\) | variance within repeat group \(i\) |
| \(R_j^2\) | R squared from regressing term \(j\) on the other terms |
| \(Z\) | the standardized design matrix |
Sequential sums of squares¶
The sequential sum of squares for a term is the amount the regression sum of squares drops when that term is the last one removed from the model. Writing \(SSR(\cdot)\) for the regression sum of squares of a model,
so the value depends on the order the terms are entered. The sequential sums of squares add up to the regression sum of squares.
A categorical term occupies several columns, and its sequential sum of squares is the total of theirs. That is why the table has one row per term rather than one row per column.
Adjusted sums of squares¶
The adjusted sum of squares for a term is the amount the regression sum of squares drops when that term is removed from the full model, with every other term left in:
It does not depend on the order the terms were entered. For a categorical term every column belonging to it is removed together.
The two differ in what they depend on. An adjusted sum of squares does not depend on the order the terms were entered, since every other term is left in the model while this one is taken out. A sequential sum of squares does depend on that order, and the sequential values are the ones that add up to the regression total.
When the two agree¶
In an orthogonal design the two sums of squares are equal, because removing a term takes nothing away from the others. Quantum XL relies on that rather than recomputing: where a term occupies a single column and its variance inflation factor is \(1\), the adjusted sum of squares is taken to be the sequential one.
The test on a term¶
Each term's sum of squares is turned into a mean square first, using that term's own degrees of freedom, and the mean square is then tested against the residual mean square:
Both degrees of freedom matter. A term occupying one column has \(df_{\text{term}} = 1\), and only then does \(F\) reduce to \(SS_{\text{term}} / MS_{\text{error}}\). A categorical term or a block occupies several columns, so its \(F\) and \(p\) are built from its own degrees of freedom rather than from \(1\).
Blocking¶
When the model contains a block term, the block is not part of the effect of the factors, so its sum of squares is taken out of the regression sum of squares at the end:
and the regression degrees of freedom drop by one for each blocking column, after which the mean square is recomputed as \(MSR = SSR_{\text{reported}} / df_{\text{reg}}\).
The order matters. The per-term sequential and adjusted sums of squares are computed from the fitted regression sum of squares, the one that still includes the block. Only the reported total has the block removed.
Pure error¶
Pure error is estimated from runs that repeat the same settings. Rows are grouped by their whole row of the design matrix, which is the model's own columns and the blocking columns rather than the factor columns on the sheet, and a group of \(n_i \ge 2\) responses contributes its variance
Summing over the groups,
A group with a single run contributes nothing. If no settings repeat then \(df_{\text{pure}} = 0\) and there is no pure error estimate at all, which is why a design without replicates reports no lack of fit test.
Lack of fit¶
What the residuals contain beyond pure error is lack of fit:
The test compares the lack of fit mean square against the pure error mean square. The lack of fit sum of squares is reported as \(0\) if the arithmetic makes it negative, and every cell of the test reads NA in any of three cases: there is no pure error, the pure error mean square is at or below \(10^{-11}\), or \(df_{\text{lof}}\) comes out at zero or below. The last of those happens when the model uses up every degree of freedom that is not a replicate, so a saturated model shows no test even though it has pure error.
Variance inflation factors¶
The variance inflation factor of a term measures how much the other terms in the model overlap with it. For an unweighted fit the design matrix is standardized column by column,
and the factors are the diagonal of \((Z'Z)^{-1}\). For a weighted fit they are computed one term at a time by regressing that term on all the others:
A value of \(1\) means the term is orthogonal to every other term in the model.
See Also¶
References¶
- Montgomery, D. C. (2013). Design and Analysis of Experiments, 8th edition. Wiley.
- Draper, N. R., and Smith, H. (1998). Applied Regression Analysis, 3rd edition. Wiley.
- Cook, R. D. (1977). Detection of Influential Observation in Linear Regression. Technometrics, 19(1), 15 to 18.
- NIST/SEMATECH e-Handbook of Statistical Methods, section on process modeling and lack of fit.