Home / Shared Math Details / DOE / Regression Diagnostics
Regression Diagnostics¶
A DOE regression produces one row of diagnostics per fitted row and one row per model term. For a quantitative fit a fitted row is a run; a binary logistic fit collapses identical design rows before it fits, so its rows are covariate patterns. The per-row quantities are leverage, the two studentized residuals and Cook's D, which the Residual Report writes as columns and the Residual Plots draw. The per-term quantities are the coefficient standard error, the covariance matrix it is read from, and the confidence bounds built on it. This page defines all of them.
Where each number comes from. For a quantitative output, leverage, both studentized residuals, Cook's D, the coefficient standard errors and the coefficient bounds are produced by the regression library Quantum XL uses, not by Quantum XL's own code. The definitions given here are the standard published ones that those quantities carry, and the sections say so where it applies. The two logistic engines are different: both of them compute the coefficient standard errors and the coefficient bounds themselves, and the binary engine computes its own leverage as well, so those sections describe the computation directly.
A nominal model has no leverage. The nominal engine computes none and no output writes one, so a nominal regression has no leverage column and no leverage series on a residual plot. Leverage below means the quantitative engine and the binary logistic engine only.
Notation¶
| Term | Description |
|---|---|
| \(X\) | the design matrix, one row per run and one column per model term |
| \(\mathbf{x}_i^{\prime}\) | row \(i\) of the design matrix, the settings of run \(i\) in model terms |
| \(n\) | number of runs used in the fit |
| \(p\) | number of terms in the model, counting the constant |
| \(y_i\), \(\hat{y}_i\) | the observed and fitted response at run \(i\) |
| \(\hat{\mathbf{y}}\), \(\hat{\mathbf{y}}_{(i)}\) | the vector of fitted values from the full model, and from the model refitted without run \(i\) |
| \(e_i\) | the residual of run \(i\), \(y_i - \hat{y}_i\) |
| \(h_i\) | the leverage of run \(i\), or of covariate pattern \(i\) in a binary logistic fit |
| \(s\), \(s^2\) | the model's standard error and the residual mean square |
| \(s_{(i)}\) | the residual standard error of the model fitted without run \(i\) |
| \(r_i\) | the studentized residual of run \(i\) |
| \(t_i\) | the R-studentized residual of run \(i\) |
| \(D_i\) | Cook's D for run \(i\) |
| \(b_j\) | the coefficient of term \(j\) |
| \(SE(b_j)\) | the standard error of the coefficient of term \(j\) |
| \(C\), \(C_{jj}\) | the coefficient covariance matrix and its \(j\)th diagonal entry |
| \(H\) | the weighted hat matrix of a binary logistic fit |
| \(V\) | the diagonal weight matrix of a binary logistic fit |
| \(m_i\) | the number of trials at covariate pattern \(i\) of a binary logistic fit |
| \(\hat{\pi}_i\) | the fitted probability at covariate pattern \(i\) of a binary logistic fit; for a nominal fit, run \(i\)'s vector of fitted probabilities across the non-reference levels |
| \(K\) | the number of levels of a nominal output |
| \(t_{0.975,\,n-p}\) | the 97.5 percent point of the \(t\) distribution on \(n - p\) degrees of freedom |
Leverage¶
Leverage is the diagonal of the hat matrix, the matrix that turns the observed responses into the fitted ones. For run \(i\),
where \(\mathbf{x}_i^{\prime}\) is run \(i\)'s row of the design matrix. It measures how unusual run \(i\) is in its factor settings alone, with no reference to the response: the response does not appear in the expression at all. Leverage runs from \(0\) to \(1\), and the leverages of all runs add up to \(p\), the number of terms in the model counting the constant, so the average leverage across a design is \(p/n\).
For a quantitative output this value is produced by the regression library. The definition above is the standard one.
A binary logistic model has its own leverage, and Quantum XL computes it rather than taking it from the library. Its rows are not runs. The binary engine collapses identical design rows before it fits, so this quantity, like every other quantity of that fit, is one value per covariate pattern. It is the diagonal of the weighted hat matrix
where \(V\) is the diagonal matrix whose entry for pattern \(i\) is \(m_i \hat{\pi}_i (1 - \hat{\pi}_i)\): the number of trials at that pattern, times the fitted probability, times one minus the fitted probability. Where an entry of \(V\) is not positive, the corresponding entry of \(V^{1/2}\) is left at zero rather than having a square root taken.
The Residual Report writes this column as Leverage, and it is one of the quantities the Residual Plots offer on their Y list. See Understanding Least Square Residuals for where the columns appear, and Binomial Logistic Residuals for the logistic set.
The studentized residual¶
The residual of run \(i\) is \(e_i = y_i - \hat{y}_i\). Residuals do not all have the same variance: a run with high leverage pulls the fitted surface toward itself, so its residual is smaller than a run with low leverage would give on the same data. The studentized residual divides each residual by its own estimated standard deviation:
\(s\) is the model's own standard error, the square root of the residual mean square, computed from all \(n\) runs. Because the divisor carries the \(\sqrt{1 - h_i}\) factor, studentized residuals are comparable across runs in a way that raw residuals are not.
Run \(i\) is one of the runs \(s\) is computed from, so a run that fits badly raises the number it is then divided by.
This value is produced by the regression library. The definition above is the standard one.
The R-studentized residual¶
The R-studentized residual has the same form as the studentized residual, with one difference: the variance estimate leaves run \(i\) out.
\(s_{(i)}\) is the residual standard error of the model fitted without run \(i\), so a run cannot inflate its own denominator. It does not require refitting the model \(n\) times, because it can be written from the full fit:
where \(p\) is the number of terms counting the constant. The two forms give different numbers for the same run. They always agree in sign, and \(|t_i|\) is larger than \(|r_i|\) exactly when \(r_i^2 > 1\).
This value is produced by the regression library. The definition above is the standard one.
Cook's D¶
Cook's D measures how far the whole vector of fitted values moves when one run is dropped from the fit. Writing \(\hat{\mathbf{y}}\) for the fitted values from the full model and \(\hat{\mathbf{y}}_{(i)}\) for the fitted values from the model refitted without run \(i\),
The same quantity can be written from the full fit alone, which is why a value is available for every run without refitting anything:
That second form shows what the number combines. The first factor is built from the studentized residual, which is about how well the run was fitted. The second is a function of leverage alone, which is about where the run sits in factor space. Both factors have to be away from zero for \(D_i\) to be large: a run lying exactly on the fitted surface has \(r_i = 0\) and therefore \(D_i = 0\) however high its leverage is, and a run at the centre of the design has small \(h_i / (1 - h_i)\) however badly it fits.
This value is produced by the regression library. The definitions above are the standard ones.
The standard error of a coefficient¶
The standard error of a coefficient is the square root of the matching diagonal entry of the coefficient covariance matrix \(C\):
The \(j\)th diagonal entry belongs to the \(j\)th term of the model, the constant included. This is the number that divides the coefficient to give the ratio reported beside it: the \(t\) statistic of a quantitative fit, and the \(Z\) of either logistic fit, whose coefficient table heads that column Z.
For a quantitative output this value is produced by the regression library. The definition above is the standard one.
The two logistic engines compute their own, and they do not build the same matrix.
For a binary output the covariance matrix is the inverse of the observed information matrix at the converged coefficients, \((X^{\prime} V X)^{-1}\), with \(V\) the same diagonal matrix of fitted binomial variances used for logistic leverage, so
For a nominal output the covariance matrix is again the inverse of the engine's own observed information matrix at the converged coefficients, but that matrix is not the one above. A nominal model estimates a separate block of coefficients for each non-reference level, so with \(K\) levels and \(p\) terms the information matrix is square of side \((K-1)p\) rather than \(p\), and it is accumulated over a category-expanded design rather than over \(X\) itself. Each run contributes a term built from its own design row together with
where \(\hat{\pi}_i\) is that run's vector of fitted probabilities across the non-reference levels. That inner matrix is not diagonal, and its off-diagonal entries are what tie one level's coefficients to another's, so it is not the binary engine's \(V\) under a different name and the \((X^{\prime} V X)^{-1}\) form does not apply. The standard errors are the square roots of the diagonal of the inverse, taken in the order the coefficients are stored.
A diagonal entry that is not strictly positive gives a reported standard error of exactly \(0\) rather than the square root of a negative number. That holds in both logistic engines.
The regression sheet writes this as the SE column of the coefficient table. It is not the same number as the Std Error row in the model summary lower down the sheet, which is \(s\), the model's own standard error.
The coefficient covariance matrix¶
For an unweighted least squares fit the coefficient covariance matrix is
with \(s^2\) the residual mean square. Its diagonal holds the variances of the individual coefficients, and its off-diagonal entries hold the covariances between pairs of them, so two terms whose design columns overlap have a large off-diagonal entry and their coefficients move together from one experiment to the next. In the standard definition it is the matrix the coefficient standard errors are read off, and the same overlap it records is what the variance inflation factors report in scaled form.
Quantum XL does not write this matrix anywhere. The quantitative engine declares a field for it and the one line that would fill that field is commented out, so no sheet reports the covariances themselves. What reaches a sheet is the SE column, which the regression library supplies as finished numbers. By the standard definition those are the square roots of this matrix's diagonal, but Quantum XL never forms the matrix and cannot see how the library arrives at them.
The scaled description of the same overlap is on Sums of Squares, under variance inflation factors.
Confidence bounds on a coefficient¶
A confidence bound on a coefficient is the coefficient plus or minus a multiplier times its standard error. The two engine families use different multipliers, and the difference shows up in the numbers.
For a quantitative output the bounds come from the regression library, which is asked for a 95 percent interval on each coefficient. The standard definition of such an interval is
In that definition the multiplier is the \(97.5\) percent point of the \(t\) distribution on the residual degrees of freedom, so it depends on both the number of runs and the number of terms. Quantum XL asks the library for the interval and reads back the two bounds, so the multiplier the library used is not visible to Quantum XL and is not reported anywhere. These bounds are computed but no output sheet reports them.
The logistic engines use a fixed normal multiplier instead of a \(t\) value. Both the binary and the nominal engine multiply by the literal constant \(1.95996398454005\), the 95 percent two-sided standard normal value:
That number does not change with the number of runs or the number of terms. A bound recomputed with a \(t\) multiplier will therefore not reproduce the product's number.
The coefficient bounds themselves are not written to a sheet on the logistic side either. What the sheet reports is the odds ratio bounds, which are these bounds exponentiated:
Because the exponential is not linear, the odds ratio bounds are not symmetric about the odds ratio, even though the coefficient bounds are symmetric about the coefficient.
The odds ratio columns are described on the binary results page and the nominal results page.
See Also¶
References¶
- Cook, R. D. (1977). Detection of Influential Observation in Linear Regression. Technometrics, 19(1), 15 to 18.
- Belsley, D. A., Kuh, E., and Welsch, R. E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley.
- Montgomery, D. C., Peck, E. A., and Vining, G. G. (2012). Introduction to Linear Regression Analysis, 5th edition. Wiley.
- Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd edition. Wiley.