Home / DOE / Charts / Residual Plots / Math Details
Math Details¶
This page gives every quantity a residual plot can put on either axis and where each one comes from. The two output types offer completely different lists, and they are built in completely different places, so they are taken in turn.
Notation¶
| Term | Description |
|---|---|
| \(i\) | one row of the fit: a used run on a quantitative fit, a covariate pattern on a binary one |
| \(y_i\), \(\hat{y}_i\) | the observed and the fitted response at row \(i\) of a quantitative fit |
| \(e_i\) | the residual at row \(i\) of a quantitative fit |
| \(h_i\) | the leverage of row \(i\) |
| \(r_i\) | the studentized residual of row \(i\) of a quantitative fit |
| \(t_i\) | the R-studentized residual of row \(i\), the studentized deleted form |
| \(D_i\) | Cook's D at row \(i\) |
| \(p\), \(s^2\) | the number of model terms and the mean square error of the quantitative fit |
| \(\hat{\pi}_i\), \(m_i\) | the fitted probability and the number of trials at covariate pattern \(i\) of a binary fit |
| \(r_{P,i}\) | the Pearson residual at pattern \(i\) |
| \(r_{S,i}\) | the standardized Pearson residual at pattern \(i\) |
| \(d_i\) | the deviance residual at pattern \(i\) |
What one point is¶
On a quantitative output a point is a used run. A run whose response was missing, or whose row the model did not use, produces no point.
On a binary output a point is a covariate pattern, not a run. The binary engine collapses identical design rows before it fits, so a design with replication has fewer points than it has runs, and two runs at identical settings are one point carrying their combined trial count. This is not a property of the chart; it is a property of that fit, and every quantity the binary lists offer is indexed the same way.
The quantitative quantities¶
All six of them are produced by the regression library, not by Quantum XL. The fitted values, the residuals, the leverages, both studentized forms and Cook's D are read straight out of the library's regression object and stored. Quantum XL chooses the model, hands over the design matrix and the response, and then reads these vectors back; it does not form any of them.
That is worth stating because the definitions below are the standard textbook ones and the library is expected to implement them, but the page documents what the product reports rather than asserting an implementation nobody here wrote.
Actual Y is the response as loaded, and Predicted Y is the library's fitted value at that run. Run # is the value in the design's own Run column, and a factor row carries that run's uncoded setting of that factor. The block row carries the run's block.
Residual.
Leverage.
Leverage is the diagonal of the hat matrix, the matrix that turns the observed responses into the fitted ones. For run \(i\),
where \(\mathbf{x}_i^{\prime}\) is run \(i\)'s row of the design matrix. It measures how unusual run \(i\) is in its factor settings alone, with no reference to the response: the response does not appear in the expression at all. Leverage runs from \(0\) to \(1\), and the leverages of all runs add up to \(p\), the number of terms in the model counting the constant, so the average leverage across a design is \(p/n\).
For a quantitative output this value is produced by the regression library. The definition above is the standard one.
Studentized Residual.
The residual of run \(i\) is \(e_i = y_i - \hat{y}_i\). Residuals do not all have the same variance: a run with high leverage pulls the fitted surface toward itself, so its residual is smaller than a run with low leverage would give on the same data. The studentized residual divides each residual by its own estimated standard deviation:
\(s\) is the model's own standard error, the square root of the residual mean square, computed from all \(n\) runs. Because the divisor carries the \(\sqrt{1 - h_i}\) factor, studentized residuals are comparable across runs in a way that raw residuals are not.
Run \(i\) is one of the runs \(s\) is computed from, so a run that fits badly raises the number it is then divided by.
This value is produced by the regression library. The definition above is the standard one.
R-Studentized Residual.
The R-studentized residual has the same form as the studentized residual, with one difference: the variance estimate leaves run \(i\) out.
\(s_{(i)}\) is the residual standard error of the model fitted without run \(i\), so a run cannot inflate its own denominator. It does not require refitting the model \(n\) times, because it can be written from the full fit:
where \(p\) is the number of terms counting the constant. The two forms give different numbers for the same run. They always agree in sign, and \(|t_i|\) is larger than \(|r_i|\) exactly when \(r_i^2 > 1\).
This value is produced by the regression library. The definition above is the standard one.
Cook's D.
Cook's D measures how far the whole vector of fitted values moves when one run is dropped from the fit. Writing \(\hat{\mathbf{y}}\) for the fitted values from the full model and \(\hat{\mathbf{y}}_{(i)}\) for the fitted values from the model refitted without run \(i\),
The same quantity can be written from the full fit alone, which is why a value is available for every run without refitting anything:
That second form shows what the number combines. The first factor is built from the studentized residual, which is about how well the run was fitted. The second is a function of leverage alone, which is about where the run sits in factor space. Both factors have to be away from zero for \(D_i\) to be large: a run lying exactly on the fitted surface has \(r_i = 0\) and therefore \(D_i = 0\) however high its leverage is, and a run at the centre of the design has small \(h_i / (1 - h_i)\) however badly it fits.
This value is produced by the regression library. The definitions above are the standard ones.
The binary quantities¶
These four are ours. Unlike the quantitative list, the logistic diagnostics are computed by Quantum XL's own binary engine, from the Pearson and deviance residuals and the leverage of each covariate pattern. The leverage they are built on is also ours:
A binary logistic model has its own leverage, and Quantum XL computes it rather than taking it from the library. Its rows are not runs. The binary engine collapses identical design rows before it fits, so this quantity, like every other quantity of that fit, is one value per covariate pattern. It is the diagonal of the weighted hat matrix
where \(V\) is the diagonal matrix whose entry for pattern \(i\) is \(m_i \hat{\pi}_i (1 - \hat{\pi}_i)\): the number of trials at that pattern, times the fitted probability, times one minus the fitted probability. Where an entry of \(V\) is not positive, the corresponding entry of \(V^{1/2}\) is left at zero rather than having a square root taken.
Predicted is the fitted probability \(\hat{\pi}_i\) at that pattern.
The remaining four are built from the Pearson residual, its standardized form and the deviance residual, all three defined on Logistic Estimation:
Delta Chi-Squared.
Delta Deviance.
Delta Beta.
Delta Beta Standardized, which is the same form built from the standardized residual:
Three relations hold exactly and are worth using as checks. They follow from the definitions above and from nothing else, so any column that breaks one of them is not reporting what this page says it is:
So Delta Chi-Squared is exactly the square of the standardized Pearson residual, Delta Beta is that multiplied by the leverage, and the standardized Delta Beta is the plain one divided by one minus the leverage. Delta Deviance is the only one of the four that brings in the deviance residual, and its second term is Delta Beta.
Delta Deviance and Delta Beta share a term. The quantity \(r_{P,i}^2 h_i / (1 - h_i)\) is Delta Beta, and it is also the second term of Delta Deviance, so the two columns are not independent of each other. The engine computes each from its own expression rather than reusing the other.
A note on what the plots are read against¶
The thresholds a reader is likely to apply to these quantities, and the grades the DOE Advisor gives them, are on Residual Plots rather than here. This page defines the numbers; that one says what the product does with them.
Shared Math Details used here¶
This chart uses shared formulas defined once in Shared Math Details. See those pages for the exact definitions.
| Shared concept | Used here for | Reference |
|---|---|---|
| Regression diagnostics | leverage, both studentized residuals and Cook's D | Regression Diagnostics |
| Logistic estimation | the Pearson and deviance residuals the four binary quantities are built from | Logistic Estimation |
See Also¶
- Residual Plots
- Options
- Observed vs Predicted Math Details, the same fitted values before they become residuals
- Regression Diagnostics
- Chart Source
References¶
- Montgomery, D. C., Peck, E. A., and Vining, G. G. (2012). Introduction to Linear Regression Analysis, 5th edition. Wiley.
- Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd edition. Wiley.