Home / DOE / Analyze / Ordinary Least Squares Regression / Results
Regression Results for Ordinary Least Squares¶
Run Regression writes its results to a new worksheet called Regression, inserted after the sheet that was active when you ran it. This page explains every part of that sheet for a quantitative output.
Every run writes a new sheet. The second run does not overwrite the first: it takes the next free name, so a workbook builds up Regression, Regression (2), Regression (3) and so on. Reducing a model therefore leaves the earlier report intact on its own sheet, which is how two versions of a model can be compared.
Top to bottom, the sheet is laid out like this:
| Area | What it is |
|---|---|
| The heading block | Quantum XL Regression Analysis, the design sheet's name, and whether the regression is in coded units |
| Factors and Set point | one row per factor with its range, and the cells you type settings into |
| The prediction area | beside the factors, on the same rows: the model's answer at those settings |
| A divider line | separates the live top of the sheet from the fit below |
| The coefficient table, the model summary and the ANOVA table | the record of the fit |
So the prediction part of the sheet is at the top, not the bottom, and it sits next to the set point cells that drive it.
The coefficient table¶
One row per term in the model, headed Factor, with these columns.
Coeff¶
The coefficient, in the units of the model. Because designs are autocoded by default, this is normally per coded unit rather than per real unit, which is what makes coefficients comparable between factors measured on different scales.
To read the same model in real units, use Uncoded Coefficients.
SE¶
The standard error of the coefficient: how much this estimate would move if the experiment were repeated. Smaller is a better determined coefficient. Its main use is as the denominator of the t statistic.
Do not confuse this column with the Std Error row in the model summary further down the sheet, which is the model's own standard error rather than any one coefficient's.
The standard error of a coefficient is the square root of the matching diagonal entry of the coefficient covariance matrix \(C\):
The \(j\)th diagonal entry belongs to the \(j\)th term of the model, the constant included. This is the number that divides the coefficient to give its \(t\) statistic.
This value is produced by the regression library. The definition above is the standard one.
Full details: Regression Diagnostics.
T¶
The t statistic, the coefficient divided by its standard error:
It is a signal to noise ratio for one term. A large magnitude means the coefficient is well separated from zero relative to its own uncertainty.
P¶
The 2-tail p value, obtained by comparing the t statistic against the t distribution on the residual degrees of freedom. Small means the term is distinguishable from zero.
The sheet colours these:
| Colour | p value |
|---|---|
| Red | 0.05 or below |
| Blue | above 0.05, up to and including 0.1 |
| Black | above 0.1 |
The same thresholds colour the bars on a Pareto.
A P cell reading NA is meaningful. It means the standard error for that term could not be estimated, so no t statistic and no p value exist. It is not a zero and not a missing number: there was nothing to compute.
A P column that is empty rather than NA means something else entirely. That is the uncoded report, where the whole column is suppressed. See Uncoded Coefficients.
VIF¶
The variance inflation factor of a term measures how much the other terms in the model overlap with it. For an unweighted fit the design matrix is standardized column by column,
and the factors are the diagonal of \((Z'Z)^{-1}\). For a weighted fit they are computed one term at a time by regressing that term on all the others:
A value of \(1\) means the term is orthogonal to every other term in the model.
Most designed experiments give VIFs of exactly 1, because a good design makes its terms orthogonal. Values above 1 mean the terms overlap, and the further above, the less the individual coefficients can be separated. At high VIFs coefficients can take the wrong sign and p values mislead in both directions.
This is the number that makes Uncoded Coefficients unsuitable for judging significance: without coding the VIFs inflate, so the p values that depend on them cannot be trusted.
In Model¶
A checkbox on every term except the constant, which has no checkbox and cannot be removed this way. On a design blocked with replicate the S-Hat block row has none either. Clearing a checkbox and re-running the regression drops that term from the model. This is how you reduce a model: clear the terms you do not want and run Run Regression again. The reduced fit is written to a new sheet, and this one stays as it is.
It also decides what the charts show. A term with its box cleared contributes nothing to a prediction, and gets no bar on a Pareto.
A term that is not in the regression table, or that is switched off for the output level being charted, contributes exactly zero to the prediction. It is not an error and there is no warning: the chart is simply drawn from the reduced model. So a chart reflects which terms are active at the moment it is created, and two charts of the same output can differ because the active terms changed between them.
The model summary¶
R squared¶
R squared, the proportion of the variation in the response that the model accounts for:
where \(e_i\) is the residual for run \(i\). It runs from 0 to 1. Higher is a model that explains more, but R squared and significance are different questions: a model can have a low R squared and still contain terms that are clearly real, which happens when the effects are genuine but small against the noise.
Adj R squared¶
Adjusted R squared, which charges the model for the number of terms it uses:
with \(n\) runs and \(p\) terms. Adding a term always raises R squared, even a useless one; it only raises the adjusted form if it earns its degree of freedom. Adjusted R squared can be negative.
For most designed experiments the two are close. The two diverge as \(p\) approaches \(n\), because the residual degrees of freedom \(n - p\) in the numerator shrink.
Std Error¶
The model's standard error of \(Y\), written between Adj R2 and F. It is the spread of the residuals in the units of the response, so unlike R squared it is on the same scale as the data. This is a property of the whole fit, not of any one term, which is what separates it from the SE column in the coefficient table.
F and Sig F¶
F tests the whole model at once, as the ratio of the mean square the model explains to the mean square it does not:
An F near 1 says the model explains no more than noise. Sig F is the probability of seeing an F this large from a model that is really no better than noise, so small values argue the model has genuine predictive content. Below 0.05 is the usual threshold.
On a blocked design the printed F does not divide out from the printed MSR. F and Sig F come from the fit while the block is still in it. The Regression row's sum of squares, degrees of freedom and mean square then have the block taken out of them, and neither F nor Sig F is recomputed afterwards. So the two numbers on the sheet belong to different versions of the model. See Blocking.
FLOF and Sig FLOF¶
These two need an estimate of noise that does not depend on the model being right, which only repeated settings can give. Both rows are written on every OLS report, whether the design has repeats or not.
Pure error is estimated from runs that repeat the same settings. Rows are grouped by their whole row of the design matrix, which is the model's own columns and the blocking columns rather than the factor columns on the sheet, and a group of \(n_i \ge 2\) responses contributes its variance
Summing over the groups,
A group with a single run contributes nothing. If no settings repeat then \(df_{\text{pure}} = 0\) and there is no pure error estimate at all, which is why a design without replicates reports no lack of fit test.
What the residuals contain beyond pure error is lack of fit:
The test compares the lack of fit mean square against the pure error mean square. The lack of fit sum of squares is reported as \(0\) if the arithmetic makes it negative, and every cell of the test reads NA in any of three cases: there is no pure error, the pure error mean square is at or below \(10^{-11}\), or \(df_{\text{lof}}\) comes out at zero or below. The last of those happens when the model uses up every degree of freedom that is not a replicate, so a saturated model shows no test even though it has pure error.
FLOF, its degrees of freedom and Sig FLOF are written to the sheet and nothing else is done with them. Quantum XL applies no threshold to Sig FLOF, so the comparison is yours to make. See Type I and Type II Error.
On a design with no repeated settings both cells read NA. The rows are still there. NA is not a fault and it is not a pass: it means the experiment cannot separate lack of fit from noise at all. The Error pure and Error lof rows of the ANOVA table go the same way, with NA in their sums of squares and mean squares. Their DF cells are the exception and show 0 rather than NA.
The ANOVA table¶
Headed Source, with a row per term and then five summary rows, and these columns.
| Summary row | What it holds |
|---|---|
| Regression | the model's sum of squares, degrees of freedom and mean square |
| Error | the residual sum of squares, its degrees of freedom and MSE |
| Error pure | pure error, from the repeated settings. NA when there are none |
| Error lof | lack of fit, the rest of the residual. NA when there is no pure error |
| Total | the total sum of squares and total degrees of freedom. Its mean square cell is left empty |
The four rows above Total carry one sum of squares, printed twice. Each of them writes the same number into the Seq SS and the Adj SS column, because the split between sequential and adjusted is a property of a term rather than of the model as a whole. Total writes Seq SS only and leaves its Adj SS cell empty.
Seq SS¶
The sequential sum of squares for a term is the amount the regression sum of squares drops when that term is the last one removed from the model. Writing \(SSR(\cdot)\) for the regression sum of squares of a model,
so the value depends on the order the terms are entered. The sequential sums of squares add up to the regression sum of squares.
A categorical term occupies several columns, and its sequential sum of squares is the total of theirs. That is why the table has one row per term rather than one row per column.
Adj SS¶
The adjusted sum of squares for a term is the amount the regression sum of squares drops when that term is removed from the full model, with every other term left in:
It does not depend on the order the terms were entered. For a categorical term every column belonging to it is removed together.
Most experimenters read Adj SS, because each term is judged with all the others already in the model. Seq SS has the property that the values add up to the regression sum of squares.
In an orthogonal design the two sums of squares are equal, because removing a term takes nothing away from the others. Quantum XL relies on that rather than recomputing: where a term occupies a single column and its variance inflation factor is \(1\), the adjusted sum of squares is taken to be the sequential one.
DF, and the mean square column¶
Degrees of freedom for the row, then a mean square headed Adj MS. The report has a second heading, Seq MS, for the sequential case, but nothing in Quantum XL selects it: the sums of squares type is set to adjusted and there is no control that changes it, so the column is always Adj MS in practice.
The bold group-heading rows are only partly filled in. The table groups its terms under a heading for each kind of term present, written exactly as Main, 2 - way and 3 - way with spaces around the hyphen, Quadratic, and Custom for a custom interaction. Those heading rows carry Seq SS and DF and leave Adj SS, Adj MS, F and P empty, which follows from the column always being the adjusted one. The blocking row, written in lower case as block, is the exception and is filled in throughout. Every individual term's own row carries the full set.
Each term's sum of squares is turned into a mean square first, using that term's own degrees of freedom, and the mean square is then tested against the residual mean square:
Both degrees of freedom matter. A term occupying one column has \(df_{\text{term}} = 1\), and only then does \(F\) reduce to \(SS_{\text{term}} / MS_{\text{error}}\). A categorical term or a block occupies several columns, so its \(F\) and \(p\) are built from its own degrees of freedom rather than from \(1\).
Blocking¶
When the model contains a block term, the block is not part of the effect of the factors, so its sum of squares is taken out of the regression sum of squares at the end:
and the regression degrees of freedom drop by one for each blocking column, after which the mean square is recomputed as \(MSR = SSR_{\text{reported}} / df_{\text{reg}}\).
The order matters. The per-term sequential and adjusted sums of squares are computed from the fitted regression sum of squares, the one that still includes the block. Only the reported total has the block removed.
The prediction block¶
Type factor settings into the set point cells and the sheet answers with the model's prediction there. This is a live part of the sheet, not a record of the fit.
| Row | What it holds |
|---|---|
| Y-Hat | the predicted response |
| S-Hat | the predicted standard deviation, from the standard deviation model |
| PI Lower, PI Upper | the prediction interval |
| USL, LSL | specification limits, which you enter |
| Cp, Cpk | capability estimates against those limits |
| DPM estimate | estimated defects per million |
The prediction interval on the regression sheet is three standard deviations either side of the predicted mean, using the predicted standard deviation at those same settings:
The sheet labels this a 99 percent interval. It is a three sigma interval on the fitted standard deviation, not an interval derived from the \(t\) distribution and the residual degrees of freedom.
When the response has no standard deviation model, or every term has been removed from it, that cell holds the standard error of the response regression instead of a prediction. It is then a single constant, so the interval stops varying with the set points. The software says so in the S-Hat cell's own comment.
The capability figures beside the prediction treat the predicted mean and predicted standard deviation as the mean and standard deviation of a normal distribution:
A third figure, Cp written with a subscript p, sits above Cpk on the sheet and needs both limits:
With only one specification limit set, Cp is not reported, and Cpk and DPM use that limit's term alone. The tail areas are multiplied by one million with no bound applied to the result.
The same S-Hat substitution applies here. With no standard deviation model, or none left after terms have been removed, all three figures are computed from the response regression's standard error, so they stop varying with the set points.
A set point outside the factor's low and high is an extrapolation, and the sheet says so in the set point cell's own comment. The model was fitted inside the design region and has no evidence outside it.
What else can appear on the sheet¶
Cell comments explaining each statistic, controlled by Display comments in the regression sheet.
The DOE Advisor, a block of plain-language findings, controlled by Display DOE Advisor on regression sheet. It also marks runs red on the design sheet, on either of two counts:
- A large standardized residual, at plus or minus 2 and again at plus or minus 3.
- High leverage, meaning the run's leverage reaches 3 times the number of active terms divided by the number of rows. A run with an unremarkable residual can be marked red on leverage alone.
Both settings are on the Regression page of DOE Options. Every section the Advisor can write, and the exact conditions for each, is on The DOE Advisor.